Update 05/08/2016: The issue is fixed with ESXi 6.0 Patch 03: VMware ESXi 6.0, Patch ESXi600-201608401-BG: Updates esx-base, vsanhealth, vsan VIBs (2145664)
—- —– —– ——
Update 25/06/2016: VMware published a KB article covering the issue described in this blog post two days ago (VMware KB 2145895). If you read the KB and the post carefully, you will find some differences. Eg. VMware states in the KB: “No data is lost. Due to this issue, incremental backups are effectively full backups.” As I have seen various affected virtual machines where the backup was not equal to a full backup, I decided not to change the blog post. I still recommend performing a CBT reset to ensure that you can rely on your backup.
—- —– —– ——
Just a short notice if you are already on ESXi 6:
There is an issue with Change Block Tracking (CBT) in Express Patch 6 for ESXi 6 (Build 3825889) that can affect your backup if your backup software relies on CBT.
- affected are only VMs with Operating System Server 2008 or newer (Linux VMs are unaffected.)
- all vHW Versions (7, 8, 9, 10 and 11) and all VMware tools versions are affected
- the issue occurs with thick and thin provisioned disks
- the issue only occurs when the backup software uses quiesced snapshots
Error messages:
You can find errors in the vmware.log of the affected VMs similar to:
vcpu-0| I120: SNAPSHOT:SnapshotBranchDisk: Failed to acquire current epoch for disk /vmfs/volumes/
vmdk : Change tracking is not active for this disk 572.
Other symptoms:
- you have to backup an abnormal amount of data (it is almost a full backup even if it is an incremental one)
- longer backup durations due to the abnormal amount of data sent to the backup software
Resolution:
There are currently 2 workarounds available:
- don’t use quiesced (application-aware) snapshots
- Revert to a previous version of ESXi, e.g. vSphere ESXi 6 U2 (build 3620759) (https://kb.vmware.com/kb/1033604)
If you already have affected virtual machines, migrate them to a host using an unaffected ESXi version (eg. VMware ESXi 6 Update 2, build 3620759).
Then you have to reset CBT of the affected VMs and perform a Full Backup to ensure that your backup is reliable (How to reset CBT on multiple VMs using PowerCLI)
Here you can find a short description: How to rollback/revert to a previous ESXi Version
Additional information:
- VMware VDP seems to be affected by the issue
- IBM TSM seems to be affected by the issue
- Veeam Backup & Replication seems not to be affected as long as you do not enable VMware Tools quiescence (in the backup job settings):
Thanks for sharing..
f*** this is the reason for the increased backups – thx for sharing
Another bug with CBT
same here
Hey there…is there an official VMwareKB for this bug yet?
Thanks for info! What about replicas? same?
@Anthony: As far as I know it should be published soon
Hi, if you have enabled Veeam InGuest processing it will overwrite a set “VMware Tools quiescence” setting in the job configuration. So even if you have enabled quiescence you are maybe not affected because you had enabled Veeam InGuest processing.
@Andreas: thank you for this valuable input
Hi Andreas. Chris from Rubrik here. Just to confirm, we don’t use the VMware Tools quiescence for application consistent backups and are thus not affected by this issue. Cheers!
Hi Andreas,
We use Veeam and I have this issue today
SNAPSHOT:SnapshotBranchDisk: Failed to acquire current epoch for disk /vmfs/volumes/2f6294e0-bc15bc5d/edeacwdp2vnf01/edeacwdp2vnf01.vmdk : Change tracking is not active for this disk 572.
And we use VMware Tools quiescence
Until now I see this only in 2 VMs (Windows 2012)
Still investigating if I have more issues.
@Luciano: take a look at the amount of data sent to your backup system. There is a good Chance that it is abnormal high compared to incremental backups before applying EP6.
@Andreas for now I don’t see to much difference in the backup sizes.
But since I had some full backups in some of the jobs, need to double check this in the next incremental backup jobs.
Isn’t this fixed in VMWare Tools 10.0.9?
KB article:
http://kb.vmware.com/kb/2145895
VMware has responded with this: http://blogs.vmware.com/kb/2016/06/windows-2008-incremental-backups-become-full-backups-esxi-6-0-b3825889.html
VMware top brass are super egomaniac, I really think they have no motivation to make quality software because bugs generate a lot o support revenues for them, and they don’t give a damn about customer complains at all, people should start moving away from VMware products, it’s getting so ridiculously complicated, it will be very buggy and super expensive.
Time to plan to migrate to other products in the next 1-4 years.