Update 26.11.2015: This issue is resolved in VMware ESXi 6.0 patch ESXi600-201511001. For more information, see VMware ESXi 6.0, Patch Release ESXi600-201511001 (2137545).
Not long ago VMware fixed a CBT bug in vSphere ESXi 6.0 (fixed since build 2715440 – read KB 2114076).
Today VMware posted a KB article describing a new CBT issue with ESXi 6.0:
Backups with Changed Block Tracking can return incorrect changed sectors in ESXi 6.0 (2136854)
So if you use vSphere ESXi 6.0 and a backup solution that is relying on CBT (changed block tracking) you should take care of this KB article, as your virtual machine backup might be inconsistent. And maybe you know it for sure when you need your backup most.
Here is what VMware writes about the issue:
This issue occurs due to an issue with CBT in the disklib area, this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost. The main backup payload data is never lost and it is always written to the backend device. However, the corresponding change tracking information entries which occur during the consolidation task are missed. Subsequent QueryDiskChangedAreas() calls do not include these missed blocks, hence a backup based on this CBT data is inconsistent.
At the moment there is no real resolution for the issue.
VMware offers three workarounds – but i guess none of them will make you happy:
- downgrade your ESXi hosts to 5.5 and the virtual hardware from 11 to 10
- do full virtual machine backups instead of incremental backups
- shutdown the virtual machines before doing an incremental backup
Nice, isn’t it?
Number one is a lot of work depending on the size of your environment of course. And – above all it means downtime to your virtual machines when you downgrade the virtual hardware.
Number two may be a possibility if you only have to backup a small number of VMs/small amount of data. Otherwise your backup window may be too short and/or you will blow up your backup pool.
Number three: shutdown during incremental backup = downtime = bad… maybe a workaround for some not so important VMs but definitely not a resolution for the bulk of your VMs.
Hopefully VMware brings a hotfix/resolution for this issue very soon… check back to the KB articles and this blog post and you will not miss any news.
Would be interesting to see if you could disable CBT per VM to workaround this mess… http://kb.vmware.com/kb/1020128
Also, please note that in Veeam you can disable CBT in the job.
of course you can disable CBT – but then you perform full backups instead of incremental backups. In this case you need more time for your backup and/or you will blow up your backup pools.
another awful bug, that’s 6 major ones ive tracked / am tracking now. As far as i know the next round of patches are not due out till January…. 🙁
VMware has announced a Patch for this today.
After applying the patch you have to reset CBT on all your VMs. You can use this powershell cmdlets to automate reset o one or multiple VMs.