Attention: new CBT bug in vSphere ESXi 6.0

Update 26.11.2015: This issue is resolved in VMware ESXi 6.0 patch ESXi600-201511001. For more information, see VMware ESXi 6.0, Patch Release ESXi600-201511001 (2137545).

—

Not long ago VMware fixed a CBT bug in vSphere ESXi 6.0 (fixed since build 2715440 – read KB 2114076).

Today VMware posted a KB article describing a new CBT issue with ESXi 6.0:

Backups with Changed Block Tracking can return incorrect changed sectors in ESXi 6.0 (2136854)

So if you use vSphere ESXi 6.0 and a backup solution that is relying on CBT (changed block tracking) you should take care of this KB article, as your virtual machine backup might be inconsistent. And maybe you know it for sure when you need your backup most.

Here is what VMware writes about the issue:

This issue occurs due to an issue with CBT in the disklib area, this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost. The main backup payload data is never lost and it is always written to the backend device. However, the corresponding change tracking information entries which occur during the consolidation task are missed. Subsequent QueryDiskChangedAreas() calls do not include these missed blocks, hence a backup based on this CBT data is inconsistent.

At the moment there is no real resolution for the issue.

VMware offers three workarounds – but i guess none of them will make you happy:

• downgrade your ESXi hosts to 5.5 and the virtual hardware from 11 to 10
• do full virtual machine backups instead of incremental backups
• shutdown the virtual machines before doing an incremental backup

Nice, isn’t it?

Number one is a lot of work depending on the size of your environment of course. And – above all it means downtime to your virtual machines when you downgrade the virtual hardware.

Number two may be a possibility if you only have to backup a small number of VMs/small amount of data. Otherwise your backup window may be too short and/or you will blow up your backup pool.

Number three: shutdown during incremental  backup = downtime = bad… maybe a workaround for some not so important VMs but definitely not a resolution for the bulk of your VMs.

Hopefully VMware brings a hotfix/resolution for this issue very soon… check back to the KB articles and this blog post and you will not miss any news.