Troubleshooting - VMware Backup

Backup Restore  

The following section provides information on some of the troubleshooting scenarios related to backup:

Backup Failure

During create snapshot phase When the virtual machine version is upgraded, make sure all the VMware tools are upgraded.
After Upgrading Proxy Server from 8.0 to 9.0 Uninstall VDDK 1.1 on the proxy computer and re-install VDDK 1.2 before performing backup.

*For 64-bit operating systems: From the VDDK x64 installation directory , unzip the vddk64 folder to a temporary location. Once unzipped, copy the bin, lib, and plugins folders to:

C:\Program Files\VMware\VMware Virtual Disk Development Kit
Backup may fail with an error that a driver cannot be found Cause 1: You may be using an older version of VDDK. To resolve this issue, upgrade your VDDK software to version 1.2.1 or higher.

Cause 2: The driver may be in a stopping state. To resolve this issue, ensure the VMware driver service is in a healthy state. This may require a reboot.

With "Received empty disk extents Mount VM_VCB4 failed" error Use the utility Change tracking to reset the Change tracking. Follow the commands given below:

CheckChangeTracking /host x-vcenter /user branzini /password Brown010 /vm iks-dc1.iks.it /disable

CheckChangeTracking /host x-vcenter /user branzini /password Brown010 /vm iks-dc1.iks.it /enable

CheckChangeTracking /host x-vcenter /user branzini /password Brown010 /vm iks-dc1.iks.it

It may be necessary to run this recycling of change tracking with the virtual machine powered down. It may also be necessary to power cycle the virtual machine after disabling the change tracking and again after enabling it.
With "Disk query change failures" error Power cycle the virtual machine under backup.

 
With "Duplicate index entry" error Check to see if there are virtual machines with duplicate disk names and, if so, rename one.

 
For a VADP backup Changed Block Tracking (CBT) must be enabled at the time of the backup. While the Virtual Server iDataAgent enables this automatically, it may not be able to do so for cloned and migrated VMs. Check to ensure CBT is enabled for the VM for which a VADP backup failed as described below.

To check if a virtual disk has CBT enabled, open the vSphere Client, select a powered-off VM, and click Edit...Settings>Options>Advanced/General>Configuration Parameters. Ensure that:

The VM's (.vmx) file contains the entry:

ctkEnabled = "TRUE"

For each virtual disk, the .vmx file contains the entry:

scsix:x.ctkEnabled = "TRUE"

For each virtual disk and snapshot disk there is a .ctk file. For example:

vmname.vmdk
vmname-flat.vmdk
vmname-ctk.vmdk
vmname-000001.vmdk
vmname-000001-delta.vmdk
vmname-000001-ctk.vmdk

HotAdd backups that have a hard failure may result in mounted disks on the proxy that are not automatically cleaned up To resolve this issue, manually remove the disks from the proxy virtual machine and delete the \tmp\vmware directory created in the current user's Documents and Settings directory, typically found at C:\Documents and Settings\username\Local Settings\Temp\vmware-username\. The next time a snapshot is created or deleted for the virtual machine the redo logs will get cleaned up.
File-Level and Volume-Level Backup Failures If the virtual machine that is backed up contains disks that have not been initialized, then the volume-level and file-level backup operations will fail for that virtual machine.
with following error:

FAILED to download config file [Some path].vmx, [1][12002][Unknown native error code 12002

 This error appears if WinHTTP proxy is set on the client computer where Virtual Server iDataAgent is installed.

Before performing backup, ensure that WinHTTP proxy is not set on the client computer. Follow the steps given below to  verify WinHTTP proxy configuration:

  1. On the client computer, open the command prompt and enter following command:

    C:\>netsh WinHTTP Show Proxy

    If you get following reponse, it indicates that the WinHTTP proxy is set on the client computer.

    Current WinHTTP proxy settings:

    Direct access (no proxy server)

  2. Use the following command to reset the WinHTTP proxy:

    C:\>netsh WinHTTP Reset Proxy

Incorrect Virtual Machine is backed up during incremental backup

During incremental backup, if there are virtual machines with duplicate GUIDs, rename the virtual machine GUIDS to ensure they are unique.

Version 4.0 vStorage option, backs up full disk rather than data only copy

To back up virtual machine as a data only copy using vStorage option, upgrade the virtual machine to Version 7.0.

If a virtual machine has been cloned or migrated from another ESX Server, ensure that the virtual machine is rebooted prior to running your first backup.

How to Fix virtual machines that have the same GUID across vCentre/ESX server

If there are multiple virtual machines with same GUID:

  1. Configure the instance with Virtual Center option
  2. Right-click the BackupSet and select Properties
  3. Select Automatically add new VMs that do not qualify for membership in any of the subclients option.
  4. Run a Full backup. This will automatically fix the virtual machines with duplicate GUID's and update the subclient.

If the GUIDs of the virtual machines associated to a subclient are modified:

  1. Run a manual discovery. Follow the steps given below to run a manual discovery:
  2. The GUIDs of the virtual machines that were modified, will be marked for deletion.
  3. Re-associate the virtual machines in order for the backup to continue.

Disk level backup of Windows virtual machine completes with error

If the disk level backup of a Windows virtual machine completes with the following error: "Metadata collection failed for Virtual machine [B_VM1_disk_Full], File and Volume browse will not be available for this VM. [Initialization from Disk failed, <6001><1><MBR signature does not match>]":

The virtual machines may contain disks that are in uninitialized state. Initialize the disks before running a backup, in order to collect metadata enabling file level restores.

consolidate helper-0 snapshot on vCenter Snapshot Manager

Symptom

The consolidate helper-0 snapshots are created on the vCenter Snapshot Manager and they cannot be deleted after the backup job completes. The following error message appears if you delete the snapshot from vCenter Snapshot Manager:

Unable to access file <unspecified filename> since it is locked.

Cause

This scenario can arise when the consolidate helper-0 snapshot is not commited to VMDK. This issue can be VMware I/O issue.

Solution

Adjust the virtual environment to resolve high I/O.  Restart the virtual machine to give the quiet time that is required commit the snapshot.

If you are unable to delete the snapshot, contact VMware Support for assistance.

VMware Driver Not Found

Symptom

After installing any service pack, the following error message appears when you start the backup operation:

CVMDiskInfo::_MountVolumes() - VixMntapi_MountVolume failed Err=5ef3 Driver not found

Solution

  1. Check the status of the driver services using the following commands:

    sc query vstor2-mntapi10-shared

    sc start vstor2-mntapi10-shared

  2. If the driver service is in stop or pending state, reboot the machine where you have installed Virtual Server iDataAgent.

Completed with One or More Errors

Backup jobs from Virtual Server iDataAgent will be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

Backup Fails after Modifications in vCenter Credentials at the Instance Level

Symptom

If an ESX server moves from one vCenter to another, you have to modify the instance created for the vCenter. You must provide the Host name and login credentials for the new vCenter. In such scenario, the the subsequent backup of any virtual machine on the ESX server may fail.

Cause

In the CommSeve database the vCenter hostname and login credentials are updated only for the instance and not updated for all the subclients in the instance.

Solution

If you update the vCenter hostname and login credentials for any instance, perform the following:

  1. From the CommCell Browser, navigate to <Client> | Virtual Server | <Instance> | defaultBackupset.
  2. Right- click any subclient and click Properties.
  3. Click OK.
  4. Repeat above steps for all the subclients in the instance. You can now perform backup of any subclient in the instance.

Backup fails to download VMX files

Symptom

The backup operation fails with a error: Unable to download VMX files.

Cause

The name of the datastore contains + or @.

Solution

  1. Check if the datastore of any virtual machine included in the backup contains + or @ characters in its name.
  2. Rename the datastore and remove + and @ characters from the name.
  3. Perform the backup operation.

Virtual machine client names are getting created with '_1' appended to the original client name

Symptom

When viewing virtual machines in the Client Computers list, you may see duplicated client names (for example, <ClientName> and <ClientName>_1).

Cause

Virtual machine information is added to the database during the discovery phase of a backup. If information changes, a new client is discovered during the discovery phase.

Making changes to information such as the virtual machine fully qualified domain name (FQDN), the GUID, or the host name can cause this issue.

Resolution

To avoid duplicate client entries caused by different client or host name:

Use the install software option for the client level in the CommCell Console (rather than interactively installing software to the virtual machine Guest Operating System).

To correct existing clients:

If duplicate clients are created with '_1" appended to the original client name, you can perform the following steps to merge the duplicate client back into the original: 

  1. At a command prompt, navigate to the software installation path, log in to the CommServer, and run the following script:

    qoperation execscript -sn QS_SetVMClient -si @sourceClient='<ClientName>_1' -si @destClient='<ClientName>'

    where <ClientName> is the original client name and <ClientName>_1 is the duplicate.

    This script reassigns all backup history from <ClientName>_1 to <ClientName>. This enables you to view backup history, and to generate Job Summary Reports with the Include Protected VMs option enabled.

  2. Remove the duplicate clients:
    1. In the CommCell Console, go to Control Panel | User Preferences.
    2. Click the Client Computer Filter tab.
    3. Select the Show Virtual Server Discovered Clients option.
    4. Delete the duplicate clients from the CommCell Browser.