PrKB12133 : TID0908131438

How to use EVC feature on VMware clusters with Vmotion ? (How to expand clusters with latest technology servers) procedure intervention procedure

VMware cluster expansion, Vmotion and CPU compatibility

Ideally, Vmotion feature requires exact same CPU model on source and target ESXs to allow hot migration. The reason of this constraint is linked to CPU instructions: for example, a VM using SSE 4.1 requires that all ESXs are SSE 4.1 compliant.

A major difficulty occurs on VMware cluster expansion with a new server: a different CPU with new instructions is added and Vmotion will not be possible.

In order to have more CPU compatibility, VMware sorted CPU per families and allowed Vmotion between two ESXs as far as CPUs are part of the same family. This is adding more possibilities but quite restrictive, migration is possible only between CPU of same generation Xeon 5105 & 5150 for instance.

Another solution is ?CPU masking?, the idea is to hide instructions of new CPU in order to ?align? on instructions set of oldest CPU in cluster. (=CPU instructions limitation). This procedure requires a lot of work: first, each CPU of the cluster has to be checked in order to find the common instructions for the cluster. Second, CPU masking setting has to be done on each Virtual Machine manually. Third, ?CPU masking? is not supported by VMware.

In order to solve this problem definitely, VMware added a new feature call EVC: Enhanced Vmotion Compatibility. The way that EVC works is the same than ?CPU masking? but this is automatic, simple and supported. It requires recent CPU and two BIOS options enabled (details listed below). Practically on Bull products, EVC is supported since NovaScale R440 and R460 servers and all next servers.Important notice: EVC will not allow Vmotion between AMD and Intel CPUs.

More information about EVC on following article: http://kb.vmware.com/kb/1003212 .

The coming section will explain practically how to have EVC cluster, what?s the production impact and how to add a new server into an EVC cluster.

Requirements and production Impact to enable EVC

Hardware:

All ESX hosts must have a CPU listed in ?EVC Cluster Baseline? part of VMware article.

Software:

ESX 3.5 update 2 or higher / vCenter 2.5 update 2 or higher.

ESX 4.0 / vCenter 4.0 natively support EVC.

Production Impact:

Each ESX needs to be rebooted in order to have the two BIOS options enabled.
EVC is effective only after a shut down of all the VMs running into a cluster.

Procedure to enable EVC mode

1 - Check and apply hardware/software requirements listed below.

2 - On Virtual Center, check cluster properties about EVC.

Note: EVC baseline compliant hardware is marked as ?incompatible? until two BIOS options are enabled.

3 - On each ESX of the cluster, enable two BIOS option listed below : (For Intel CPU)

?Hardware Virtualization?
?eXecute Disable? ( a.k.a. XD)

4 - Check cluster properties again after a reboot :

NUMA setting

Note: the hardware is not marked as incompatible any more.

5 - Last step is a complete shutdown of all virtual machines of cluster.

6 - After this, check cluster properties again :

Cluster_CPCC Settings

Note: Virtual Center automatically selected the baseline feature according to CPUs of cluster.

7- Now, VM can be restarted. Vmotion is possible between servers of different generation.

How to add a new server into a EVC cluster ?

Check and apply requirements to the new server (ESX version and BIOS option).
Add host to the cluster.
Done !

Practical example with Bull servers

EVC has been tested successfully on following servers:

R460 with Xeon 5100 series.
R460 E2 with Xeon 5500 series.

Both servers are running ESX 3.5 update 4.

A VM has been used to check that additional instructions have been masked by EVC.

(VM is running XP and CPU instruction are tested using CPU-Z software)

Before :

VM is running on R460 E2. R460 E2 is part of a non EVC cluster.

=> All instructions of Xeon 5500 series are available: SSE4.1/SSE4.2

After :

VM is still running on R460 E2 but R460 E2 is now a part of EVC cluster containing R460 server also. This cluster is set to ?Intel Core 2? EVC mode.

CPU-Z

=> Only instructions common to Xeon 5100 and Xeon 5500 are available, no SSE4.1/SSE4.2. Vmotion is now possible between R460 E2 and R460.

FAQ Article

— filed under: ns_ul_all