blog howto server virtualization

HP VAAI plugin missing from Vmware ESXi 5.1 U1 HP Customized edition sep 2013

Also read the updated EDIT section at the end.

HP has left out the VAAI plugin in the September 2013 ISO’s for ESXi 5.1 and 5.5. People adding a hosts to a SAN (like a P2000G3) will have troubles without this plugin.
New VMFS-5 datastores that are created on a host that has the VAAI plugin will use ATS-Only locking mechanism for the datastore. Adding another host without this VAAI plugin will keep that host for correctly seeing the datastores.

Server HP Proliant DL380 G6 freshly installed with VmWare ESXi 5.1 U1, using the “HP customized sep 2013 ISO”.
A couple of months ago I already had installed 2 other servers with VmWare ESXi 5.1 U1 using the “HP customized apr 2013 ISO”.
These 2 previosly installed hosts had been connected via ISCSI to an HP P2000G3 SAN with 2 LUN’s and 2 datastores were created without problems. Both hosts saw these datastores.


The new vmware host would briefly show one or both datastores after a RESCAN HBA, but they disappeared after 2 seconds. Also the capacity values shown during those 2 seconds were wrong.

I switched the ISCSI from the Intel NIC to the Broadcom NIC, but the problem remained the same.
I updated all the firmware on the server using the Servicepack for Proliant CD.
After that I further updated using the “VMware vSphere 5 Supplement for HP Service Pack for ProLiant” and the included HP SUM.
I checked the vmkernellog file on /tmp/scratch and saw these errors:
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: FSAts: 1304: Denying reservation access on an ATS-only vol 'P2000LUN12'
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: HBX: 1955: ATS-Only VMFS volume 'P2000LUN12' not mounted. Host does not support ATS or ATS initialization has failed.
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: HBX: 1968: Failed to initialize VMFS distributed locking on volume 51f29f9b-26f5059a-39c6-00237deeceda: Not supported
2013-11-28T13:53:58.536Z cpu11:9191)Vol3: 2359: Failed to get object 28 type 1 uuid 51f29f9b-26f5059a-39c6-00237deeceda FD 0 gen 0 :Not supported
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: Fil3: 2492: Failed to reserve volume f530 28 1 51f29f9b 26f5059a 230039c6 daceee7d 0 0 0 0 0 0 0
2013-11-28T13:53:58.536Z cpu11:9191)Vol3: 2359: Failed to get object 28 type 2 uuid 51f29f9b-26f5059a-39c6-00237deeceda FD 4 gen 1 :Not supported
2013-11-28T13:53:58.581Z cpu11:9191)HBX: 707: Setting pulse [HB state abcdef02 offset 3440640 gen 1 stampUS 5568056524 uuid 529735be-0b3c273d-6396-18a90550fb2c jrnl drv 14.58] on vol 'P2000LUN12' failed: Not supported

So after researching on the internet I read up on VAAI what is responsible for the ATS locking of the ISCSI volumes. You could turn it off, but you have to do it on all the hosts and power all the VM’s down. This was not desirable and the other hosts worked fine. So I wanted to fix this one new host.
I logged in using SSH on the hosts and check the VAAI status with this command:
esxcfg-scsidevs -l | egrep "Display Name:|VAAI Status:"
On the working hosts it showed my ISCSI disks and next to VAAI status showed: supported.
On the troublesome host it showed the ISCSI disks and the VAAI status showed: unknown.

VmWare support were no help, just asking me for a collection of all the logfiles and not reading my log excerpt containing the errors shown above. I gave them the logs, but 5 hours later they still hadn’t responded. That’s not what I call good support.
I read up on more of the VAAI stuff, and apparently it’s HP plugin in VmWare. So I looked around to find a download for it, to force it to update.
To my surprise I find it on this page:
It’s listed as a component for the April 2013 version of the HP customized ISO, but they have now left it out in the September 2013 version.
So I download the HP VAAI plugin from here:|swItem=MTX_30e09de4fc7e4498bfd9102a99

Download the zip file, extract it. Inside you’ll find another ZIP file with the word bundle in the filename. Upload this zip to your ESXi server’s /tmp/scratch directory (to do this: enable SSH using the client and use WinSCP or FastSCP).
Login to the SSH shell (using Putty) and execute:
esxcli software vib install -d /tmp/scratch/
Reboot the host, and the problem is gone. You’re datastores now appear under storage tab for this host.

Just a heads up, it seems HP has also left this HP VAAI plugin out of the ESXi 5.5 ISO’s.
No idea if they just forgot, or are intentionally doing it.

I have asked to close the VmWare support case stating I have found the solution myself some 9 hours ago, but they haven’t even looked at it, it remains open.

It seems HP removed the VAAI plugin on purpose, because of a bug with some RAID controllers. Read the advisory here at this LINK.
Since I don’t have these RAID controllers, I don’t have any problem enabling the plugin.
You can read more in this forum topic:
VAAI support with ESX 5.1U1 on P2000 G3 MSA

blog howto server virtualization

MONITOR PANIC: Unable to decompress PPN from swap slot for VM


My VM would power off without apparent reason.
Looking in to the logs this error appears.

MONITOR PANIC: Unable to decompress PPN from swap slot for VM

I believe the underlying storage (a single SATA disk in my case) to be at fault, or almost dying I guess.
I storage vmotion’ed the VM to another disk.

blog howto server windows

VMware image customization in progress

Recently I came across an issue with a virtual machine that was deployed from a template. After the newly created virtual machine was booted up it seemed that the generalization did not fully complete. The password was not changed and some other minor things were not as it should have been after full customization runs.

We then saw that the VM kept saying “VMware image customization in progress” on every reboot right before windows boots. If you already have configured and/or are using the VM and don’t want to start over, here’s what you can do.

Open regedit and find the key:
HKLM\SOFTWARE\VMWare,Inc.\Guest Cutomization\
or on a 64-bit system:
HKLM\SOFTWARE\Wow6432Node\VMWare,Inc.\Guest Cutomization\
Change the value for “CustomizationInProgress” to “0”.

Then navigate to the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
Change the value for “BootExecute” to the default of “autocheck autochk *”, you should just delete the “sysprepDecryptor.exe” part and the enters after it. Be careful or your system won’t boot.

You should probably check your SID and see if it’s unique, to make sure the customization did indeed change this. Altough duplicate SID’s should be not a real problem (according to Microsoft), don’t do it on your servers!

PS: In my case the customization failed because sysprep had already been run 3 times (template was cloned from previous deployed machine) and apparently you can only do it 3 times unless you add the special option not to rearm the licensing state (which vmware doesn’t add when customizing).

PS2: If you manually did sysprep but got stuck because with error

A fatal error occurred while trying to sysprep the machine

because of the rearm failure, try this:
Put the following in a bat file called c:\delwpa.bat

reg load HKLM\MY_SYSTEM “%~dp0Windows\System32\config\system”
reg delete HKLM\MY_SYSTEM\WPA /f
reg unload HKLM\MY_SYSTEM

reboot and using F8 choose repair computer, choose keyboard, type password and choose command prompt. Find your correct drive could be C or D or higher (depends on reserved partitions), and execute the BAT file. Rearm state will be wiped, after reboot you will see not genuine but you can rearm again 3 times now.

blog howto server virtualization

VMware ESXi 5.1 on USB stick won’t boot Proliant DL380 G5

Installation using the cdrom was succesful but after restarting the server won’t boot from the USB stick.
Make sure you set the correct BIOS options to allow to boot from USB.


ESXIi formatwithmbr runweasel

If still doesn’t boot than it probably has to do with GPT/MBR formatting of the USB stick done by VMWARE.
You need to boot from the ESXi install CD again and right after you press enter to choose “ESXi5.1 installer ISO …” you see in the lower righthand corner the text “Shift + o” press this key combination (shift and the letter o). Now you see the text “runweasel” remove any chars after this, type a space and then “formatwithmbr”.

Now install as normal but now VMware should format your USB stick as MBR instead of GPT and you should be able to boot from it after the install finishes.