Also read the updated EDIT section at the end.
Summary:
HP has left out the VAAI plugin in the September 2013 ISO’s for ESXi 5.1 and 5.5. People adding a hosts to a SAN (like a P2000G3) will have troubles without this plugin.
New VMFS-5 datastores that are created on a host that has the VAAI plugin will use ATS-Only locking mechanism for the datastore. Adding another host without this VAAI plugin will keep that host for correctly seeing the datastores.
Situation:
Server HP Proliant DL380 G6 freshly installed with VmWare ESXi 5.1 U1, using the “HP customized sep 2013 ISO”.
A couple of months ago I already had installed 2 other servers with VmWare ESXi 5.1 U1 using the “HP customized apr 2013 ISO”.
These 2 previosly installed hosts had been connected via ISCSI to an HP P2000G3 SAN with 2 LUN’s and 2 datastores were created without problems. Both hosts saw these datastores.
Problem:
The new vmware host would briefly show one or both datastores after a RESCAN HBA, but they disappeared after 2 seconds. Also the capacity values shown during those 2 seconds were wrong.
Troubleshooting:
I switched the ISCSI from the Intel NIC to the Broadcom NIC, but the problem remained the same.
I updated all the firmware on the server using the Servicepack for Proliant CD.
After that I further updated using the “VMware vSphere 5 Supplement for HP Service Pack for ProLiant” and the included HP SUM.
I checked the vmkernellog file on /tmp/scratch and saw these errors:
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: FSAts: 1304: Denying reservation access on an ATS-only vol 'P2000LUN12'
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: HBX: 1955: ATS-Only VMFS volume 'P2000LUN12' not mounted. Host does not support ATS or ATS initialization has failed.
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: HBX: 1968: Failed to initialize VMFS distributed locking on volume 51f29f9b-26f5059a-39c6-00237deeceda: Not supported
2013-11-28T13:53:58.536Z cpu11:9191)Vol3: 2359: Failed to get object 28 type 1 uuid 51f29f9b-26f5059a-39c6-00237deeceda FD 0 gen 0 :Not supported
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: Fil3: 2492: Failed to reserve volume f530 28 1 51f29f9b 26f5059a 230039c6 daceee7d 0 0 0 0 0 0 0
2013-11-28T13:53:58.536Z cpu11:9191)Vol3: 2359: Failed to get object 28 type 2 uuid 51f29f9b-26f5059a-39c6-00237deeceda FD 4 gen 1 :Not supported
2013-11-28T13:53:58.581Z cpu11:9191)HBX: 707: Setting pulse [HB state abcdef02 offset 3440640 gen 1 stampUS 5568056524 uuid 529735be-0b3c273d-6396-18a90550fb2c jrnl drv 14.58] on vol 'P2000LUN12' failed: Not supported
So after researching on the internet I read up on VAAI what is responsible for the ATS locking of the ISCSI volumes. You could turn it off, but you have to do it on all the hosts and power all the VM’s down. This was not desirable and the other hosts worked fine. So I wanted to fix this one new host.
I logged in using SSH on the hosts and check the VAAI status with this command:
esxcfg-scsidevs -l | egrep "Display Name:|VAAI Status:"
On the working hosts it showed my ISCSI disks and next to VAAI status showed: supported.
On the troublesome host it showed the ISCSI disks and the VAAI status showed: unknown.
Solution:
VmWare support were no help, just asking me for a collection of all the logfiles and not reading my log excerpt containing the errors shown above. I gave them the logs, but 5 hours later they still hadn’t responded. That’s not what I call good support.
I read up on more of the VAAI stuff, and apparently it’s HP plugin in VmWare. So I looked around to find a download for it, to force it to update.
To my surprise I find it on this page: http://h18004.www1.hp.com/products/servers/software/vmware-esxi/driver_version.html
It’s listed as a component for the April 2013 version of the HP customized ISO, but they have now left it out in the September 2013 version.
So I download the HP VAAI plugin from here: http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState=idx=|swItem=MTX_30e09de4fc7e4498bfd9102a99
Download the zip file, extract it. Inside you’ll find another ZIP file with the word bundle in the filename. Upload this zip to your ESXi server’s /tmp/scratch directory (to do this: enable SSH using the client and use WinSCP or FastSCP).
Login to the SSH shell (using Putty) and execute:
esxcli software vib install -d /tmp/scratch/hp_vaaip_p2000_offline-bundle-210.zip
Reboot the host, and the problem is gone. You’re datastores now appear under storage tab for this host.
Notes:
Just a heads up, it seems HP has also left this HP VAAI plugin out of the ESXi 5.5 ISO’s.
No idea if they just forgot, or are intentionally doing it.
I have asked to close the VmWare support case stating I have found the solution myself some 9 hours ago, but they haven’t even looked at it, it remains open.
EDIT:
It seems HP removed the VAAI plugin on purpose, because of a bug with some RAID controllers. Read the advisory here at this LINK.
Since I don’t have these RAID controllers, I don’t have any problem enabling the plugin.
You can read more in this forum topic:
VAAI support with ESX 5.1U1 on P2000 G3 MSA
