Categories
blog howto server virtualization

HP VAAI plugin missing from Vmware ESXi 5.1 U1 HP Customized edition sep 2013

Also read the updated EDIT section at the end.

Summary:
HP has left out the VAAI plugin in the September 2013 ISO’s for ESXi 5.1 and 5.5. People adding a hosts to a SAN (like a P2000G3) will have troubles without this plugin.
New VMFS-5 datastores that are created on a host that has the VAAI plugin will use ATS-Only locking mechanism for the datastore. Adding another host without this VAAI plugin will keep that host for correctly seeing the datastores.

Situation:
Server HP Proliant DL380 G6 freshly installed with VmWare ESXi 5.1 U1, using the “HP customized sep 2013 ISO”.
A couple of months ago I already had installed 2 other servers with VmWare ESXi 5.1 U1 using the “HP customized apr 2013 ISO”.
These 2 previosly installed hosts had been connected via ISCSI to an HP P2000G3 SAN with 2 LUN’s and 2 datastores were created without problems. Both hosts saw these datastores.

Problem:

The new vmware host would briefly show one or both datastores after a RESCAN HBA, but they disappeared after 2 seconds. Also the capacity values shown during those 2 seconds were wrong.

Troubleshooting:
I switched the ISCSI from the Intel NIC to the Broadcom NIC, but the problem remained the same.
I updated all the firmware on the server using the Servicepack for Proliant CD.
After that I further updated using the “VMware vSphere 5 Supplement for HP Service Pack for ProLiant” and the included HP SUM.
I checked the vmkernellog file on /tmp/scratch and saw these errors:
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: FSAts: 1304: Denying reservation access on an ATS-only vol 'P2000LUN12'
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: HBX: 1955: ATS-Only VMFS volume 'P2000LUN12' not mounted. Host does not support ATS or ATS initialization has failed.
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: HBX: 1968: Failed to initialize VMFS distributed locking on volume 51f29f9b-26f5059a-39c6-00237deeceda: Not supported
2013-11-28T13:53:58.536Z cpu11:9191)Vol3: 2359: Failed to get object 28 type 1 uuid 51f29f9b-26f5059a-39c6-00237deeceda FD 0 gen 0 :Not supported
2013-11-28T13:53:58.536Z cpu11:9191)WARNING: Fil3: 2492: Failed to reserve volume f530 28 1 51f29f9b 26f5059a 230039c6 daceee7d 0 0 0 0 0 0 0
2013-11-28T13:53:58.536Z cpu11:9191)Vol3: 2359: Failed to get object 28 type 2 uuid 51f29f9b-26f5059a-39c6-00237deeceda FD 4 gen 1 :Not supported
2013-11-28T13:53:58.581Z cpu11:9191)HBX: 707: Setting pulse [HB state abcdef02 offset 3440640 gen 1 stampUS 5568056524 uuid 529735be-0b3c273d-6396-18a90550fb2c jrnl drv 14.58] on vol 'P2000LUN12' failed: Not supported

So after researching on the internet I read up on VAAI what is responsible for the ATS locking of the ISCSI volumes. You could turn it off, but you have to do it on all the hosts and power all the VM’s down. This was not desirable and the other hosts worked fine. So I wanted to fix this one new host.
I logged in using SSH on the hosts and check the VAAI status with this command:
esxcfg-scsidevs -l | egrep "Display Name:|VAAI Status:"
On the working hosts it showed my ISCSI disks and next to VAAI status showed: supported.
On the troublesome host it showed the ISCSI disks and the VAAI status showed: unknown.

Solution:
VmWare support were no help, just asking me for a collection of all the logfiles and not reading my log excerpt containing the errors shown above. I gave them the logs, but 5 hours later they still hadn’t responded. That’s not what I call good support.
I read up on more of the VAAI stuff, and apparently it’s HP plugin in VmWare. So I looked around to find a download for it, to force it to update.
To my surprise I find it on this page: http://h18004.www1.hp.com/products/servers/software/vmware-esxi/driver_version.html
It’s listed as a component for the April 2013 version of the HP customized ISO, but they have now left it out in the September 2013 version.
So I download the HP VAAI plugin from here: http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState=idx=|swItem=MTX_30e09de4fc7e4498bfd9102a99

Download the zip file, extract it. Inside you’ll find another ZIP file with the word bundle in the filename. Upload this zip to your ESXi server’s /tmp/scratch directory (to do this: enable SSH using the client and use WinSCP or FastSCP).
Login to the SSH shell (using Putty) and execute:
esxcli software vib install -d /tmp/scratch/hp_vaaip_p2000_offline-bundle-210.zip
Reboot the host, and the problem is gone. You’re datastores now appear under storage tab for this host.

Notes:
Just a heads up, it seems HP has also left this HP VAAI plugin out of the ESXi 5.5 ISO’s.
No idea if they just forgot, or are intentionally doing it.

I have asked to close the VmWare support case stating I have found the solution myself some 9 hours ago, but they haven’t even looked at it, it remains open.

EDIT:
It seems HP removed the VAAI plugin on purpose, because of a bug with some RAID controllers. Read the advisory here at this LINK.
Since I don’t have these RAID controllers, I don’t have any problem enabling the plugin.
You can read more in this forum topic:
VAAI support with ESX 5.1U1 on P2000 G3 MSA

Categories
blog howto server virtualization

MONITOR PANIC: Unable to decompress PPN from swap slot for VM

VMWARE ESXi 5.1U1

My VM would power off without apparent reason.
Looking in to the logs this error appears.

MONITOR PANIC: Unable to decompress PPN from swap slot for VM

I believe the underlying storage (a single SATA disk in my case) to be at fault, or almost dying I guess.
I storage vmotion’ed the VM to another disk.

Categories
blog howto server windows

VMware image customization in progress

Recently I came across an issue with a virtual machine that was deployed from a template. After the newly created virtual machine was booted up it seemed that the generalization did not fully complete. The password was not changed and some other minor things were not as it should have been after full customization runs.

We then saw that the VM kept saying “VMware image customization in progress” on every reboot right before windows boots. If you already have configured and/or are using the VM and don’t want to start over, here’s what you can do.

Open regedit and find the key:
HKLM\SOFTWARE\VMWare,Inc.\Guest Cutomization\
or on a 64-bit system:
HKLM\SOFTWARE\Wow6432Node\VMWare,Inc.\Guest Cutomization\
Change the value for “CustomizationInProgress” to “0”.

Then navigate to the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
Change the value for “BootExecute” to the default of “autocheck autochk *”, you should just delete the “sysprepDecryptor.exe” part and the enters after it. Be careful or your system won’t boot.

You should probably check your SID and see if it’s unique, to make sure the customization did indeed change this. Altough duplicate SID’s should be not a real problem (according to Microsoft), don’t do it on your servers!
http://technet.microsoft.com/en-us/sysinternals/bb897417.aspx

PS: In my case the customization failed because sysprep had already been run 3 times (template was cloned from previous deployed machine) and apparently you can only do it 3 times unless you add the special option not to rearm the licensing state (which vmware doesn’t add when customizing).

PS2: If you manually did sysprep but got stuck because with error

A fatal error occurred while trying to sysprep the machine

because of the rearm failure, try this: http://itfunk.wordpress.com/2013/01/09/resetting-rearm-count-in-windows-7/
Put the following in a bat file called c:\delwpa.bat

reg load HKLM\MY_SYSTEM “%~dp0Windows\System32\config\system”
reg delete HKLM\MY_SYSTEM\WPA /f
reg unload HKLM\MY_SYSTEM

reboot and using F8 choose repair computer, choose keyboard, type password and choose command prompt. Find your correct drive could be C or D or higher (depends on reserved partitions), and execute the BAT file. Rearm state will be wiped, after reboot you will see not genuine but you can rearm again 3 times now.

Categories
blog howto network server windows

Backup domain controller sync issues KRB_AP_ERR_MODIFIED 0x80090322 target principal name incorrect

My case:
1x Windows 2008 Small Business Server (named: SBS2008)
1x Windows 2008 R2 standard on off-site location (named: TS2008) BACKUP DOMAIN CONTROLLER & GC
Connection between the 2 servers was lost for nearly 3 months.
Replication would only work from SBS2008 to TS2008 but not from TS2008 to SBS2008.
I couldn’t view the shares on \\SBS2008 from the console on TS2008, i received the error “The target principal name is incorrect”. On SBS2008 I could view the shares on TS2008.
In the eventlog there were errors:

The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.

Sites:
CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domein,DC=local

The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.

Directory partition:
DC=domain,DC=local

There is insufficient site connectivity information for the KCC to create a spanning tree replication topology. Or, one or more directory servers with this directory partition are unable to replicate the directory partition information. This is probably due to inaccessible directory servers.

All directory servers in the following site that can replicate the directory partition over this transport are currently unavailable.

Site:
CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=local
Directory partition:
DC=domain,DC=local
Transport:

CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=domain,DC=local

The File Replication Service is having trouble enabling replication from SBS2008 to TS2008 for c:\windows\sysvol\domain using the DNS name SBS2008.domein.local. FRS will keep retrying.
Following are some of the reasons you would see this warning.

The session setup from the computer SBS2008 failed to authenticate. The name(s) of the account(s) referenced in the security database is SBS2008$. The following error occurred:
Access is denied.

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server sbs2008$. The target name used was E3514xx-xxxxxxxxxxxxxxx/yyyyyyyyyyyyyyy/domain.local@domain.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please …

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server sbs2008$. The target name used was DNS/sbs2008.domain.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please …

Demote and the promote for TS2008 would not work without forcing and doing a lot of NTDS cleanup on the PDC. So this was my last resort.

1st reboot -> did not work.
Doing a lot of searching and looking up the SPN for SBS2008 on both DC’s did not show any differences.

Luckily I found an older KB from microsoft on how to reset the kerberos secure channel between two DC’s.
http://support.microsoft.com/kb/288167

I had to disable the KDC (Kerberos Key Distribution Center) service on TS2008 and then reboot.
Immediately after reboot I noticed I could browse the shares on the SBS2008 without error.
This is because TS2008 was no longer supplied Kerberos tickets itself but requesting them from SBS2008.
Now I opened an elevated command prompt and forced a sync of all replica partitions and triggered the KCC checker.
repadmin /syncall /ade
repadmin /kcc

To check the replication backlog queue use:
repadmin /queue

After replication was succesful I put the KDC service back to automatic and started it. Problem solved.

If you can’t get replication working yet, you’ll need these extra steps.
klist /purge
netdom resetpwd /server:sbs2008 /userd:domain\Administrator /passwordd:* (the * will make it prompt for password).

Also you might need to check your DNS settings and put the IP adres of SBS2008 as primary DNS IP on the NIC of TS2008.

Other helpful information:
http://support.microsoft.com/kb/2090913

Categories
blog howto windows

Excel has problems with space as thousand seperator

Change the system settings to use a . instead (configuration panel, region and local settings, more settings, number).

For existing files, use search and replace to delete spaces. If still left justified multiply all with the number 1 using paste special, see link below.

http://answers.microsoft.com/en-us/office/forum/office_2007-excel/space-as-thousand-separator-in-a-csv-file/90c4a714-7a4e-4deb-ac89-0bdd2fcadc40?msgId=8b3ef2f1-7170-4720-95bf-86263c5add7e

Categories
blog howto windows

Acrobat X pro as part of CS6 suite does not start 30days after installation

Acrobat X pro as part of CS6 suite does not start 30days after installation

Use solution 2.

http://helpx.adobe.com/creative-suite/kb/acrobat-failed-launch-30-days.html

Does not work for regular standalone Adobe X problems, only when installed (and activated) as part of CS6 suite, and stops to work after 30 days.

1) Download: http://helpx.adobe.com/creative-suite/kb/acrobat-failed-launch-30-days/_jcr_content/main-pars/download/file.res/Acrofix.zip
2) Unzip to a convenient location
3) Open a command prompt as administrator (in start menu search for cmd and right click run as administrator)
4) go to the location where you unzipped
cd c:\temp\adobefix\
5) Execute the executable
Acrofix.exe
6) Exit Code: 0 means succesfully patched!
7) Try Adobe X Pro again.

Categories
blog howto server virtualization

VMware ESXi 5.1 on USB stick won’t boot Proliant DL380 G5

Installation using the cdrom was succesful but after restarting the server won’t boot from the USB stick.
Make sure you set the correct BIOS options to allow to boot from USB.
bios_boot_order

bios_usb_enable

ESXIi formatwithmbr runweasel

If still doesn’t boot than it probably has to do with GPT/MBR formatting of the USB stick done by VMWARE.
You need to boot from the ESXi install CD again and right after you press enter to choose “ESXi5.1 installer ISO …” you see in the lower righthand corner the text “Shift + o” press this key combination (shift and the letter o). Now you see the text “runweasel” remove any chars after this, type a space and then “formatwithmbr”.

Now install as normal but now VMware should format your USB stick as MBR instead of GPT and you should be able to boot from it after the install finishes.

sources:
http://vmtoday.com/2012/09/esxi-5-wont-boot-from-usb/
http://communities.vmware.com/thread/430852?start=0&tstart=0
http://communities.vmware.com/message/1824957#1824957
http://community.spiceworks.com/topic/247715-sandisk-cruzer-fit-esxi-5

Categories
blog exchange howto server

Exchange 2010 Outlook Anywhere proxy security certificate wildcard *.domain.com

When you use a wildcard certificate on your Exchange 2007 or 2010 environment you can receive an error on outlook:
There is a problem with the proxy servers's security certificate.
The name on the security certificate is invalid or does not match the name of the target site mail.domain.com
Outlook is unable to connect to the proxy server (Error code 0)

Error1
When using internally your outlook may connect just fine using the normal RPC-TCP method.
This happens because outlook is checking the name on the certificate for mutual authentication to ensure your are connecting to the right server. Outlook gets this information from the autodiscover service.
error2
If you manually change the value to msstd:*.domain.com it works, but the autodiscover will put the other value back in a matter of minutes. Autodiscover assumes a value equal to the external name set on your CAS server (in my case mail.domain.com) and uses this.

To override this behavior use the following exchange shell command:
Set-OutlookProvider EXPR -CertPrincipalName msstd:*.domain.com
After you adjust this, you need to restart the “world wide web publishing” service, because of caching.

You could also disable this “Mutual authentication”, but it’s a good security feature, so I wouldn’t.
Set-OutlookProvider EXPR -CertPrincipalName none
Remember to restart the W3P service.

Be aware that when you set $null instead of none Exchange will go back to default behavior and use the external name from the CAS server.

reference: http://blogs.technet.com/b/umutg/archive/2011/01/31/all-about-set-outlookprovider.aspx

Categories
blog howto network server windows

RDWeb shows no icons with Internet Explorer 10 TSWeb (server side fix)

RDWeb or TSWeb is the Microsoft Remote Desktop service web access page from Windows 2008 or Windows 2008 R2.

When using Internet Explorer 10 to browse to an RDWeb (>=2008 R2) or TSWeb (=2008) the icons for the published apps are missing or blank and WKSPRT.EXE (provides SSO) does not load. You have to click the compatibility icon in the browser and then it does work.

If you want to make sure that users don’t have to click compatibility mode, you can send the header to emulate IE9 from server side, so Internet Explorer automatically uses the right mode.

You can choose between 2 options (I prefer option2, easier and less work):

1) Edit all the *.aspx pages (except logoff.aspx) under the %windir%\Web\RDWeb\Pages\en-US\ folder.
Add <META HTTP-EQUIV="X-UA-COMPATIBLE" CONTENT="IE=9"> right below the lines <html><head id=”Head1″ runat=”server”>
screenshot source code

2) Add the header in IIS for the RDweb so that it is automatically added to all pages served from RDweb.
Open IIS Management under Administrative tools in the start menu.
Expand the Default Web Site using the plus sign next to it. Click on RDWeb and then in the middle pane double click on HTTP Response Headers (under the IIS category). In the list with headers right click on the empty space and click Add. Under name fill in: X-UA-Compatible and under value fill in: IE=9
IIS-add-header-compatibility-IE9

Categories
blog exchange howto server windows

Exchange missing public folder database after adsiedit changes

I had to use Adsiedit.msc to manually remove the Public folder database on an Exchange 2007 i was trying to uninstall. I already had a new public folder database with all replica’s present on my Exchange 2010 server, so i was confident in removing the older Exchange 2007 PF database through Adsiedit. In the meantime I also removed the “First Administrative group” since this was left from Exchange 2003.

I then proceeded to uninstall Exchange 2007 without further issues.
I was surprised and horrified to find the Exchange 2010 Public Folder database missing in the “Database management tab” on “Organization – Mailbox” in the Exchange 2010 Management console. I tried various things, such as:
1) Restart information store
2) Recreate PF DB with exact same name, but this error saying it already had a DB with that name.

I ran the Best Practices Analyzer and it told me “Site folder server deleted”. I clicked on the help and it showed me how to fix this.

Open an Active Directory editor, such as ADSI Edit.
Locate the public folder information store that you want to designate as the Site Folder Server. For Exchange Server 2000 through Exchange Server 2007, expand the following nodes in the Configuration container:
CN=Configuration,DC=,DC=com, CN=Services, CN=Microsoft Exchange, CN=, CN=Administrative Groups, CN= CN=Servers, CN=, CN=InformationStore, CN=
For Exchange Server 2010, expand the following nodes in the Configuration container:
CN=Configuration,DC=,DC=com, CN=Services, CN=Microsoft Exchange, CN=, CN=Administrative Groups, CN=Exchange Administrative Group (FYDIBOHF23SPDLT), CN=Databases
In the right pane, right-click CN=, and then click Properties.
In the Attributes field, scroll down and select the distinguishedName attribute.
Click Edit, and then copy the entire attribute to the Clipboard.
Expand the Configuration container, and then expand CN=Configuration,CN=,CN=com, CN=Services, CN=Microsoft Exchange, CN=, CN=Administrative Groups
Right-click the administrative group you want to modify, and then click Properties.
In the Attributes field, scroll down and select the siteFolderServer attribute.
Click Edit, and then paste the value for the distinguishedName attribute into the Value field.
Double-check the contents of the Value field to ensure the paste was performed correctly, and then click OK to save the change.
Click OK to close the Administrative Group properties.
Exit the Active Directory editor and restart the Microsoft Exchange Server Information Store service on all Exchange Server computers in the site for the change to take effect.

I reran the BPA and the error was gone, but my PF database was still missing.
After some research on the internet I came by the solution thanks to “BFTech Impressions”.
Specifically in my case the “msExchOwningPFTree” attribute was empty on my PF database container and needed to be filled with the value from the “distinguishedName” attribute from the Public Folder container uner “Folder Hierarchies”, the first 2 steps in my case were not needed, these were still present.

Here is the link to the article:
http://blog.bruteforcetech.com/archives/766
Please click on the links for detailed instructions and screenshots.

These are his instructions I quote here so that in the case the original disappears the information is not lost.

Here are the instructions to fix it:
Open ADSI Edit, connect to a Domain Controller, change the context to Configuration.

Create the Folder Hierarchies under the Exchange Administrative Group
Navigate to Configuration ⇒ Services ⇒ Microsoft Exchange ⇒ [your organization] ⇒Administrative Groups ⇒ [your administrative group]
Right click on your administrative group and select New Object
Select msExchContainer as class and click Next
Enter the following as value: Folder Hierarchies, click Next, Finish

Create the Public Folders Tree Object
Right click Folder Hierarchies and select New Object
Select msExchPFTree as class, click Next
Enter the following as value: Public Folders, click Next
Click on More Attributes button, drop down the “select a property to view” list, select msExchPFTreeType and set the attribute to 1 (it should populate into the value field).
Click OK, Finish

Populate the msExchOwningPFTreeBL attribute object of the PF Store
(Tell the Public Folder database where to find the new folder hierarchy you just created)
Double click the newly created “Public Folders” object
Double click distinguishedName, copy the value to the clipboard, click Cancel
Exchange 2007: open properties of Configuration ⇒ Services ⇒ Microsoft Exchange ⇒ [your organization] ⇒ Administrative Groups ⇒ [your administrative group]⇒ Servers ⇒ [your server] ⇒ Information Store
Exchange 2010: open properties of Configuration ⇒ Services ⇒ Microsoft Exchange ⇒ [your organization] ⇒ Administrative Groups ⇒ [your administrative group] ⇒ Databases ⇒ [your Public Folder database]
Double click the msExchOwningPFTree attribute, paste the value that was copied to the clipboard in step 2
Click OK twice