server virtualization

Enable software iSCSI adaptor on ESXi v5

Through SSH or on local console:
esxcli iscsi software set --enabled=true

blog server virtualization

HP ESXi 5.0 image license problem

The vmware.lic file is read only with a free ESXi key inside.

Execute on the command line of the particular ESX host:
esxcli software vib remove -n hp-esx-license --no-live-install
Connect with the VI client directly as root to the host and select by right-click to “shutdown” the host.

blog howto server windows

Errors with ntbackup caused by shadow copy.

Portions of FILENAME cannot be read. The backed up data is corrupt or incomplete. This file will not restore correctly.

I would receive this error numerous times in my ntbackup log.

In my case the C: drive would backup without any issue, but the D: drive would show these errors for every file in use.

My backup destination was a 2TB USB disk (G:) with a newer cluster size that is greater than 512. This is not really an issue for storing the backups. But somehow my shadow copy settings on the D: drive were using the G: backup drive as the place to store the shadow copies for the D: drive. And storing shadow copies on a drive with cluster size != 512 will fail. Note: on the G: drive itself shadow copies are disabled. Not needed for backup drive and will fail anyways because of above mentioned reason.

blog howto linux server

Ubuntu upgrade from 10.10 to 11.04 broke grub

After upgrading my Ubuntu server from 10.10 to 11.04 using the “do-release-upgrade” method, my system would no longer boot.
It was stuck on a GRUB shell. I downloaded the liveCD, booted from that and ran the excellent Boot-Repair utility. I had to install it using apt-get, so you need an internet connection. They should include it in the liveCD. I belive you can also download a Boot-Repair ISO file to boot from.

But the Boot-Repair tool worked great and fixed my GRUB in no time.
I see a lot of problems with GRUB after the upgrade. Maybe it’s a bug that affects people who, in the past, already upgraded from 9.x or 8.x to 10.x. So people who had GRUB v1 at some point in time.

linux server virtualization

GhettoVCB ESX(i) VM’s backup: E-mail logfile

UPDATE: GhettoVCB now has it’s own function to send a report mail. It only works on ESXi > 4.1.

I’m using an excellent script to backup my virtual machines called GhettoVCB. This script works perfect. I wanted to get the daily logfile in my e-mal inbox. I have ESXi (4.0) and i don’t know whether it can e-mail by itself, i think not. But i used a linux VM i am running, to pull the logfile and mail it to me. The linux VM is set up with sendmail and outgoing mailhost is configured. I you don’t have a linux VM, a windows VM or PC can work too. You can use wget for windows to pull the file (syntax would maybe change a little bit) and then blat to mail the file. I won’t go in detail about this.

I have altered the GhettoVCB script to make the backups & logiles use this date syntax “date +%F”. I find it easier to read for humans. You have to adjust this in the Cronjob that calls the ghettoVCB script, the logfile is given as a paramter.
In the script below, please adjust the youruser and yourpass to valid login credentials for browsing the datastore trough https. Usually the user you login with directly on the ESXi. Also adjust the you@domain.tld, place your e-mail address. Pretty simple & basic but it works excellent. Last point, adjust the 192.168.x.y to your ESXi’s IP address.

#grab esxi backup log file and mail it v0.1
#Note the WGET part should be 1 line, no line breaks
#Adjust the youruser & yourpass to valid credentials for logging in to ESXi (browse the datastore)
#Adjust the IP address (192.168.x.y) to your ESXi ip addresss
#In the wget line below, my logs were located in a folder called backup on a datastore called 750GBdisk2
#You could check the syntax of the link by browsing to your datastore and locate the backup log files then look at the addressbar
cd /tmp
wget "https://192.168.x.y/folder/backup/ghettoVCB-backup-$(date +\%F).log?dcPath=ha-datacenter&dsName=750GBdisk2" --user youruser --password yourpass --no-check-certificate
cat /tmp/ghettoVCB* | mail -s "Backup log ESXi4" you@domain.tld
rm -rf /tmp/ghettoVCB*

blog linux server virtualization

Installed ESXi 4 (with FTP & SSH enabled) & migrated VM’s from workstation

UDATE: This is an outdated article, since ESXi 4.1 you can enable SSH (remote & local) from the VI client through “security features” in “configuration”.

After my XenServer5.5 experience i decided to use VMWare ESXi, let’s be honest, the leader in the market.
First problem, installation would fail because no supported NIC was found. I tried several NIC’s such as 3Com 3C509 & Realtek RTL8139 but no luck.
I ordered an Intel PRO/1000 Desktop PCI adapter. I tested with the ESXi 4 CD and it was detected and supported.
Success, then I made a custom ESXi 4 CD with FTP & SSH enabled (trough custom oem.tgz) and installed ESXi. All went flawless.
Since it took some time for me to test the different NIC’s and finally order the Intel NIC. I had already installed VMWare Workstation evaluation on my desktop and had recreated my VM’s. Of course my PC was not 24/24 online so the website I’m writing this on was only at some times available.

After ESXi was installed and running, I used the VMWare Converter Standalone to convert my workstation VM’s to ESXi. The Windows VM went excellent.
The Ubuntu VM was installed through VMWare Workstation easy install mode, and on the ESXi it would boot fine but would not load the correct drivers for the NIC.
I was trying different things on it, but without NIC there wasn’t much i could do. I tried compiling the VMware tools and drivers but had failures.
I finally had to make a new VM with Ubuntu server fresh install and copied the files and exported the SQL database (on my VMWare workstation i could still boot the VM with network access). But after at least 6 hours of work ESXi was running with the 2 VM’s and the website was operational.

Since i use Windows 7 I had to follow a how-to because Vsphere Client fails on Windows 7 (have to copy system.dll and make a batch script). After updates from VMWare the latest Vsphere Client works fine on Windows 7.
Yesterday and today I finished up the Windows VM and moved PC’s in to the domain, copied back files & setup shares.

Now on to some last items, torrentflux-b4rt, truecrypt, moblock, RAS, …


Citrix XenServer 5: SR on fakeraid (onboard Intel Raid) and tape drive trough ISCSI to windows VM

EDIT: I eventually ran in to problems with my SR on fakeraid. So please don’t try this if you can’t afford to loose data.

In this article i want to describe my experiences with Citrix XenServer 5.0 Express and particular in combination with cheap PC hardware.

Let’s start of by describing the hardware.

Motherboard: Asus P5Q ( Intel® P45 chipset with ICH10R)
Processor: Intel Core2Quad Q9300 2.5Ghz 6MB cache FSB 1333 (Intel VT, XD)
Memory: Kingston Valueram 4GB DDR2-800 (kit of 2)
Storage: Samsung Spinpoint 750GB x2
Graphics: ATI Radeon X1300 PCI-E
Network: 3Com 3C905 PCI (onboard NIC’s not recognized)

I want to use the onboard RAID 1 to create a redundant storage repository for placing all the VM’s in.

When i started the setup by booting from the CD i only had the option to install on either of the disks.
Setup did not see or care about the RAID1 mirror i had already created trough BIOS tools.

After some research on the internet it seemed that most people would point out to use software raid instead of “Fakeraid” (this is how they call onboard firmware raid). But after researching more i found out that Citrix XenServer does not support software raid either, out of the box.
More research led me to the conclusion that fakeraid is easy to get working under linux, especially if the software (dmraid) is already present.
At least easy if the raidset is not your bootdisk.

I added an 80GB IDE drive i had lying around, to the system. I booted the CD and ran setup. I choose NOT to create a “Storage Repository” and installed on the 80GB IDE drive. After booting up the system i went to the console and typed the following command:

dmraid -ay

This gave me an output stating the raidset was active and was to be found at:


Note: XEN-MIRROR was the name i gave the raidset in the BIOS tools.

I then looked up the command for manually creating the the “Storage Repository” as LVM thus wiping the disks clean.

xe sr-create host-uuid=YOUR-OWN-UUID name-label=FAKERAID1 type=lvm content-type=user shared=false device-config-device=/dev/mapper/isw_ccjhgbjbcc_XEN-MIRROR

To get your UUID you can issue the following command:

xe host-list

After this i started my XenCenter on the windows management PC, and i saw the new SR (Storage Repository) when the right click menu, i choose this SR as default. Next i started making my virtual machines on it.

One particular challenging problem is, for instance, using a locally (to the XenServer) attached SCSI tape drive.
I read about PCI passtrough options called “pciback” and “pcifront” but this is only documented for the original old XenSource and highly unsupported by Citrix. I almost gave up, but stumbled upon a different approach. Since the tape drive is a SCSI device we can expose it as an ISCSI target with the help of some ISCSI software. I read a great articly about it here: (article from Daniel Lawson). I hope the author doesn’t mind me copy/pasting his article, but i would not want the knowledge to be lost.

Making a tape drive available to a guest via iSCSI

This is specifically for Citrix XenServer, although the principles will of course work in other Xen implementations

I recently had a scenario where I was replacing two Windows servers with XenServer guests. This was fine, but we needed a way to backup to the existing SCSI DDS4 DAT drive. After failing to make PCI passthrough work, I settled on the much nicer method of providing the tape drive via an iSCSI target on the XenServer Host (Dom0). Here is how I achieved this.

Note 1: This is totally unsupported by Citrix

Note 2: I’ve used the XenServer terminology “host” instead of Dom0, as this applies to the Citrix commercial implementation of Xen. It will probably work fine on OSS Xen, but you can just install the normal kernel dev packages and ignore the DDK stuff.

Note 3: This is for XenServer 4.1.0, but the principles are the same for previous versions. Just ensure you understand each step rather than following blindly.

* Download the Xen DDK from
* The DDK is an ISO containing a VM with a development environment. Import it to your Xen host, and start it.
* Download iscsitarget 0.4.14 (you MUST use this version, as the patch for rawio support will not cleanly apply to 0.4.15) from into your DDK VM.
* Download the patch for rawio support from into your DDK VM (assuming you’re saving it in /tmp)
* Now do the following:

yum install kernel-devel bison flex
tar -zxvf iscsitarget-0.4.14.tar.gz
cd iscsitarget-0.4.14
patch -p0 < /tmp/raw.p

* scp the entire iscsitarget-0.4.14 directory to your destination Xen host, and on that host (after enabling the base repo in /etc/yum.repos.d/CentOS-Base.repo) do:

yum install make gcc
cd iscsitarget-0.4.14
make install
mkdir /lib/modules/`uname -r`/kernel/iscsi
cp kernel/iscsi_trgt.ko /lib/modules/`uname -r`/kernel/iscsi
depmod -aq

The last three steps are required because make install will not copy the kernel module correctly outside the target environment.

* Now edit your /etc/ietd.conf and configure the tape as per the following example snippet (cat /proc/scsi/scsi for the correct HCIL values for your SCSI tape drive, this is an example only):

Lun 0 H=1,C=0,I=6,L=0,Type=rawio
Type 1

* Save and do /etc/init.d/iscsi-target start
* Modify /etc/sysconfig/iptables to allow port 3260 tcp from the IP addresses running the initiator.
* Attach to the target using the initiator of your choice.

More on this later, but i have it working, just not stable yet.