RHEL 2.1 P2V Fails at 99%

I have a physical RHEL 2.1 box that I am trying to P2V to an ESXi host using the VMware vCenter Converter Standalone 4.3 app and it is failing at 98% and 99%. The log is showing the following errors:

[#2] [2011-02-11 16:45:39.209 07840 info 'task-1'] Worker CloneTask updates, state: 1, percentage: 98, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:04.379 05412 info 'task-2'] Remote Helper VM is reconfiguring, data clone is finished
[#2] [2011-02-11 16:46:04.379 05412 info 'task-2'] Volume-based cloning dlinux5.XX.XXX--> dlinux5 updates, state: 1, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:04.379 05412 info 'task-2'] CloneTask updates, state: 1, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:04.379 07840 info 'task-1'] WorkerCloneTask: Remote Helper VM is reconfiguring, data clone is finished
[#2] [2011-02-11 16:46:04.379 07840 info 'task-1'] Worker CloneTask updates, state: 1, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:16.489 05412 info 'task-2'] Volume-based cloning dlinux5.XX.XXX--> dlinux5 updates, state: 4, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:16.489 05412 info 'task-2'] Generating helperVM task bundle for task with id="task-1".
[#2] [2011-02-11 16:46:16.519 05412 info 'task-2'] Retrieving helper VM log bundle to "C:\Windows\TEMP\vmware-temp\vmware-SYSTEM\helperVM-task-1-sljnzvkf.zip".
[#2] [2011-02-11 16:46:16.619 05412 info 'task-2'] Bundle successfully retrieved to "C:\Windows\TEMP\vmware-temp\vmware-SYSTEM\helperVM-task-1-sljnzvkf.zip".
[#2] [2011-02-11 16:46:16.619 05412 info 'task-2'] powering off vm after linux p2v ...
[#2] [2011-02-11 16:46:16.619 05412 info 'task-2'] Reusing existing VIM connection to XXXXXXesx2.XX.XXX
[#2] [2011-02-11 16:46:18.099 05412 info 'task-2'] power off vm succeeded
[#2] [2011-02-11 16:46:18.099 05412 info 'task-2'] Reusing existing VIM connection to XXXXXXesx2.XX.XXX
[#2] [2011-02-11 16:46:18.579 05412 info 'task-2'] successfully reconfigured target vm
[#2] [2011-02-11 16:46:18.579 05412 info 'task-2'] CloneTask updates, state: 4, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:18.579 05412 info 'task-2'] CloneTask failed
[#2] [2011-02-11 16:46:18.579 05412 error 'App'] Task failed:

And earlier in the log:
#2] [2011-02-11 15:44:34.036 05412 error 'App'] Found dangling SSL error: [0] error:00000001:lib(0):func(0):reason(1)

And again:
[#1] [2011-02-11 15:43:08.256 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.266 03284 info 'App'] [,0] Disk number 1 has been skipped because of errors while reading partition table
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Disk number 2 has been skipped because of errors while reading partition table
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Disk number 1 has been skipped because of errors while reading dynamic disks header or LDM database is corrupted
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Disk number 2 has been skipped because of errors while reading dynamic disks header or LDM database is corrupted
[#1] [2011-02-11 15:43:08.286 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.296 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.296 03284 warning 'App'] [MoveActiveDiskIfNeeded] GetFirstBootDisk failed, mntapi error: 176

The great thing about having a VM is that you can snapshot it. DO IT. I destroyed around a dozen copies of my VM playing with this trying to get it right.

I have already manually run an fsck against the physical machine before the P2V attempts so I am not sure what is wrong but when I attempt to power up the virtual machine I get a nice “Error Loading Operating System”. Looks like a bootloader issue, possibly when the VM Reconfiguration Job was going something bad happened. No worries, I just dropped in a Fedora 13 DVD and selected the “Rescue Installed System” option. Follow the prompts (networking not required) and then “chroot /mnt/sysimage”. From here I can see the data on my VM and it all looks good. Now I go to /boot/grub and notice that I only have 4 files:
1. Grub.conf
2. Device.map
3. Menu.1sl
4. Splash.xpm.gz
Looks like the boot loader files are gone. Now I need to somehow recreate them. To do this you need to do the following. First, figure out what your partitions look like using the “fdisk –l” command. Then using that information open grub by typing “grub”.

Once inside the grub shell type in “root” and if you have the same issue that I had you should see the following returned value: “(fd0) : filesystem type unknown, partition type 0x0” We need to change that to our existing boot partition which in most cases is ‘hd0,0’. To do that type in “root (hd0,0)” and hit enter. If you get lost you can hit tab to see your options. From there you need to execute the changes by typing “setup (hd0,0)”. You should see it execute and give some output. After it is complete you can “quit” and then do an “ls” on the /boot/grub directory and you should see that grub has added several files.

Now you might need to convert the linux configuration from IDE to SCSI based. To do this you need to modify all entries that are “hdXY” where X is a letter starting with a and Y is a number starting with 1. You need to modify these to become sdXY. As an example hda1 becomes sda1. You need to do this in the following files:
1. /boot/grub.conf
2. /boot/device.map
3. /etc/fstab
Note: for some weird reason your 2nd drive’s partitions may show up as in the /etc/fstab as hdc1 instead on hdb1 as I would have expected. This will cause errors upon boot if you don’t change that to sdb1 instead of sdc1. Not sure what causes that initial value to be wrong.

Now you need to add scsi support. Open the /etc/modules.conf file and edit the following things:
Find ‘alias ethX module’ entires and replace the value of the module to “pcnet32”. This is the network adapter that you will be using in ESX(i).
Now find ‘alias scsi_hostadapter’ and if it does not exist add the following:
“alias scsi_hostadapter BusLogic”
For multiple controllers I assume you would have to add another line per controller like:
“alias scsi_hostadapter1 BusLogic”

After this you need to rebuild the ramdisk image. Locate the .img file in the /boot directory and make a note of the file name. Now you want to run the following command:
Mkinitrd –v –f /boot/initrd-X.X.X-Y.X.X.img X.X.X-Y.X.X where the X and Y symbolize the actual values on your system. For mine the actual command was
“mkinitrd –v –f /boot/initrd-2.4.9.0-e.34.img 2.4.9.0-e.34”
After that runs you should be able to reboot and it should work.

Thanks to everybody who posted bits and pieces of the fixes on various blogs and papers. Hope this helps somebody else so you don't have to do all the research and tweaking that I did.

Bind DNS on CentOS 5

Here is my quick and dirty cheatsheet on how to get bind working on CentOS5.
1. Make sure that iptables is not blocking your DNS queries ("service iptables stop" will disable the firewall)
2. Install Bind "yum install bind*"
3. Copy example configs to working directories:
a. "cp /usr/share/doc/bind-x.x.x/sample/etc /var/named/chroot/etc"
b. "cp /usr/share/doc/bind-x.x.x/sample/var /var/named/chroot/var/named"
4. Modify the named.conf in /var/named/chroot/etc and remove the un-needed views (I only need the internal view)
5. Modify the named.conf to have the correct zone information
6. Run /usr/sbin/dns-keygen and copy the generated key to the named.conf file under the ddns_key section.
7. Copy your my.internal.zone.db from /var/named/chroot/var/named or rename it to your correct zone name
8. Add your A records to the new zone.db file
9. Automatically start bind on boot
a. chkconfig named --add
b. chkconfig on level 2,3,4,5

Transparent Windows Command Prompt

So one of the things that I really wished Windows 7 had was a transparent command prompt. I have been using it on Linux and gotten spoiled having it. Today I found one at http://www.askvg.com/get-glass-cmd-command-prompt-in-windows-vista-and-7/ that seems to work pretty well on Windows 7 x64.

VMware HA

You will need this if you have environments with both ESXi and ESX.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006541

Collecting from ESX/ESXi with VMware vCM 5.4

If you have VMware vCM (Configuresoft ECM, EMC Ionix SCM) and want to collect from ESX/ESXi you have to create a local user on the ESXi host and give that user Administrator permissions. That can be quite a daunting task if you have more than a handful of machines so I wrote the below powershell program that allows interactive use or automated use with a .CSV file.


---Start Script
Function NextUser
{
if ($newUser)
{
Write-host "Do you want to specify a new service account username or password? [y],[n]" -Fore Green
$continueUser = read-host
if ($continueUser -eq "y")
{
DoIt
}
if ($continueUser -eq "n")
{
DoItUsers
}
}
}

Function ExistingUsers
{
if ($newUser)
{
Write-host "Do you want to specify a new service account username or password? [y],[n]" -Fore Green
$continueUser = read-host
if ($continueUser -eq "n")
{
DoItUsers
}
if ($continueUser -eq "y")
{
DoIt
}
}
if (!$newUser)
{
DoIt
}
}

Function DoIt
{
write-host "Please enter the username that you would like to use for the vCM Service Account:" -Fore Green
$newUser = read-host
write-host "Please enter the password that will be used for the service account:" -Fore Green
$newPass = read-host
write-host "Ok, I am going to add $newUser with a password of $newPass to $connectedServer" -Fore Green
Start-Sleep -seconds 3
New-VMHostAccount -ID $newUser -password $newPass -Description "vCM Service Account"
write-host "Ok, I am now adding administrator permissions to the account we just created" -Fore Green
New-VIPermission -Role admin -Principal $newUser -Entity $connectedServer
write-host "Congratulations, you have added the new vCM service account successfully." -Fore Green
write-host "Would you like to configure another host? [y],[n]" -Fore Green
$nextHost = read-host
if ($nextHost -eq "y")
{
NextHost
}
if ($nextHost -eq "n")
{
write-host "Thanks, have a great day!" -Fore Green
exit
}
}


Function DoItUsers
{
write-host "Ok, I am going to add $newUser with a password of $newPass to $connectedServer" -Fore Green
Start-Sleep -seconds 3
New-VMHostAccount -ID $newUser -password $newPass -Description "vCM Service Account"
write-host "Ok, I am now adding admin permissions to the account we just created" -Fore Green
New-VIPermission -Role admin -Principal $newUser -Entity $connectedServer
write-host "Congratulations, you have added the new vCM service account successfully." -Fore Green
write-host "Would you like to configure another host? [y],[n]" -Fore Green
$nextHost = read-host
if ($nextHost -eq "y")
{
NextHost
}
if ($nextHost -eq "n")
{
write-host "Thanks, have a great day!" -Fore Green
}
}


Function NextHost
{
write-host "Please enter the FQDN of the server that you would like to connect to:" -Fore Green
$server = read-host
write-host "Connecting to $server using the credentials that you specified..." -Fore Green
Connect-VIServer -Server $server -User $existingUser -Password $existingPass
$connectedServer = Get-VMHost
write-host "Now using $connectedServer..." -Fore Green
if (!$connectedServer)
{
write-host ""
write-host "That host does not appear to be valid, please try again." Fore Yellow
NextHost
}

ExistingUsers
}


write-host ""
write-host ""
write-host "-- Welcome to my VMware vCM ESX/ESXi Service Account user deployment tool. This tool will allow you to easily deploy a vCM Service Account to your ESX/ESXi hosts -calebs71.blogspot.com --" -Fore Cyan
write-host ""
if (!$args[0])
{
write-host "Did you know that you can provide a CSV file with your server FQDN, ESX user, ESX Password, New Service User, New Service Password and automate the whole process?" -Fore Yellow
write-host ""
write-host "Example: vCM Service Acct on ESX.ps1 c:\list.csv"
write-host "FQDN,User,Password,SVCuser,SVCPass"
write-host "machine1.test,root,password,vcm_svc,password2"
write-host ""
write-host "Please enter a valid ESX/ESXi account with permissions to add a new user:" -Fore Green
$existingUser = read-host
write-host "Please enter the password:" -Fore Green
$existingPass = read-host
$connectedServer = Get-VMHost
if (!$connectedServer)
{
write-host ""
write-host "You are not connected to an ESX host..." -Fore Yellow
NextHost
}
write-host "You are currently connected to $connectedServer do you want to change servers? [y], [n]" -Fore Green
$response = read-host
if ($response -eq "y")
{
NextHost
}
if ($response -eq "n")
{
write-host "Continuing to use $connectedServer" -Fore Green
DoIt
}
}
if ($args[0])
{
$listMachines = import-csv $args[0]
if ($listMachines)
{
foreach ($MachineItem in $listMachines)
{
# write-host "FQDN : " $MachineItem.FQDN
# write-host "User : " $MachineItem.User
# write-host "Password : " $MachineItem.Password
# write-host "SVC User : " $MachineItem.SVCUser
# write-host "SVC Pass : " $MachineItem.SVCPass
# write-host "Connect-VIServer -Server "$MachineItem.FQDN " -User "$MachineItem.User " -Password "$MachineItem.Password
# write-host "New-VMHostAccount -ID "$MachineItem.SVCUser " -password "$MachineItem.SVCPass " -Description "vCM Service Account""
# write-host "New-VIPermission -Role admin -Principal " $MachineItem.SVCUser " -Entity " $MachineItem.FQDN

Connect-VIServer -Server $MachineItem.FQDN -User $MachineItem.User -Password $MachineItem.Password
New-VMHostAccount -ID $MachineItem.SVCUser -password $MachineItem.SVCPass -Description "vCM Service Account"
New-VIPermission -Role admin -Principal $MachineItem.SVCUser -Entity $MachineItem.FQDN
}

}

}

Error Removing Host from vSphere

I have a host (ESX 4.0) that I need to remove from one of my clusters but after putting it into maintenance mode and shutting it down the Remove option is grayed out. I found some references online that mention using the PowerCLI to manually remove the host but ran into a caveat that none of the other posts mentioned. When I ran the remove-vmhost command I got the following error: "The method is disabled by 'com.vmware.vcintegrity'". To get around this I had to take the disconnected host and remove it from the cluster and then run the remove-vmhost command to remove it from vSphere. I imagine that this is related to capacity issues on our old cluster but we are trying to move our blades and accompanying guests to a new cluster so HA and DRS is not high on the priority list.

Syntax to install HP CIM Providers on ESX4i

First, download and install the vCLI tools from vmware.com on your desktop.

Second, download the CIM Providers from HP.com

Third, Place host in Maintenance Mode and then scan the ESXi host for the Bulletin ID to be installed

c:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server -username root - scan - bundle

should return a name like

hpq-esxi4.1uX-bundle-1.0 (note: this is case sensitive)

Fourth, install the bulletin

c:\program files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server -username root -bundle c:\hpq-esxi4.1uX-bundle-1.0.zip -install - bulletin hpq-esxi4.1uX-bundle-1.0

Fifth, reboot the host.