vSphere: VM Stuck during Power down at 95%

Occasionally I've run into a VM that gets stuck at 95% while powering down. I know the issue isn't unheard of, but I didn't run into it until working with a few ESXi 4.0.0 208167 servers. So - if you have a virtual machine hangs while shutting down - and you're certain that you're just not waiting for it to finish powering down, and you've already tried to "power off" from the client- but the power-off command is stuck at 95%, you may have to manually kill the hung VM.

1. Login to the host with the hung machine via SSH (enable SSH if you haven't already)

2. do a /sbin/services.sh restart (or services vmware-mgmt restart on ESX)... which is the same thing as doing this from the ESXi console

1. This command will restart the agents that are installed in /etc/init.d/ ... including hostd, ntpd, sfcbd, sfcbd-watchdog, slpd and wsmand (and HA if you have it)

2. When you do this, the VI/vsphere client will loose connectivity as those services restart, but VM's that are running will not be affected

3. After the services have restarted, you can re-connect via the VI client.

3. Via SSH, go to the right datastore (such as, /vmfs/volumes/DatastoreName/VMname), and delete (rm -r) the *.vswp file (the swap file).

4. If you can't delete it, and you're getting an error message to the effect... can not remove VM: device or resource busy... go find the processes associated with the VM.

1. "ps auxfww|grep "vmname"

2. "kill -9 ProcessIDNumber"

5. After doing so, remove the orphaned VM from inventory... just right-click the "unknown" VM, and select "remove from inventory", being careful to not delete it.

6. Then delete the *.log, and *.0*. If you don't, re-adding the VM may cause the interface to hang, and you'll have to go through some of this all over again.

7. Add the VM back to the inventory, and you should be able to start the VM.

I have run into a situation once, where a host reboot was the only way to solve the problem. But other than that, this seems to be quite effective.

Adding bginfo to your vSphere Templates

I don't know about you, but I personally find Sysinternal's BgInfo tool quite useful... having comptuer names, IP addresses, and maybe one or two other things sitting on my desktop helps me keep track of which machine I'm doing work on. Assuming that it makese sense in your situation, you might want to consider adding BgInfo to your vSphere/ESX templates. If you're already familiar with BgInfo - just put a shortcut into your allusers startup profile in the template. Here's the step-by-step procedure.

1. Grab a copy of BgInfo.
2. Create your BgInfo template (save-as), and place it and bginfo somewhere like this: c:\windows\system32\bginfo
3. Create a shortcut, and put it here so that it runs regardless of who logs in:
   %ALLUSERSPROFILE%\Start Menu\Programs\Startup
4. Edit the shortcut target like so: C:\windows\system32\bginfo\Bginfo.exe c:\windows\system32\bginfo\update.bgi /NOLICPROMPT /TIMER:0
5. Now whenever you deploy a new VM, and someone logs into the machine, bginfo puts the relevent data on their desktop.

In your case, it might make more sense to put your bginfo template on a network share somwhere instead of the local machine.

vSPhere 4 on GA-EX58-UD3R

If you're looking to put together a vSphere whitebox, the Gigabyte GA-EX58-UD3R motherboard with an Intel Core i7 920 isn't a bad choice. If you're thinking about going down this path - there are a few things worth mentioning... and while there's nothing particularly challenging or "show-stopping" about getting this working properly (except maybe the NIC), reading this post might save you some time.

Networking

You should know that the onboard Realtek RTL8168C NIC doesn't work - vSphere doesn't detect it. Ultimatewhitebox has a note about this, as do some forum posts... and sure enough, they're right. I found a few Intel PRO/1000 MT PCI Desktop Adapter's on eBay that work just fine, but if you're buying parts for this - go ahead and get a couple of the Intel NICs now. And you might want to consider disabling the onboard NIC during the setup.

Optical Drive

The takeaway here - make certain that your optical drive and SATA drive are on different channels. Also, if you want to avoid the below consider using a SATA optical drive instead of bothering with using an old IDE drive that you might have laying around.

Things to avoid...

I was using the eval version of vSphere 4 (4.0.0, 208167) on disc. The disc was detected on boot up, and I was able to get launch the vSphere GUI setup and get the installation started.

Initially, I was using an old IDE DVD drive. While the boot process went mostly as expected (with an odd exception sometime after the setup process started I had a message "CDROM Failed to Mount") I was able to get to the point in the setup where I had to create partitions. At that point, the setup process wasn't seeing my onboard SATA hard drive. So I rebooted, checked the BIOS and confirmed that the drive was being detected... I did happen to notice that it was on the same channel as the hard drive. After moving them to different channels, I launched the GUI setup again and when it got to the point of detecting the SATA hard drive, it was there. After seeing that, the setup process needs to copy some files off of the CD... which it suddenly failed to do... no drive detected. That seemed kind of strange, given that I had booted off the DVD drive, but recalling the CDROM failed to mount message - perhaps not. I went back into the BIOS, and changed the mode that the Optical drive was in - rebooted, but again failed to detect CD at the create partitions window. Next, I swapped in a SATA DVD writer from a different machine, launched the vSphere GUI setup (no more CDROM failed to mount message), detected the SATA hard drive, created partitions, and it completed setup without incident.

Overall, not too challenging for a Friday night... and now I have a vSphere host in my lab to start playing with.

Setup

vSphere 4 (4.0.0, 208167) disc

Gigabyte GA-EX58-UD3R bios version FB

Intel Core i7 920 CPU

Realtek RTL8168C on-board NIC disabled

Intel PRO/1000 MT PCI Desktop Adapters

SATA DVD Writer (SH-S223)

Western Digital WD5000 Blue Label, 500GB 7200RPM SATA Drive
Update in 2012 - added a Intel 128GB SSD