Password recovery on VMWare ESXi 4 - problems with tar

One of my VMWare servers had a master password (to the VMWare console) that I had forgot. I.e. I had forgot to write it down, and not had any need to access the console for a while.

Started to look for a routine for password recovery, but after quite a lot of searching, I found that for ESXi 4, there is no way to recover the password. Would that stop me from trying? Not really, as reinstall would be quite difficult for various reasons, if nothing else because I did not have any CD to burn the VMWare software on.

I looked around, and found this process, which seems to work. I boot the VMWare ESXi from a USB stick, so the easiest thing was to just plug the USB stick into my Mac. Untar the local.tgz file (that btw was in the Hypervisor1 partition on the USB stick), edit the etc/shadow file etc as described. Pack the local.tgz file, place it on the USB and reboot.

Which failed with the error “Failed to find the boot partition” in the console.

The trick here is to understand what went wrong. You can access the console at this stage by typing the word “unsupported” (without quotes), and then just press return (as the root account do not have any password set at this stage). If things are sort of ok, you should get a prompt.

If you look in the /var/log/messages file, towards the end, you should see (if you have the same problem as I) that the file local.tgz could not be un-tar:ed. Or rather, the error message could be just anything.

If you try to untar manually (with the -tvzf flags), you will see the “tar: corrupted octal value in tar header” message from tar. A message that is not visible at boot, and not visible in the /var/log/messages file. But, this is the key…

You see, VMWare has built their system based on the BusyBox package. Pretty common, using many GNU utils etc. Including tar. But, the tar command they have is very sensitive. It can only handle one of the possibly 4-5 different formats that exists out there.

And of course on Snow Leopard, the tar installed is not creating a tar file with the correct format (MacOSX changed the tar binary between 10.5 and 10.6). Using FreeBSD is not helping either.

The correct solution on Snow Leopard is to give the following command to create the file:

# /usr/bin/gnutar --format=gnu -czf local.tgz etc

According to the manual page for gnutar on the mac, the format gnu refers to GNU tar 1.13. Which obviously is one of the few (maybe the only) format the tar that is installed in the Busybox that VMWare uses can parse.