VMTools Version Mismatch

Before I get started, I just want to say that I don't know for sure if this is an issue that is causing problems for some VMs that I have been looking at or not. I'd welcome any feedback, comment or help anyone has to offer on the topic. I thought that I'd share this just in case it helps anyone at any point.

The environment that I've been working in is a shared web hosting platform built on ESX 3.5 and vCenter 2.5. Recently the focus of effort has been on data centre migrations, service migrations and office moves. Business as usual (BAU) activities have taken a back seat because of resource constraints. That includes upgrading the virtual infrastructure to the latest version. Currently vCenter is at Update 4 (build 147633) and needs to be brought up to date following the release of Update 5 a few weeks ago. The ESX hosts are slightly further downlevel however. Two thirds of them are on Update 3 (build 123630) with a couple of extra patches applied to bring them up to build 143128. The other third is on Update 2 (build 110268).

Windows Server 2008 R2 has recently started to be used for some of the VMs and this is not officially supported until Update 5. Some of these VMs have had minor issues, things like random slowness and occasional failures to remove snapshots following their scheduled backups. While looking into these issues I discovered quite a variety of VMTools versions being used.

Ideally all ESX hosts in a cluster should be on the same update level. The version of VMTools that a VM uses should ideally match the lowest build version of ESX that is being used in a cluster as the VM could potentially run on any host in the cluster. On one VM vCenter said that the VMTools version was “OK” but I wanted to run a manual “Install / Upgrade” on the VMTools anyway just to run a repair and make sure that the drivers were installed correctly. When I tried to do this, the installer popped back an error stating that a newer version was already installed.

That got me thinking, which versions are installed on all the VMs? Now I could have used the excellent RVTools or something like that or I could craft my own brief PowerCLI script. I couldn't resist the latter…

[ps]$vmhosts = get-vmhost | sort Name

$myCol = @()
foreach ($vmhost in $vmhosts)
{
$vmhostdetail = $vmhost | get-view
$versionmajor = $vmhostdetail.config.product.version
$versionbuild = $vmhostdetail.config.product.build
$vms = $vmhost | get-vm | sort Name
foreach ($vm in $vms)
{
$myObj = "" | Select Host,Version,Build,Guest,ToolsVersion
$vmdetail = $vm | get-view
$vmtoolsversion = $vmdetail.config.tools.toolsversion
$myObj.Host = $vmhost.Name
$myObj.Version = $versionmajor
$myObj.Build = $versionbuild
$myObj.Guest = $vm.Name
$myObj.ToolsVersion = $vmtoolsversion
$myCol += $myObj
}
}

$myCol | Export-Csv -NoTypeInformation "C:VMToolsVersions.csv"
ii "C:VMToolsVersions.csv"[/ps]

Running this script produces a CSV file with a line for each VM show the host, host version and build number, guest name and VMtools version for each VM. I've simplified the output that I got to remove the host names and guest names and also I've removed duplicate combinations of version numbers. That just leaves the following:

This shows the different combinations of ESX hosts versions and VMTools versions being used. What's very interesting are the VMs running VMTools version 164009. That is the build number for vSphere 4.0.0!

I know how this has come about. Rather than using templates, the preference here is to use a Windows ISO file that has been customised to be a practically unattended install that also includes a number of automatically installed applications. One of these is the VMTools installation. As there is a vSphere infrastructure here also the ISO has been modified to include the latest tools version so that one ISO fits all. But does it? The VMs are running butis the version of VMTools causing problems? That's the bit that I don't know the answer to yet. If anyone does know, please feel free to comment.

For now the way forward is going to be creating multiple ISO files or removing the VMTools installation from the ISO altogether. That way new builds will get their VMTools installed from the ESX server that they're running on. An upgrade to Update 5 is also called for and with next week looking to be a bit quieter I think that it'll be planned then to be executed early in the new year.