Have you ever wondered how to install VMtools on a Windows 2008 Server R2 Core guest running on an ESX 3.5 host? “Hang on,” you say, “R2 is officially supported on ESX 3.5!” And you'd be right, for the moment at least. ESX 3.5 U4 does indeed not support R2 officially yet although VMware support hinted that U5 is due at some point. Still, the customer is king and mine wanted an R2 VM.
First things first, the WoW64 Execution Layer needs to be installed and the server rebooted to allow R2 to run 32 bit applications.
[text]Start /w ocsetup ServerCore-WOW64
shutdown /r /t 0[/text]
Once the reboot is complete and you've logged in again, select to Install the VM Tools for the VM. Nothing will actually happen except that ESX will mount the VMtools iso on the VM. To install the tools you'll need to navigate to the drive where the iso has been mounted. In the example below it is the D: drive.
Installing the tools is then simply a case of silently executing the installer. This will automatically reboot the server afterwards.
[text]msiexec /i "VMware Tools.msi" /passive[/text]
That's it, you're done.