VMware Bug Shuts Down ESX 3.5 Users

It’s bad enough to have a technical problem with one key piece of software. It’s a lot worse when the problem crops up in the software on which all your other systems run.

Some VMware users are living the nightmare after discovering that a glitch in the licensing code in ESX 3.5 Update 2 falsely identifies their implementations as being out of license, prevents new virtual machines from launching, or existing VMs from migrating to new hosts.

The problem apparently exists within the software licensing code, rather than the functional part of ESX software.

“An issue has been uncovered with ESX 3.5 Update 2 and ESXi 3.5 that causes the product license to expire on August 12,” says a statement from VMware’s public relations company in response to press inquiries. “VMware is alerting customers and partners of this issue. Updated product bits with correct licensing will be made available for download as soon as possible. VMware regrets any inconvenience to customers. VMware is working on an immediate patch for customers in production. VMware expects to fix the issue in code in the next 36 hours once QA testing has been completed,” the statement concludes.

Users who discovered the problems spread a warning on VMware Communities forums and offered both their own and VMware’s recommendations for workarounds:

“Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion: Aug 12 10:40:10.792: vmx| http://msg.License.product.expired This product has expired. Aug 12 10:40:10.792: vmx| Be sure that your host machine’s date and time are set correctly. Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: “http://www.vmware.com/info?id=4”. “

Most of the workarounds amount to turning off the Network Time Protocol (NTP) and resetting the timestamp on physical servers back to Aug. 10.

“We used the following workaround to power on the VM’s. Find the host where a VM is located run ‘ vmware-cmd -l ‘ to list the vms. issue the commands: service ntpd stop date -s 08/01/2008 vmware-cmd /vmfs/volumes/ service ntpd start “

Until Vmware releases a fix, expected within 36 hours, customers should avoid closing down running VMs or move them from one host to another, according to posters in the VMware support forums.