VMware CEO Apologizes for 'Time Bomb' Mess

Computerworld — Just minutes after VMware Inc. issued a patch late yesterday that allowed enterprises to start up crippled virtual servers, the company's CEO said he was sorry for the snafu.

"I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners," VMware CEO Paul Maritz said in an open letter posted to the company's Web site just before midnight EDT Tuesday.

Yesterday, reports flooded VMware's support forums from users unable to power up virtual servers that had been updated to the newest software, which was released in late July. Users said they were seeing error messages claiming that the virtualization software's license had expired as of Aug. 12.

Maritz, who was appointed CEO about a month ago, confirmed that VMware developers had left code in recent ESX 3.5 and ESXi Server 3.5 updates that prevented users from powering up virtual machines when the calendar flipped to Aug. 12. The code, he said, was left over from beta versions of the update.

"We failed in two areas," said Maritz in his mea culpa. "[We failed in] not disabling the code in the final release of Update 2 and not catching it in our quality assurance process."

Although it's common practice for developers to code a time limit into their betas in order to force users to upgrade to the final version, the deadline—often called a "time bomb"—is supposed to be removed before the final version is released.

"We have kicked off a comprehensive, in-depth review of our QA and release processes and will quickly make the needed changes," said Maritz.

Tuesday night, about two hours later than original scheduled, VMware issued a patch for the problem. The fix—available in separate versions for ESX 3.5 and ESXi Server 3.5—works, according to reports by users to VMware's support forum.

"I updated one of my five hosts, and everything appears to be fine so far," said a user named Robert Greenlee on a long support message thread. "Thanks for all the hard work getting this patch out. Unfortunately, I think you've gotten a serious black eye today. We were finally getting management happy with the idea of using ESX for production servers, and this set us back a little bit."

Greenlee wasn't the only user on the support forum taking shots at VMware for the bug, which left virtual machines unusable for more than 20 hours.

"As a VMware enterprise partner and VMware authorized consultant, I can tell you this is a big deal for VMware to release a product that has such grave consequences for even a relatively small portion of the total VMware user population," said a user identified as "wwcusa" from Jacksonville, Fla. "This could have been prevented by not rushing an update to market which was intended to be free and compete with [Microsoft Corp.'s] Hyper-V. VMware ran face first into the very hurdle it was trying to clear.

"This will cast doubt about the reliability of VMware in the enterprise," the user added.

VMware also quashed rumors that the time bomb was anything but a failure of its quality-control process. In an entry posted to a company blog Tuesday night, the company denied that the problem was security-related.

"The ESX(i) 3.5 Update 2 power-on problem that surfaced today is not related to exploitation of a security issue on ESX," said VMware. "Several customers have been worried that their ESX systems had been compromised by an attack and that this was the cause for not booting of their ESX update 2 Virtual Machines today."