The company's problems started with a 2006 presentation at the Black Hat security conference by Joanna Rutkowska, CEO of Invisible Things Lab. Ironically, Rutkowska's "Blue Pill" talk had nothing to do with VMware. It was about creating undetectable malicious software using the virtualization technology built into microprocessors.

But nevertheless, VMware is the world's best-known virtualization company, so any questions about virtualization and security "naturally became a VMware problem," said Nand Mulchandani, the company's senior director for security products.

"Blue Pill kind of set things off, but unfortunately it set things off on the wrong foot," he said. Soon VMware was fielding questions from worried customers. "They escalated it to our team and they said, 'Oh my God, we're going to get attacked by Blue Pill. What do we do?'"

Mulchandani has been trying to get the message across that the Blue Pill CPU virtualization hack is not connected to VMware's software, which is widely used on data center servers to simultaneously run many copies of the operating system on a single computer.

It's one of several security messages that Mulchandani is trying to convey these days, as the company looks to repair its reputation in the security community while developing new products that will keep it one step ahead of rivals.

Critics say VMware must shoulder some of the blame for the Blue Pill confusion and that it harmed itself by attacking Blue Pill in company blog postings. "They took the easy route, which was to attack Joanna's research," said Tom Liston, a senior security consultant with Intelguardians Network Intelligence. "It was just a big brouhaha with VMware jumping in where they didn't belong."

The feud with Rutkowska flared up at a low point in the company's relationship with independent security researchers. Employees who had been working with researchers like Liston left, and by early 2007 the company had developed a reputation as being unresponsive to bug reports, something Mulchandani calls "Fortress VMware."

Mulchandani says the issue was simply that VMware didn't have the people in place to respond to the community. That changed, however, with the company's 2007 acquisition of intrusion-prevention software vendor Determina.

"With the Determina acquisition, a lot of the focus was on acquiring a team that had very fundamental and deep relationships with the security industry," said Mulchandani, formerly Determina's CEO. "We've really embraced the security community in a way we didn't before."

Since the acquisition VMware has restructured its bug response team, revamped its security portal and reached out to independent security consultants from I/O Active and the Metasploit team, to ask them to help hack their products and teach the company's engineering team.