"By late Sunday afternoon, Microsoft had two rooms set up on campus," says Cooper. "Services guys are in one room figuring out what to say to customers. A security response team is in the other room trying to figure out how to repackage the patches and do technical damage control.

"I’m on a cell phone, and there’s a guy there running me between the two rooms." Cooper laughs at the thought of it.

Why Every Patch Starts from Zero

As the volume and complexity of software increases, so do the volume and complexity of patches. The problem with this, says SEI’s Hernan, is that there’s nothing standard about the patch infrastructure or managing the onslaught of patches.

There are no standard naming conventions for patches; vulnerability disclosure comes from whatever competitive vendor can get the news out first. Distribution might be automated or manual; and installation could be a double-click .exe file or a manual process.

Microsoft alone uses a hierarchy of eight different patching mechanisms (the company says it wants to reduce that number). But that only adds to more customer confusion.

"How do I know when I need to reapply a security rollup patch? Do I then need to reapply Win2K Service Pack 2? Do I need to reinstall hot fixes after more recent SPs?" Similar questions were posed to a third-party services company in a security newsletter. The answer was a page-and-a-half long.

There’s also little record-keeping or archiving around patches, leaving vendors to make the same mistakes over and over without building up knowledge about when and where vulnerabilities arise and how to avoid them. For example, Apple’s Safari Web browser contained a significant security flaw in the way it validated certificates using SSL encryption, which required a patch. Every browser ever built before Safari, Hernan says, had contained the same flaw.

"I’d like to think there’s a way to improve the process here," says Mykolas Rambus, CIO of financial services company W.P. Carey. "It would take an industry body—a nonprofit consortium-type setup—to create standard naming conventions, to production test an insane number of these things, and to keep a database of knowledge on the patches so I could look up what other companies like mine did with their patching and what happened."

Rambus doesn’t sound hopeful.

Slammer Dopeslaps the Software Industry

Slammer has become something of a turning point. The fury of its 10-minute conflagration and the ensuing comedy of a gaggle of firefighters untangling their hoses, rushing to the scene and finding that the building has already burnt down, left enough of an impression to convince many that patching, as it is currently practiced, doesn’t work.