Then, on October 30, Microsoft released Q317748, a nonsecurity hot fix for SQL Server.

Danger: Patch Under Construction

Q317748 repaired a performance-degrading memory leak. But the team that built it had used an old, vulnerable version of ssnetlib.dll. When Q317748 was installed, it could overwrite the secure version of the file and thus make that server as vulnerable to a worm like Slammer as one that had never been patched.

"As bad as software can be, at least when a company develops a product, it looks at it holistically," says SEI’s Hernan. "It’s given the attention of senior developers and architects, and if quality metrics exist, that’s when they’re used."

Which is not the case with patches.

Patch writing is usually assigned to entry-level maintenance programmers, says Hernan. They fix problems where they’re found. They have no authority to look for recurrences or to audit code. And the patch coders face severe time constraints—remember there’s a footrace on. They don’t have time to communicate with other groups writing other patches that might conflict with theirs. (Not that they’re set up to communicate. Russ Cooper, who manages NTBugtraq, the Windows vulnerability mailing list, says companies often divide maintenance by product group and let them develop their own tools and strategies for patching.) There’s little, if any, testing of patches by the vendors that create them.

Ironically, maintenance programmers write patches using the same software development methodologies employed to create the insecure, buggy code that they are supposed to be fixing. It’s no surprise then that these Dr. FrankenPatches produce poorly written products that can break as much as they fix. For example, an esoteric flaw found last summer in an encryption program—one so arcane it might never have been exploited—was patched. The patch itself had a gaping buffer overflow written into it, and that was quickly exploited, says Hernan. In another case last April, Microsoft released patch MS03-013 to fix a serious vulnerability in Windows XP. On some systems, it also degraded performance by roughly 90 percent. The performance degradation required another patch, which wasn’t released for a month.

Slammer feasted on such methodological deficiencies. It infected both servers made vulnerable by conflicting patches and servers that were never patched at all because the SQL patching scheme was kludgy. These particular patches required scripting, file moves, and registry and permission changes to install. (After the Slammer outbreak, even Microsoft engineers struggled with the patches.) Many avoided the patch because they feared breaking SQL Server, one of their critical platforms. It was as if their car had been recalled and the automaker mailed them a transmission with installation instructions.