"We’re between a rock and a hard place," says Bob Wynn, former CISO of the state of Georgia. "No one can manage this effectively. I can’t just automatically deploy a patch. And because the time it takes for a virus to spread is so compressed now, I don’t have time to test them before I patch either."

How to Build a Monster

Patching is, by most accounts, as old as software itself. Unique among engineered artifacts, software is not beholden to the laws of physics; it can endure fundamental change relatively easily even after it’s been "built." Automobile engines, by contrast, don’t take to piston redesigns once they roll off the assembly line nearly so well.

This unique characteristic of software has contributed to a software engineering culture that generally regards quality and security as obstacles. An adage among programmers suggests that when it comes to software, you can pick only two of three: speed to market, number of features, level of quality. Programmer’s egos are wrapped up in the first two; rarely do they pick the third (since, of course, software is so easily repaired later, by someone else).

Such an approach has never been more dangerous. Software today is massive (Windows XP contains 45 million lines of code) and the rate of sloppy coding (10 to 20 errors per 1,000 lines of code) has led to thousands of vulnerabilities. CERT published 4,200 new vulnerabilities last year—that’s 3,000 more than it published three years ago. Meanwhile, software continues to find itself running evermore critical business functions, where its failure carries profound implications. In other words, right when quality should be getting better, it’s getting exponentially worse.

Patch and Pray

stitching patches into these complex systems, which sit within labyrinthine networks of similarly complex systems, makes it impossible to know if a patch will solve the problem it’s meant to without creating unintended consequences. One patch, for example, worked fine for everyone—except those unlucky users who happened to have a certain Compaq system connected to a certain RAID array without certain updated drivers. In which case the patch knocked out the storage array.

Tim Rice, network systems analyst at Duke University, was one of the unlucky ones. "If you just jump in and apply patches, you get nailed," he says. "You can set up six different systems the same way, apply the same patch to each, and get one system behaving differently."

Raleigh Burns, former security administrator at St. Elizabeth’s Medical Center, agrees. "Executives think this stuff has a Mickey Mouse GUI, but even chintzy patches are complicated."