To combat the problem, many companies are coming to rely on products that automate the process. The general consensus among IT managers who use these patch management tools is that they are well worth the investment—saving time, labor and money. But be warned: Patch management alone won’t provide a complete solution. Organizations need to combine automation with an effort to rein in the out-of-control computing environments that helped make patch deployment so complex in the first place, say IT managers and analysts.

Patch Wars

Patch deployments are often major endeavors, requiring companies to devote thousands of man-hours to manual fixes. Deploy the wrong patches, or fail to patch the right machines, and the resulting vulnerabilities can become major problems. (The SQL Slammer worm, for instance, took advantage of a flaw in Microsoft’s software that already had a patch.) A survey last year by Aberdeen Group showed that companies and government agencies worldwide are spending in excess of $2 billion annually to deal with patches. And Digex, a provider of managed Web and application hosting services, calculates the annual cost of manually managing patch deployment to be about $14,400 per server.

"[That cost is] extraordinary, and that’s just at the Microsoft level; other vendors have vulnerabilities as well," says Bobby Patrick, vice president of strategy at Digex. "It’s war out there to manage patches." And the situation is getting worse as companies get inundated with patch releases. (CERT, for example, reported 4,129 security vulnerabilities in 2002, nearly twice as many as in the previous year.)

"It’s like drinking from a fire hose," says Eric Hemmendinger, a research director at Aberdeen. Even when a company devotes people to patch deployments, "a lot of time has to be spent on this if you want to be diligent about it," he notes.

Patches are generally issued "in a way that’s convenient for the supplier but not necessarily for the user," Hemmendinger adds. "Supplier assessments as to whether patches are critical are judgments rendered in a vacuum; vendors don’t know what the customer is doing with their product."

Automation Opportunity

Some analysts say companies shouldn’t even consider applying patches manually. "It’s impossible. If you think about a company with thousands of desktops and hundreds of servers, manual processes don’t scale," says Gartner Research Director Mark Nicolett. "For each system, you have to look at what software is installed and understand which patches apply to that machine. It takes lots of analysis to figure out which one goes on which machine."