Patch Management: Simplifying IT Managers’ Lives While Improving Security

Until recently, patch management was something most technology managers didn’t think much about. Security holes and bugs appeared; vendors released patches to repair them—and that was that. But software companies now release thousands of patches every year, and CIOs are running into potentially costly problems because they don’t have an efficient, automated way to manage and deploy these fixes.

To combat the problem, many companies are coming to rely on products that automate the process. The general consensus among IT managers who use these patch management tools is that they are well worth the investment—saving time, labor and money. But be warned: Patch management alone won’t provide a complete solution. Organizations need to combine automation with an effort to rein in the out-of-control computing environments that helped make patch deployment so complex in the first place, say IT managers and analysts.

Patch Wars

Patch deployments are often major endeavors, requiring companies to devote thousands of man-hours to manual fixes. Deploy the wrong patches, or fail to patch the right machines, and the resulting vulnerabilities can become major problems. (The SQL Slammer worm, for instance, took advantage of a flaw in Microsoft’s software that already had a patch.) A survey last year by Aberdeen Group showed that companies and government agencies worldwide are spending in excess of $2 billion annually to deal with patches. And Digex, a provider of managed Web and application hosting services, calculates the annual cost of manually managing patch deployment to be about $14,400 per server.

“[That cost is] extraordinary, and that’s just at the Microsoft level; other vendors have vulnerabilities as well,” says Bobby Patrick, vice president of strategy at Digex. “It’s war out there to manage patches.” And the situation is getting worse as companies get inundated with patch releases. (CERT, for example, reported 4,129 security vulnerabilities in 2002, nearly twice as many as in the previous year.)

“It’s like drinking from a fire hose,” says Eric Hemmendinger, a research director at Aberdeen. Even when a company devotes people to patch deployments, “a lot of time has to be spent on this if you want to be diligent about it,” he notes.

Patches are generally issued “in a way that’s convenient for the supplier but not necessarily for the user,” Hemmendinger adds. “Supplier assessments as to whether patches are critical are judgments rendered in a vacuum; vendors don’t know what the customer is doing with their product.”

Automation Opportunity

Some analysts say companies shouldn’t even consider applying patches manually. “It’s impossible. If you think about a company with thousands of desktops and hundreds of servers, manual processes don’t scale,” says Gartner Research Director Mark Nicolett. “For each system, you have to look at what software is installed and understand which patches apply to that machine. It takes lots of analysis to figure out which one goes on which machine.”

Patch management tools look for and analyze new patches, scan devices on networks to find vulnerabilities, and deploy the correct patches. Some products also test the patches to verify that they work. But automated systems aren’t a cure-all. Truck manufacturer Paccar has been using several patch management tools “with varying degrees of success,” says Vice President and CIO Patrick Flynn.

Paccar has used products from McAfee Security, Microsoft and On Technology to automatically deploy patches to PCs and servers, Flynn says, cutting the time it takes for deployments and improving their accuracy. “We don’t have to send an army of people out there [to deploy patches]. From a labor perspective, the return on investment has to be pushing 100 percent,” he says. “But there’s still a long way to go before we get to where we need to be” because the tools don’t ensure that all of the company’s more than 12,000 PCs and servers are getting the correct patches when they need them.

“If your success rate is 90 percent, you still have 1,000 devices to find and update,” Flynn says. “There isn’t really an integrated one-size-fits-all tool for patch management. If you’re a global IT organization, you need a variety of tools to deploy patches for security, operating systems, browsers, applications. And patches have to be applied to servers, desktops and PDAs. How do we manage all that software? It’s one of the biggest problems facing our industry.”

The management tools have been fairly easy to set up, Flynn says, but there can be some complexity when integrating patch tools with existing software. He says companies should have someone dedicated to overseeing patch management software to ensure it’s being used most efficiently.

A Hard Patch

Paccar’s experience is mirrored by others. After the Slammer attack illustrated companies’ vulnerabilities, RBC Centura Banks issued a corporate mandate to bring all its production servers up to date on required patches. “We checked into doing it manually and knew we had a sizable task in front of us,” says James Williams, manager of information delivery.

Managers estimated it would take more than 12,000 man-hours to complete the task, including deploying and testing patches on more than 220 servers. RBC opted instead to use Ecora’s PatchMeister to deploy patches, and the process took about 2,000 hours in just over a month. Williams says there was still manual work involved, when technicians had to tweak some of the patch deployments for particular servers. But the automation software did provide time and labor savings, Williams says. “It helped us to quickly identify the status of the servers and told us which patches we needed to apply in our NT and Windows 2000 environment.” The bank uses the software to evaluate and deploy anywhere from two to 15 patches per month.

RKA Petroleum faced the problem of the faulty patch. The petroleum products wholesaler had been struggling for months to keep current with the constant release of patches from Microsoft and other vendors. Things came to a head earlier this year when a critical server crashed because it was using the wrong software patch; IT staffers spent 15 hours on the phone with Microsoft reps trying to figure out which patch to deploy. Ultimately they ended up rebuilding the server.

“The problem was Microsoft assigned a patch that was not needed for the machine. There wasn’t enough disk space for the patch, and it started deleting files. Services started to fail, and people couldn’t log on to our network,” says IS Manager Jason Hittleman.

Soon after the incident, RKA bought a patch management product called Update from PatchLink, Hittleman says. RKA now uses the software to automatically deploy patches to its PCs and servers after approval by a human administrator.

The problem could have been avoided if the software had been in place to deploy the correct patch, Hittleman says. “It knows exactly what’s on each machine,” so there’s less chance for errors in deployment, he says. “Before this problem, we had put all our trust in Microsoft. Whenever they recommended a patch, we wouldn’t think twice about it. This opened our eyes to the issue.” He says the sub-$2,000 software has already paid for itself in reduced labor costs.

Vendors to the Rescue?

Some companies are trying to get software vendors to help ease patch management. Qualcomm, for instance, is using five different patch management vendors for various platforms, including Kintana (which was purchased by Mercury Interactive) for patches on Oracle applications.

“We are working with Oracle to [try to] simplify the patching process,” says Tom Fisher, vice president of IT for Qualcomm’s CDMA technologies division. “There are a lot of patches required in their product set because of the [sheer volume] of applications. We’re pushing Oracle to simplify the patching process and either help us provide a better patch solution or adopt [technology from a vendor] like Kintana as a standard.”

Fisher considers logs that track how well patches are distributed to be a critical feature. “Give me a log that tells me [the status of a patch distribution], so I know that it only happened on this machine, and I don’t have to worry about the 18 other machines I pushed to today. Logs are critical. I want to know how many [patches] failed. Why did they fail?”

Fisher says it’s difficult to know which of the many patch management products are best. “We have Solaris, HP-UX and Linux, and we’re trying to figure out a solution that will cover all three,” he says. “Typically we’ve found we’d have to write [such a solution]. I’d love to be able to use one tool, but I know that applications are different from firmware, which is different from operating systems.”

Even with automated patch management, analysts say, the process can still be complex. “If a company has an unstructured environment with a large combination of software products on PCs and servers, it’s not in a good position—no matter what patch management tool it’s using,” says Gartner’s Nicolett.

“Patch management software is helpful in that you at least have a vehicle for delivering a high volume of patches automatically. But if you have five configurations of Windows and 500 flavors of production systems, it’s still very difficult to deploy patches,” says Laura Koetzle, a senior analyst at Forrester Research. “Having too many configurations of Windows is a huge problem.”

Koetzle suggests enterprises select three or four standard configurations—for example, a Web server, an application server and a back office—and standardize on those. “This ensures that eventually you will have a standard pool of Windows configurations that you can more easily test patches against,” she says.

Paccar’s Flynn agrees that computing environments are getting too complex. “For the last several years, we’ve worked to get control of our computing environment,” he says. “The more you add things, the more difficult it becomes.” For that reason, Paccar has begun to restrict company PC specifications and software packages, and restricts who can modify settings.

Meanwhile, software vendors (see “Patch Tools,” Page 80) are lining up to provide patch management tools. Microsoft is attempting to improve its own patch solutions. In June, the company launched a redesigned patch management tool and announced it would make continued improvements to its system for issuing security patches for its products. Microsoft later this year will consolidate the ways it distributes patches to customers, to simplify the process.

Will such efforts decrease the need for patch management tools? Not likely. “The tools vendors will always be able to provide some additional value on top of what Microsoft is doing,” says Koetzle.