RKA Petroleum faced the problem of the faulty patch. The petroleum products wholesaler had been struggling for months to keep current with the constant release of patches from Microsoft and other vendors. Things came to a head earlier this year when a critical server crashed because it was using the wrong software patch; IT staffers spent 15 hours on the phone with Microsoft reps trying to figure out which patch to deploy. Ultimately they ended up rebuilding the server.

"The problem was Microsoft assigned a patch that was not needed for the machine. There wasn’t enough disk space for the patch, and it started deleting files. Services started to fail, and people couldn’t log on to our network," says IS Manager Jason Hittleman.

Soon after the incident, RKA bought a patch management product called Update from PatchLink, Hittleman says. RKA now uses the software to automatically deploy patches to its PCs and servers after approval by a human administrator.

The problem could have been avoided if the software had been in place to deploy the correct patch, Hittleman says. "It knows exactly what’s on each machine," so there’s less chance for errors in deployment, he says. "Before this problem, we had put all our trust in Microsoft. Whenever they recommended a patch, we wouldn’t think twice about it. This opened our eyes to the issue." He says the sub-$2,000 software has already paid for itself in reduced labor costs.

Vendors to the Rescue?

Some companies are trying to get software vendors to help ease patch management. Qualcomm, for instance, is using five different patch management vendors for various platforms, including Kintana (which was purchased by Mercury Interactive) for patches on Oracle applications.

"We are working with Oracle to [try to] simplify the patching process," says Tom Fisher, vice president of IT for Qualcomm’s CDMA technologies division. "There are a lot of patches required in their product set because of the [sheer volume] of applications. We’re pushing Oracle to simplify the patching process and either help us provide a better patch solution or adopt [technology from a vendor] like Kintana as a standard."

Fisher considers logs that track how well patches are distributed to be a critical feature. "Give me a log that tells me [the status of a patch distribution], so I know that it only happened on this machine, and I don’t have to worry about the 18 other machines I pushed to today. Logs are critical. I want to know how many [patches] failed. Why did they fail?"