IT Security Management: Spam, Viruses and Software Patches

It’s never a good night for the IT department when the first person to get hit by a new virus is the CEO.

That’s exactly what happened when the W32.Blaster Internet worm slipped onto the notebook of ABM Industries chief Henrik Slipsager. Slipsager was booting up during a business trip in Los Angeles in August 2003 when the error message that defined the Blaster popped up, paralyzing his machine and millions of others across the globe. The CEO began calling cell phones of top IT staffers in San Francisco looking for help.

“It was 5:30 on a Wednesday,” recalls Sean Finley, assistant vice president and deputy director of electronic services at ABM, a $2.3 billion company that provides janitorial, lighting and security services to high-rise buildings. Finley, a 15-year veteran of the company, says he called an ABM website administrator in Los Angeles. “I said: ’Listen, you’ve got to do me a big favor,’” he recalls. Slipsager left his notebook with a hotel bellhop as the employee raced there with antivirus software. The CEO’s computer was fixed. But after that night, the way ABM dealt with viruses changed.

Instead of putting out fires, ABM’s IT group moved to set up policies that mandate how employees use antivirus software. One user mandate: No network log-on without the latest virus update download.

After disasters like the 2003 blackout in the Northeast and the devastation of 9/11, you’d think CIOs would be wearing hard hats and duck boots to the office. After all, they’ve been training?prodded by worried CEOs and boards of directors?to prepare for the catastrophic: floods, earthquakes, power outages, even terrorist attacks. Not surprising, IT spending on disaster recovery by global financial services companies after 9/11 spiked 19.2 percent to $3.4 billion?up from sleepier 3 percent to 5 percent annual increases throughout the 1990s, according to Tower Group. Although spending dipped by 6.4 percent in 2003, businesses are still shelling out unprecedented amounts of their IT budgets on security. An estimated 5.4 percent in 2003 went to bulk up security compared with 3.1 percent in 2001, according to Gartner.

Of course, the annoying headaches an IT staff tackles every day might seem insignificant when stacked up against natural disasters. But to the average company, they aren’t. The total effect of spam, viruses, software upgrades and other niggling problems is a plague that cost U.S. businesses billions last year. Most CIOs know this. They realize that the real threat isn’t Armageddon; it’s being nibbled to death by ducks.

“The majority of our time is spent on the little things that prevent the big things from happening,” says Dan Yee, CIO of the California Independent System Operator Corp. (the not-for-profit organization that manages the state’s power grid established to prevent electricity shortages and blackouts). Yee says focusing on the “little things” means, for example, splitting end users into different classes (like executives and other workers), and using automated tools to monitor what software gets onto their PCs in an effort to head off problems before they occur.

CIOs could be excused for delegating these nuisance issues to their staff. It makes sense to divide and conquer, to quash each snafu as it comes up. Many IT executives interviewed for this story continue to follow that approach.

But it’s also not hard to see that CIOs who fail to treat these nuisances holistically, as a class of problems that deserve management’s attention and a plan of attack, do so at their peril.

Spam, for one, cost corporations $10 billion in 2003, according to Ferris Research. Look at viruses: Computer Economics estimates that in 2003 the endless parade of 7,064 new viruses, worms and Trojan horses cost companies more than $13 billion. Even seemingly benign problems like employee password changes add up. These requests account for up to half the help desk calls in a given year and cost a company about $38 per annoying reset, according to Gartner. Add password updates to never-ending nuisances such as the employee who never deletes a single e-mail in 10 years or the PC user who crashes his computer during massive MP3 downloading, and the road leads to one all-encompassing term that could use its own army: nuisance management.

The good news is that CIOs have plenty of weapons in their utility belts to fend off many of these recurring problems. Ideas as simple as enforcing a better written policy for e-mail and banning certain kinds of instant messaging applications from the company’s desktops can make a big difference. Ultimately, dealing with nuisances is about being proactive and learning from mistakes. The problems might never go away, but they can be controlled.

Engage EVERY Nuisance, But Avoid Big Brother

CIOs often walk a tightrope: Trusting employees is important. The staff shouldn’t be forced to play Big Brother, censoring every software download or website visit. But trusting too much can lead to big budget trouble.

Most any tech administrator knows that the sneakiest network bandwidth stealers are often music and video file-sharing programs such as Audiogalaxy, Kazaa, LimeWire, Morpheus and NeoModus’s Direct Connect. MP3 files, at only 3MB to 5MB per song, might seem trivial?until 100 people download dozens of them simultaneously. Universities that cap bandwidth use are finding that MP3 downloads can hog up to 40 percent of network bandwidth at peak times. And it’s not just kids doing it. A May 2003 Jupiter Research survey of 2,835 consumers found that 12.3 percent of all 18- to 24-year-olds (compared with 4.5 percent in all age groups in the survey) regularly download MP3s at work.

Tools like Packeteer’s PacketShaper or Allot Communication’s NetReality detect when fat files are causing network slowdown. They examine packets as they move from the local to the wide area network and classify hundreds of applications. Companies can use the data collected to set policies?for example, allotting half of all bandwidth to Oracle applications and just a small percentage for specific file-sharing applications. Other tools made by vendors including Blue Coat Systems, SurfControl or Websense help filter unwanted applications. Evident Software takes bandwidth nuisance management a step further: It lets corporations track in dollars which corporate departments consume the most bandwidth. Then it’s up to the company to decide whether departments will be charged accordingly for their usage.

If charging bandwidth hogs doesn’t work, CIOs can always place a bandwidth cap on users who take more than their fair share.

LandAmerica Financial Group did this after analyzing bandwidth use. The real estate title insurance company has more than 700 offices in the United States that access the Internet through data centers in Richmond, Va., and Dallas. LandAmerica initially set out to use Packeteer’s network appliance to improve performance of its network, which often crawled because of peer-to-peer applications or if a worker simply opened a 20MB FTP file. Congestion took its toll on critical applications such as e-mail. To remedy the problem, LandAmerica set a 100K bandwidth limit to weed out heavy use of file-

sharing applications?like Gnutella and Kazaa?and file-sharing on instant chat. “People can use whatever they want up to 100K,” says Matt Matin, a systems engineer. LandAmerica figures it’s avoiding $500,000 in bandwidth upgrade costs by using Packeteer for application filtering and data compression.

Others handle the bandwidth problem differently. At Oklahoma State University, Michael White, the university’s interim director of telecommunications, uses NAT (network address translation) to deter file-sharing. NAT lets him set up network nodes so that many end users share few IP addresses; 750 kids in a dorm might share six IP addresses, for example. That way, the outside machine seeking to copy files can’t easily contact an individual machine in the dorm. However, White says a lot of the peer-to-peer software is able to query the network “super node” to find a single user. He concludes the best antidote is educating students to set their computers so that they aren’t open for file-sharing 24/7. “Most students just want to download music,” not share all their computer files, he says.

Instant Chat Campaign

Instant messaging might not hog as much network space as multiple Lord of the Rings downloads, but it can pose problems. Aside from bandwidth issues, many managers find it hard to track the panoply of IM software versions on user PCs. (Microsoft, Time Warner’s America Online unit, Yahoo, IBM’s Lotus division, Sun Microsystems and Oracle all make a corporate version of IM.) Just 26 percent of organizations have standardized on a common corporate IM application, according to market researchers at The Radicati Group.

Yet IM software is now installed within 90 percent of all corporate networks, according to research firm Osterman Research. Often it’s used by employees to get real work done. But some CIOs view it as a bandwidth-sucking productivity blaster.

“It’s a huge problem,” says Richard Ortiz, IT manager at Palace Entertainment, which runs water and amusement parks. Ortiz says he kept noticing strange spikes in traffic on his frame relay routers last year. So he used network reports to hunt down the culprit. It was IM. “The guys are worse than the girls,” says Ortiz. “They play poker. They’re talking to their friends about the football game.” In October 2003, Ortiz ended the fun, installing Akonix Systems software, which, like similar products including SurfControl’s Instant Message Filter, blocks IM use and helps stop end users from downloading pirated software and peer-to-peer file-sharing.

Akonix works by grabbing packets related to the application and blocking them from leaving the network. It also tells Ortiz who is trying to do what. “If Mary Jo in New York is downloading illegal software from Kazaa, it runs a report. She gets a [pop up] message that says what she’s trying to do isn’t company policy and that it will be reported to a manager,” Ortiz says. The reports are working. During the first week of using Akonix, 60 people received warning notices advising them that IM was no longer allowed. “Now we barely have 10 or five” offenders per week, he says.

Operation Auto Respond

For the worst nuisances?e-mail maintenance, antivirus updates and server software upgrades?companies are finding that automation works by saving time and labor. For Ron Rose, CIO of Priceline.com, the biggest headache used to be the hands-on part of software upgrades. Priceline’s business, which allows Web users to haggle the prices they pay for airfare, hotel rooms and other services, is powered by a farm of 300 Microsoft Windows servers that require between 100 and 200 software changes each per month, Rose says. “Before, we had a team of six people applying application updates on a machine-by-machine basis to each of the servers,” he says. “It would take up to an hour to deploy the software to a small group of the servers manually?every time we had to do an update.”

Now, Rose uses BladeLogic to consistently deploy software upgrades to not only Windows servers but to the company’s Sun Microsystems servers too. Others providing similar data-center automation offerings include IBM’s Tivoli, CenterRun (acquired in 2003 by Sun Microsystems), Moonlight Systems and Opsware.

Rose figures the technology has made a 50 percent increase in the efficiency of technicians doing software loading by eliminating all the hours they once spent manually loading software onto servers and debugging machines that were misconfigured during that manual process.

Allies in the Spam War

Other nuisances?like getting employees to delete e-mail regularly or to quit saving 25 versions of the same Excel spreadsheets on their hard drives?are harder to tackle. But for spam and virus management, more CIOs are looking to outsource the headache.

Spam now makes up about 60 percent of all messages pouring into corporate e-mail boxes. But it’s not so much the spam that strikes fear in the heart of CIOs, it’s the potential viruses lurking within the unwanted messages, says Andy Toner, a partner at PricewaterhouseCoopers who advises clients on security policy.

The average company shells out at least $2.5 million a year to deal with spam?when you add up productivity lost, bandwidth and storage consumption and support costs, according to server software maker NetIQ. Remedies range from simple whitelists and blacklists for filtering approved and disapproved mail, to software that analyzes the algorithms used to write e-mails. Many CIOs are using an army of tools to tackle spam, including CipherTrust’s IronMail, Brightmail’s Anti-Spam Enterprise Edition and Postini’s Perimeter Manager.

At Daiwa Securities America, Co-CIO Stephen McCabe outsources e-mail filtering to MessageLabs, which uses an artificial intelligence tool to weed out viruses and spam from incoming e-mails. Now, when MessageLabs finds either offender, it’s quarantined for further review and McCabe (rather than the end user) is notified. Filtering e-mail offsite before it enters the network has helped solve the problem, he says; they haven’t had a bad virus in more than two years. “When a virus hits an organization of our size, everything stops,” he says. “You have to quarantine and clean the machines. It takes a day to resolve.”

Mary Finlay, deputy CIO of Partners HealthCare System in Boston, says she’s considering outsourcing virus management because it’s become just too much of a nuisance. In January 2003, Slammer attacked a vulnerability in Microsoft SQL 2000 Web servers. For Finlay, whose company is largely standardized on Microsoft servers, Slammer was the last straw. She realized that her 1,000 IT workers weren’t communicating well and that they should be handling viruses?big and small?in a better way. “We were managing each virus as it came,” says Finlay, whose network ferries crucial information on everything from patient registration to lab tests to medication orders among employees at 10 hospitals. That virus-by-virus approach?done differently within each department?is typical at big corporations where each manager wants to handle the problem in his own way.

But that method made the company’s network vulnerable, Finlay says. So she initiated a system to keep antivirus software consistently up to date on both Windows NT servers and desktops at all 10 sites. “Whenever we’ve been hit since, having this process in place makes things run more smoothly,” Finlay says. Still, she says she’s looking to companies such as Symantec for help. “It’s very labor-intensive and confusing to gather intelligence around an impending virus,” she says.

Recognizing Friendly E-Mails Among the Foes

CIOs face a quandary when they get scrupulous about e-mail filtering. They want to keep out the “Cheap Viagra” messages, but they don’t want to filter out serious, work-related e-mail in the process.

Sean Bagshaw, CTO of Mortgage Bankers Association (MBA), tackled this problem recently. E-mail the company sends out?newsletters and other information?is often blocked as spam because some of MBA’s banking members include the word mortgage on their e-mail blacklists. “You are constantly fighting to get off the [black]list,” he says.

To fix the problem, Bagshaw asks member companies to add MBA’s e-mail to their white-list. But Bagshaw is also considering joining a Bonded Sender program through IronPort Systems. Bonded Sender uses a third party to certify that an e-mail sender has met specific antispam standards and has put up cash to back that. Under the system, MBA would put up a bond for, say, $50,000, which ensures a marketing campaign is not spam. A debit from the bond is collected if the third party finds MBA in violation. About 18,000 organizations participate in the program.

At ABM, the building services contractor whose CEO was hit by the Blaster virus, a blacklist mishap last year prompted a policy change, Finley says.

Blacklists blocked important e-mail coming in to ABM and prevented the company from sending mail to its customers. Finley says the company missed a sales opportunity last year because its software filtered out an e-mail from a potential customer who had sent a business inquiry to the sales department from a home address. The prospective customer later followed up, asking why no one from ABM responded to the e-mail. (The salesman, of course, never got the e-mail, Finley says.) In response, ABM hired a full-time employee to sift through thousands of filtered spam messages to identify spam patterns and catch legitimate e-mails.

And, Finley says, he still battles the ISPs that have kept ABM on their blacklists?an unfortunate side effect that occurred when an e-mail spoofer rerouted spam e-mail messages through ABM’s corporate servers so it appeared that ABM was the sender. “We have to call and threaten legal action and say, ’You better unblock us,’” he says.

Users JOIN THE BATTLETo gain control of the biggest nuisances, IT departments need to stop viewing workers as the enemy?and start recruiting them to be part of the solution.

CIOs who send out e-mail warnings or updates to workers are fooling themselves because employees “will think it’s some techy thing that they don’t have to worry about,” says Chris Belthoff, a senior security analyst at Sophos, a corporate provider of antispam and antivirus solutions.

Belthoff advises that companies create a hands-on training program with employees to educate them about the dangers of spam and viruses. He says it’s critical to show workers what spam e-mail subject lines look like so that they recognize them in their inboxes. Programs to train IT workers to be end user teachers are available from Symantec, among others. (For more tips, see “Spam Battle Gear,” Page 64.)

Training users in good e-mail hygiene has been part of the thinking at Winstead Sechrest & Minick, a law firm with approximately 720 employees. Director Mark Garrett says the firm trains all new employees on e-mail and Internet use policies and is now looking to add training and usage policies for instant messaging users. The law firm banned IM but is considering letting Internet-reliant lawyers use a chat application to communicate with clients.

There’s no way to get rid of every single nuisance. There will always be one employee who can’t resist clicking on an infected attachment. Still, prevention stops the nuisance from becoming a nightmare. “It’s about the little proactive things,” says the California Independent System Operator Corp.’s CIO Yee. “You don’t retrofit as an afterthought.”