It\u2019s never a good night for the IT department when the first person to get hit by a new virus is the CEO. That\u2019s exactly what happened when the W32.Blaster Internet worm slipped onto the notebook of ABM Industries chief Henrik Slipsager. Slipsager was booting up during a business trip in Los Angeles in August 2003 when the error message that defined the Blaster popped up, paralyzing his machine and millions of others across the globe. The CEO began calling cell phones of top IT staffers in San Francisco looking for help."It was 5:30 on a Wednesday," recalls Sean Finley, assistant vice president and deputy director of electronic services at ABM, a $2.3 billion company that provides janitorial, lighting and security services to high-rise buildings. Finley, a 15-year veteran of the company, says he called an ABM website administrator in Los Angeles. "I said: \u2019Listen, you\u2019ve got to do me a big favor,\u2019" he recalls. Slipsager left his notebook with a hotel bellhop as the employee raced there with antivirus software. The CEO\u2019s computer was fixed. But after that night, the way ABM dealt with viruses changed. Instead of putting out fires, ABM\u2019s IT group moved to set up policies that mandate how employees use antivirus software. One user mandate: No network log-on without the latest virus update download.After disasters like the 2003 blackout in the Northeast and the devastation of 9\/11, you\u2019d think CIOs would be wearing hard hats and duck boots to the office. After all, they\u2019ve been training?prodded by worried CEOs and boards of directors?to prepare for the catastrophic: floods, earthquakes, power outages, even terrorist attacks. Not surprising, IT spending on disaster recovery by global financial services companies after 9\/11 spiked 19.2 percent to $3.4 billion?up from sleepier 3 percent to 5 percent annual increases throughout the 1990s, according to Tower Group. Although spending dipped by 6.4 percent in 2003, businesses are still shelling out unprecedented amounts of their IT budgets on security. An estimated 5.4 percent in 2003 went to bulk up security compared with 3.1 percent in 2001, according to Gartner.Of course, the annoying headaches an IT staff tackles every day might seem insignificant when stacked up against natural disasters. But to the average company, they aren\u2019t. The total effect of spam, viruses, software upgrades and other niggling problems is a plague that cost U.S. businesses billions last year. Most CIOs know this. They realize that the real threat isn\u2019t Armageddon; it\u2019s being nibbled to death by ducks. "The majority of our time is spent on the little things that prevent the big things from happening," says Dan Yee, CIO of the California Independent System Operator Corp. (the not-for-profit organization that manages the state\u2019s power grid established to prevent electricity shortages and blackouts). Yee says focusing on the "little things" means, for example, splitting end users into different classes (like executives and other workers), and using automated tools to monitor what software gets onto their PCs in an effort to head off problems before they occur.CIOs could be excused for delegating these nuisance issues to their staff. It makes sense to divide and conquer, to quash each snafu as it comes up. Many IT executives interviewed for this story continue to follow that approach.But it\u2019s also not hard to see that CIOs who fail to treat these nuisances holistically, as a class of problems that deserve management\u2019s attention and a plan of attack, do so at their peril. Spam, for one, cost corporations $10 billion in 2003, according to Ferris Research. Look at viruses: Computer Economics estimates that in 2003 the endless parade of 7,064 new viruses, worms and Trojan horses cost companies more than $13 billion. Even seemingly benign problems like employee password changes add up. These requests account for up to half the help desk calls in a given year and cost a company about $38 per annoying reset, according to Gartner. Add password updates to never-ending nuisances such as the employee who never deletes a single e-mail in 10 years or the PC user who crashes his computer during massive MP3 downloading, and the road leads to one all-encompassing term that could use its own army: nuisance management. The good news is that CIOs have plenty of weapons in their utility belts to fend off many of these recurring problems. Ideas as simple as enforcing a better written policy for e-mail and banning certain kinds of instant messaging applications from the company\u2019s desktops can make a big difference. Ultimately, dealing with nuisances is about being proactive and learning from mistakes. The problems might never go away, but they can be controlled. Engage EVERY Nuisance, But Avoid Big BrotherCIOs often walk a tightrope: Trusting employees is important. The staff shouldn\u2019t be forced to play Big Brother, censoring every software download or website visit. But trusting too much can lead to big budget trouble.Most any tech administrator knows that the sneakiest network bandwidth stealers are often music and video file-sharing programs such as Audiogalaxy, Kazaa, LimeWire, Morpheus and NeoModus\u2019s Direct Connect. MP3 files, at only 3MB to 5MB per song, might seem trivial?until 100 people download dozens of them simultaneously. Universities that cap bandwidth use are finding that MP3 downloads can hog up to 40 percent of network bandwidth at peak times. And it\u2019s not just kids doing it. A May 2003 Jupiter Research survey of 2,835 consumers found that 12.3 percent of all 18- to 24-year-olds (compared with 4.5 percent in all age groups in the survey) regularly download MP3s at work. Tools like Packeteer\u2019s PacketShaper or Allot Communication\u2019s NetReality detect when fat files are causing network slowdown. They examine packets as they move from the local to the wide area network and classify hundreds of applications. Companies can use the data collected to set policies?for example, allotting half of all bandwidth to Oracle applications and just a small percentage for specific file-sharing applications. Other tools made by vendors including Blue Coat Systems, SurfControl or Websense help filter unwanted applications. Evident Software takes bandwidth nuisance management a step further: It lets corporations track in dollars which corporate departments consume the most bandwidth. Then it\u2019s up to the company to decide whether departments will be charged accordingly for their usage. If charging bandwidth hogs doesn\u2019t work, CIOs can always place a bandwidth cap on users who take more than their fair share. LandAmerica Financial Group did this after analyzing bandwidth use. The real estate title insurance company has more than 700 offices in the United States that access the Internet through data centers in Richmond, Va., and Dallas. LandAmerica initially set out to use Packeteer\u2019s network appliance to improve performance of its network, which often crawled because of peer-to-peer applications or if a worker simply opened a 20MB FTP file. Congestion took its toll on critical applications such as e-mail. To remedy the problem, LandAmerica set a 100K bandwidth limit to weed out heavy use of file-sharing applications?like Gnutella and Kazaa?and file-sharing on instant chat. "People can use whatever they want up to 100K," says Matt Matin, a systems engineer. LandAmerica figures it\u2019s avoiding $500,000 in bandwidth upgrade costs by using Packeteer for application filtering and data compression.Others handle the bandwidth problem differently. At Oklahoma State University, Michael White, the university\u2019s interim director of telecommunications, uses NAT (network address translation) to deter file-sharing. NAT lets him set up network nodes so that many end users share few IP addresses; 750 kids in a dorm might share six IP addresses, for example. That way, the outside machine seeking to copy files can\u2019t easily contact an individual machine in the dorm. However, White says a lot of the peer-to-peer software is able to query the network "super node" to find a single user. He concludes the best antidote is educating students to set their computers so that they aren\u2019t open for file-sharing 24\/7. "Most students just want to download music," not share all their computer files, he says. Instant Chat CampaignInstant messaging might not hog as much network space as multiple Lord of the Rings downloads, but it can pose problems. Aside from bandwidth issues, many managers find it hard to track the panoply of IM software versions on user PCs. (Microsoft, Time Warner\u2019s America Online unit, Yahoo, IBM\u2019s Lotus division, Sun Microsystems and Oracle all make a corporate version of IM.) Just 26 percent of organizations have standardized on a common corporate IM application, according to market researchers at The Radicati Group.Yet IM software is now installed within 90 percent of all corporate networks, according to research firm Osterman Research. Often it\u2019s used by employees to get real work done. But some CIOs view it as a bandwidth-sucking productivity blaster."It\u2019s a huge problem," says Richard Ortiz, IT manager at Palace Entertainment, which runs water and amusement parks. Ortiz says he kept noticing strange spikes in traffic on his frame relay routers last year. So he used network reports to hunt down the culprit. It was IM. "The guys are worse than the girls," says Ortiz. "They play poker. They\u2019re talking to their friends about the football game." In October 2003, Ortiz ended the fun, installing Akonix Systems software, which, like similar products including SurfControl\u2019s Instant Message Filter, blocks IM use and helps stop end users from downloading pirated software and peer-to-peer file-sharing.Akonix works by grabbing packets related to the application and blocking them from leaving the network. It also tells Ortiz who is trying to do what. "If Mary Jo in New York is downloading illegal software from Kazaa, it runs a report. She gets a [pop up] message that says what she\u2019s trying to do isn\u2019t company policy and that it will be reported to a manager," Ortiz says. The reports are working. During the first week of using Akonix, 60 people received warning notices advising them that IM was no longer allowed. "Now we barely have 10 or five" offenders per week, he says. Operation Auto RespondFor the worst nuisances?e-mail maintenance, antivirus updates and server software upgrades?companies are finding that automation works by saving time and labor. For Ron Rose, CIO of Priceline.com, the biggest headache used to be the hands-on part of software upgrades. Priceline\u2019s business, which allows Web users to haggle the prices they pay for airfare, hotel rooms and other services, is powered by a farm of 300 Microsoft Windows servers that require between 100 and 200 software changes each per month, Rose says. "Before, we had a team of six people applying application updates on a machine-by-machine basis to each of the servers," he says. "It would take up to an hour to deploy the software to a small group of the servers manually?every time we had to do an update." Now, Rose uses BladeLogic to consistently deploy software upgrades to not only Windows servers but to the company\u2019s Sun Microsystems servers too. Others providing similar data-center automation offerings include IBM\u2019s Tivoli, CenterRun (acquired in 2003 by Sun Microsystems), Moonlight Systems and Opsware. Rose figures the technology has made a 50 percent increase in the efficiency of technicians doing software loading by eliminating all the hours they once spent manually loading software onto servers and debugging machines that were misconfigured during that manual process.Allies in the Spam WarOther nuisances?like getting employees to delete e-mail regularly or to quit saving 25 versions of the same Excel spreadsheets on their hard drives?are harder to tackle. But for spam and virus management, more CIOs are looking to outsource the headache. Spam now makes up about 60 percent of all messages pouring into corporate e-mail boxes. But it\u2019s not so much the spam that strikes fear in the heart of CIOs, it\u2019s the potential viruses lurking within the unwanted messages, says Andy Toner, a partner at PricewaterhouseCoopers who advises clients on security policy. The average company shells out at least $2.5 million a year to deal with spam?when you add up productivity lost, bandwidth and storage consumption and support costs, according to server software maker NetIQ. Remedies range from simple whitelists and blacklists for filtering approved and disapproved mail, to software that analyzes the algorithms used to write e-mails. Many CIOs are using an army of tools to tackle spam, including CipherTrust\u2019s IronMail, Brightmail\u2019s Anti-Spam Enterprise Edition and Postini\u2019s Perimeter Manager. At Daiwa Securities America, Co-CIO Stephen McCabe outsources e-mail filtering to MessageLabs, which uses an artificial intelligence tool to weed out viruses and spam from incoming e-mails. Now, when MessageLabs finds either offender, it\u2019s quarantined for further review and McCabe (rather than the end user) is notified. Filtering e-mail offsite before it enters the network has helped solve the problem, he says; they haven\u2019t had a bad virus in more than two years. "When a virus hits an organization of our size, everything stops," he says. "You have to quarantine and clean the machines. It takes a day to resolve."Mary Finlay, deputy CIO of Partners HealthCare System in Boston, says she\u2019s considering outsourcing virus management because it\u2019s become just too much of a nuisance. In January 2003, Slammer attacked a vulnerability in Microsoft SQL 2000 Web servers. For Finlay, whose company is largely standardized on Microsoft servers, Slammer was the last straw. She realized that her 1,000 IT workers weren\u2019t communicating well and that they should be handling viruses?big and small?in a better way. "We were managing each virus as it came," says Finlay, whose network ferries crucial information on everything from patient registration to lab tests to medication orders among employees at 10 hospitals. That virus-by-virus approach?done differently within each department?is typical at big corporations where each manager wants to handle the problem in his own way. But that method made the company\u2019s network vulnerable, Finlay says. So she initiated a system to keep antivirus software consistently up to date on both Windows NT servers and desktops at all 10 sites. "Whenever we\u2019ve been hit since, having this process in place makes things run more smoothly," Finlay says. Still, she says she\u2019s looking to companies such as Symantec for help. "It\u2019s very labor-intensive and confusing to gather intelligence around an impending virus," she says.Recognizing Friendly E-Mails Among the FoesCIOs face a quandary when they get scrupulous about e-mail filtering. They want to keep out the "Cheap Viagra" messages, but they don\u2019t want to filter out serious, work-related e-mail in the process.Sean Bagshaw, CTO of Mortgage Bankers Association (MBA), tackled this problem recently. E-mail the company sends out?newsletters and other information?is often blocked as spam because some of MBA\u2019s banking members include the word mortgage on their e-mail blacklists. "You are constantly fighting to get off the [black]list," he says.To fix the problem, Bagshaw asks member companies to add MBA\u2019s e-mail to their white-list. But Bagshaw is also considering joining a Bonded Sender program through IronPort Systems. Bonded Sender uses a third party to certify that an e-mail sender has met specific antispam standards and has put up cash to back that. Under the system, MBA would put up a bond for, say, $50,000, which ensures a marketing campaign is not spam. A debit from the bond is collected if the third party finds MBA in violation. About 18,000 organizations participate in the program.At ABM, the building services contractor whose CEO was hit by the Blaster virus, a blacklist mishap last year prompted a policy change, Finley says.Blacklists blocked important e-mail coming in to ABM and prevented the company from sending mail to its customers. Finley says the company missed a sales opportunity last year because its software filtered out an e-mail from a potential customer who had sent a business inquiry to the sales department from a home address. The prospective customer later followed up, asking why no one from ABM responded to the e-mail. (The salesman, of course, never got the e-mail, Finley says.) In response, ABM hired a full-time employee to sift through thousands of filtered spam messages to identify spam patterns and catch legitimate e-mails. And, Finley says, he still battles the ISPs that have kept ABM on their blacklists?an unfortunate side effect that occurred when an e-mail spoofer rerouted spam e-mail messages through ABM\u2019s corporate servers so it appeared that ABM was the sender. "We have to call and threaten legal action and say, \u2019You better unblock us,\u2019" he says. Users JOIN THE BATTLETo gain control of the biggest nuisances, IT departments need to stop viewing workers as the enemy?and start recruiting them to be part of the solution. CIOs who send out e-mail warnings or updates to workers are fooling themselves because employees "will think it\u2019s some techy thing that they don\u2019t have to worry about," says Chris Belthoff, a senior security analyst at Sophos, a corporate provider of antispam and antivirus solutions.Belthoff advises that companies create a hands-on training program with employees to educate them about the dangers of spam and viruses. He says it\u2019s critical to show workers what spam e-mail subject lines look like so that they recognize them in their inboxes. Programs to train IT workers to be end user teachers are available from Symantec, among others. (For more tips, see "Spam Battle Gear," Page 64.)Training users in good e-mail hygiene has been part of the thinking at Winstead Sechrest & Minick, a law firm with approximately 720 employees. Director Mark Garrett says the firm trains all new employees on e-mail and Internet use policies and is now looking to add training and usage policies for instant messaging users. The law firm banned IM but is considering letting Internet-reliant lawyers use a chat application to communicate with clients. There\u2019s no way to get rid of every single nuisance. There will always be one employee who can\u2019t resist clicking on an infected attachment. Still, prevention stops the nuisance from becoming a nightmare. "It\u2019s about the little proactive things," says the California Independent System Operator Corp.\u2019s CIO Yee. "You don\u2019t retrofit as an afterthought."