Matt Kesner has been in IT long enough to take silence as a compliment. "People don\u2019t often come down to IT to say, \u2019Nice job,\u2019" says the CTO of Fenwick & West, a national law firm. "The best you get is that they don\u2019t come down at all when things are running well."But then Kesner tackled the firm\u2019s spam problem, and suddenly he found himself a hero. After he outsourced the problem to a managed service provider, the law firm\u2019s partners (whose time is worth $350 to $600 an hour) were no longer spending more than an hour a day wading through 300 to 500 spam messages to get as many legitimate messages. "We got quite a few pats on the back and attaboys after putting the spam filter in place. Users saw the difference instantly and are dealing with hundreds fewer messages a day. They actually got excited about it."Unlike the invisible foe of Y2K, the scourge of spam\u2014which plagues receptionists and CEOs alike\u2014is painfully evident to everyone. Now that spam accounts for 40 percent to 60 percent of most organizations\u2019 e-mail traffic, you scarcely need to mention that Ferris Research says spam will cost U.S. businesses at least $10 billion this year, or that Nucleus Research estimates that companies forfeit $874 per employee annually in lost productivity alone. Nor do you have to bring up the fact that spam clogs e-mail systems and siphons IT resources away from legitimate business projects. Spam is a royal pain in the server, and we all know it.As Kesner has discovered, the sheer ubiquitousness of spam affords CIOs a rare opportunity to look good. Although receiving some spam is inevitable (and employees\u2019 expectations should be set accordingly), there\u2019s plenty you can do to make things better. In fact, there\u2019s plenty you should do, since the problem is only going to get worse, and you can\u2019t count on antispam legislation to save the day. (Criminalizing spam would simply drive more spammers to send their messages through offshore ISPs.) Solve the spam problem\u2014or even just put a big dent in it\u2014and you too can be a hero. Here\u2019s a look at how otherwise mild-mannered CIOs are leaping into the spam fray to help keep e-mail viable for users.The Spam Balancing ActWhat makes it so hard to write antispam laws or antispam software is that there\u2019s no such thing as a universal litmus test for spam. "One person\u2019s spam is another person\u2019s newsletter," says Eric Ogren, a senior analyst at the Yankee Group. "There\u2019s no magic widget the CIO can put in front of the e-mail server and spam goes away." End users have to be involved in deciding what is spam, he explains, because what\u2019s unwanted can vary widely not just from one company to the next, but from one person to the next. What looks like spam to the rest of the world could be essential business communication for certain employees. Colorful language might be important to a customer service agent (displeased customers often lose their tempers, after all), anatomical references may be work-related for a doctor in a research hospital, and Viagra messages could very well be germane to someone in the pharmaceutical industry.Case in point: When John Zarb, CIO of Libbey, a manufacturer of glassware, china and flatware, tested the Guenivere (a virus and subject-line filter) and SpamAssassin (an open-source spam filter), he had to shut them off after 10 days because they were rejecting important legitimate e-mails. The filters bounce mail with a spam score of 7.5, yet they were automatically assigning 7 points to e-mails from an Asian country in which Libbey has business relationships. Another rule assigned what Zarb calls "bad points" for using all capital letters. Since using all caps is common practice in that Asian country, messages from those business partners easily racked up more than 7.5 points and therefore got zapped. "If the message is a transport document, ouch," says Zarb. His group tweaked the default settings so that Asian e-mails wouldn\u2019t automatically accrue so many points. Today, the filters block about 70 percent of Libbey\u2019s spam, and Zarb says the false positive rate is far lower but not zero. Because some messages are too critical to miss, he decided to exempt a few employees who deal with international issues from the SpamAssassin filter. As Zarb quickly discovered, once you start filtering mail, you run the risk of blocking legitimate e-mails because they look like spam. Avoiding an unacceptable level of "false positives" requires a delicate balancing act. Although most vendors will claim they capture at least 90 percent of spam, going above 90 percent will probably result in too many false positives, says Matt Cain, a senior vice president at Meta Group. "You could crank it up and catch 98 percent of spam. But you\u2019d get an unhealthy amount of false positives," he says. "And if you go down to 85 percent, you\u2019ll have very few false positives, but too much spam will be getting through." At printing ink manufacturer Flint Ink, Vice President and CIO Don Barnowski has been trying out Symantec\u2019s Norton antispam product. After initially filtering on 300 to 400 keywords, false positives were a daily occurrence. "We started to get calls from people not getting e-mail they were expecting," he says. "That was a red flag; you don\u2019t want people questioning the integrity of e-mail delivery."Cutting the keyword list in half cut the false positive rate in half, but it also let more spam through. "I\u2019ve accepted the fact that we can\u2019t prevent all spam from reaching employees," he says. "Finding out five times a day that I can improve my mortgage rate is irritating but not offensive. There\u2019s a big difference there. It\u2019s more important to reduce the number of false positives than it is to smother all spam. You can\u2019t have it both ways, unfortunately."To combat false positives, make sure you choose a spam solution that gives you a quarantine area for probable spam that users can access to check for legitimate messages. Users can be alerted in the form of an e-mail digest of all blocked spam subject lines or be directed to a Web mailbox. Outsourcers generally maintain quarantine areas on their servers so that companies don\u2019t have to tie up their own networks with suspected spam. Giving end users the ability to add addresses to trusted sender lists (often called whitelists) also ensures that legitimate senders won\u2019t get blocked."We took the approach of putting in very coarse controls at first, then tightening them up, rather than going with the \u2019big bang\u2019 theory and begging forgiveness for weeks," says Gene Fredriksen, vice president of information security at financial services company Raymond James Financial. "It\u2019s absolutely a strategy I\u2019d recommend. You have to build trust in your system first." Fredriksen uses Syntegra\u2019s managed service to filter spam for the company\u2019s 14,000 mailboxes.It\u2019s also smart to test before you buy, particularly if blocking any legitimate e-mails would harm your business. At Fenwick & West, Kesner created shadow e-mail boxes for some of the firm\u2019s biggest e-mail users, into which he put duplicates of all of their messages. He then used those shadow boxes to test antispam products. Because some of the language used in the firm\u2019s large commercial transactions\u2014buy, sell, price, dollars\u2014tends to show up in spam, he was dismayed to discover false positive rates of 1-to-1,000 and even as high as 1-to-100. "In our business, every e-mail from clients is really crucial. We can\u2019t block a high percentage of legitimate e-mail," Kesner says. "We needed to be below 0.05 percent, which seemed near unattainable with a filter."After trying out more than 18 antispam products, Kesner decided to go with Postini\u2019s antispam service. With Postini, his false positive rate approaches 1-to-10,000, in part because users can put trusted senders on a whitelist, meaning messages from those senders automatically bypass the filters and get delivered. Kesner\u2019s cautious approach of testing on duplicate messages allowed him to get a real-world read on false positive ratings without worrying about losing any legitimate messages. The Outsourcing Option Kesner\u2019s testing convinced him that the ability to filter out most spam while maintaining an extremely low false positive rate was worth the risk of outsourcing. "I was cautious of an outside service," he says. "But [being an outsourcer] allows them to respond to spam outbreaks faster than their competitors." Sending out a spam update to thousands or millions of remote users is taxing, so spam software makers tend to roll these updates into packages and send them periodically. A service provider can simply add an update to a few servers in a couple minutes and have the update apply to all customers nearly instantly.Postini also lets Fenwick & West IT employees choose how much of each kind of spam they want to filter out by setting filters for each of four subcategories of spam: explicit content, get rich quick, too good to be true and racially insensitive. Kesner pays a per-user fee, which turned out to be about half of what he\u2019d budgeted for. And because he\u2019s now blocking at least 99 percent of incoming spam (5,000 to 7,000 messages a day get trapped on Postini\u2019s servers), Kesner has been able to delay the purchase of four new servers (costing $10,000 to $20,000 each) by more than six months.Indeed, using an outsourcer can be cheaper than managing the spam problem internally. Water Pik, which manufactures personal health-care products, pool products and heating systems, also found that to be the case. "We looked at the cost of doing it internally, and it was staggering," says CIO Wallace Miceli. "We\u2019re talking one or two people full-time," he says. Miceli pays FrontBridge $1.50 per month for each of his 1,000 users, which he says is cheaper than buying and maintaining an onsite filter. Outsourcing, however, won\u2019t work for everyone. Large companies, those with multiple locations whose mail doesn\u2019t all pass through one or two points, and those that use both private and public networks, may find it tricky to outsource. And the obvious downside of outsourcing is that it requires giving someone else the authority to decide what e-mail enters your organization. "For a spam filter to work very effectively, it has to look to a certain extent at the body of the message," says John Mozena, a cofounder of the Coalition Against Unsolicited Commercial Email (CAUCE). "Something\u2014even if it\u2019s just a piece of software\u2014is reading your company\u2019s mail. For some companies, that is not acceptable." Law firms and hospitals, for example, might be wary of exposing confidential client or patient e-mail to a third party. If you choose to outsource, make sure your service provider will give you timely access to quarantined messages. When Rush Enterprises, a truck, construction and farm equipment dealer, tried outsourcing, Rush\u2019s e-mail administrator couldn\u2019t see what was being filtered and therefore couldn\u2019t tell if the company was missing good e-mails. "When you outsource, you generally lose control," says CIO Scott Kressner. If there was a problem, or if a user needed to be able to receive an important message, it took hours or even a day or two to resolve the situation. Kressner ended up purchasing the antispam appliance (a server loaded with the outsourcer\u2019s software that sits in front of the real mail server) and now uses it in conjunction with Symantec Gateway. Although the appliance was more than two or three times the annual cost of the service, Kressner says it\u2019s been well worth it to regain control. A Spam CocktailA year or two ago, subscribing to a list of known spammers (known as a black-hole list or a blacklist), or relying on a signature approach (comparing the patterns in a new message against the fingerprints of known spam messages), or using reverse DNS lookup to check whether the sending domain was legitimate might have worked. But companies can\u2019t rely on just one type of blocking anymore. "I\u2019d strongly argue that you need a spam cocktail\u2014a variety of approaches that work together to generate a probability as to whether a message is spam or not," says Meta\u2019s Cain. The most reliable products and services subject each e-mail to numerous tests that yield a probability score indicating how likely the message is spam. Companies can then set up rules that, for example, delete messages with a spam score of 95 percent or more, quarantine messages in the 85 percent to 95 percent range, and deliver (with a "suspected spam" warning) messages with scores between 75 percent and 85 percent.The managed service provided by FrontBridge, for example, uses the cocktail approach. To make it into a user\u2019s inbox, an e-mail must clear three hurdles. First, its sender can\u2019t be on FrontBridge\u2019s proprietary blacklist. Then it must pass through a spam fingerprinting layer that identifies specific characteristics unique to spam. (For instance, spam often hides a stash of unspammy words in white HTML text on a white background to try to fool filters into thinking it\u2019s real e-mail; legitimate e-mail would not include white-on-white text.) Finally, it\u2019s got to survive a heuristics layer, which involves rule-based scoring. Spamlike behaviors, such as odd characters, spacing or HTML links, earn bad points, which are offset by good points awarded for characteristics that suggest legitimacy. FrontBridge updates 250 of its 10,000-plus rules daily.Although attacking spam on multiple fronts may seem like overkill, Walter Smith can attest that it\u2019s necessary. As director of the global IT infrastructure services group at Advanced Micro Devices (AMD), he calculated that spam was costing the computer chip manufacturer more than $1.5 million a year in lost employee productivity. He first took a crack at handling the problem internally. "Our initial approach was to use fairly simple rules to identify spam and tag junk mail," he says. "We quickly found out that simple rules and spam don\u2019t go together." Before long, two full-time employees were consumed with tweaking the rules to account for all of the variations in spam, and even then, they couldn\u2019t keep up with the spammers. Only about 30 percent of spam was getting tagged, and some legitimate e-mail was wrongly identified as spam.So when AMD\u2019s e-mail firewall vendor announced an antispam product in May, the decision to use it was more or less a no-brainer, says Smith. AMD already used Tumbleweed both to scan all incoming e-mail for viruses and to prevent confidential competitive information from leaving the company. With the Tumbleweed infrastructure already in place, AMD could plug in the vendor\u2019s new spam component for an annual per-user cost of about $5, an investment that paid for itself in less than a month. Today, 90 percent to 95 percent of all incoming spam is tagged as such. And no more than a quarter of a single IT employee\u2019s time is needed for ongoing maintenance."Having a combination of rules, heuristics and blacklists is really key because of the creativity of spammers," says Smith. "Simple, obvious solutions don\u2019t work today. We quickly realized that stopping junk mail is not a core competency of our company. And we needed to get out of that business as soon as we could."In attacking AMD\u2019s spam problem, the last thing Smith wanted to do was to take on the role of corporate censor. "We didn\u2019t want to be perceived as content filterers," he says. In the interest of providing a nonhostile work environment, however, AMD does delete all spam with a high probability of containing adult content. But all other spammy mail gets sent along to users, marked as suspected spam. Users then decide for themselves whether to have Outlook filter all spam, put it in a spam folder, or keep it in their inboxes for manual scanning and deletion. Now that spam is under control at AMD, Smith and his department attained the same herolike status Kesner enjoys. "It\u2019s a huge value IT has delivered to the company, and it\u2019s been huge, positive publicity for IT," he says. Act Now, Think Long-TermLike Smith at AMD, many CIOs would prefer to turn to the same vendor for all of their e-mail security services, including spam filtering, virus protection and denial-of-service protection. "You don\u2019t want a box for virus, a box for spam, a box for content filtering, a box for something else," agrees Maurene Caplan Grey, a research director at Gartner. "You want as few boxes as possible, and you want them to work nicely together with a central console for monitoring."But you shouldn\u2019t blindly sign up for whatever antispam solution your current antivirus provider happens to have, warns Meta\u2019s Cain. He maintains that the spam offerings of many antivirus vendors are antiquated and not updated often enough to keep up with the spam threat. Keeping pace with spammers has become a full-time job; some antispam outsourcers update their rules daily, hourly or even more often if need be. Your best bet is to invest in a spam cocktail approach from a vendor or service provider with a track record of offering frequent updates (which suggests a commitment to staying current in the spam-antispam arms race) and to make sure that it does not conflict with other e-mail security services. (Ideally, all e-mail services should be integrated.)While more than 90 antispam vendors stand ready to take your money today, the market will consolidate to about a dozen serious contenders by mid-2004, Grey predicts. She anticipates that the dozen antispam products that survive will be about equally effective, catching 95 to 98 percent of spam with an 0.5 percent false positive rate, even though they may use different technologies to filter spam. She advises choosing a vendor that supports multiple detection methods, and suggests looking at the extent to which vendors are using adaptive technologies (such as Bayesian filtering) that learn about spam\u2019s characteristics and can take a more proactive approach to blocking it. Even though the antispam market is still maturing, you can\u2019t afford to wait and see how things will shake out. "Spam is too horrible a problem\u2014and it\u2019s going to get more malicious. Two years ago, spam was a little annoyance. If you had a blacklist in place, everything was OK. That\u2019s not the case today," says Grey. "You need to do something right now, even though none of this is completely baked."