For Barry L. Woolsey, CIO of Fleet Credit Card Services, the first public test of his plan for dealing with security incidents began not with a hacker, a cyberterrorist or a disgruntled employee. It started when a curious customer logged on to Mycard.Fleet.com to make a credit card payment?and inadvertently discovered a security hole that would earn headlines on MSNBC.com and force Fleet to shut down its site for five hours.Logging in on a Friday afternoon, Jonathan Bryce, a 20-year-old online operations manager, noticed that the webpage where he accessed his credit card account had a long identification number in the URL. Wondering how Fleet kept track of transaction history, he entered a random number. To his shock, he pulled up someone else\u2019s transaction."The hole allowed you to see people\u2019s personal information," says Bryce, who works for Rackspace Managed Hosting in San Antonio. "Mainly it was nonsensitive information, but there were several cases where it was personal information like Social Security numbers, account numbers and addresses."Alarmed that his account information could be compromised, Bryce called the customer service line at Fleet. After being transferred several times, he was told that someone would call him back on Monday.That wasn\u2019t good enough. Bryce called the media, and hours later both MSNBC.com and The Boston Globe online had picked up the story.Behind the scenes at Fleet Credit Card Services, the FleetBoston Financial subsidiary north of Philadelphia, Bryce\u2019s phone calls and the ensuing media coverage set in motion an incident response plan long in the making. It took more than five hours from the time Bryce called Fleet to the time Woolsey found out. So while Woolsey is generally happy with the way the plan unfolded, he acknowledges that it could have worked better?a whole lot better. At least Fleet had a plan. Throughout most of the business world, incident response planning is one of those best practices that rarely gets done. That\u2019s because of its vague underlying assumption: Something could go wrong.Obviously, prevention is key. But in a world where security is nothing if not fallible, knowing how to respond to a security incident?be it a computer worm, mistake, hacker or the mere suspicion of a problem?can save a company time, money and even its reputation. Indeed, a better response from the customer service representatives at Fleet could have kept the problem from making headlines in the first place.Nevertheless, the sad fact is that Fleet handled the situation better than many companies could have. Outside of the heavily regulated health-care and financial services industries, experts say, few companies are prepared to deal with isolated security incidents, let alone calculated attacks from organized crime or even, perhaps, cyberterrorists. (For more on this, read "The Truth About Cyberterrorism," Page 66). "There\u2019s only now the beginning of awareness that this thing is needed," says Jay Ehrenreich, senior manager in the cybercrime prevention and response group at PricewaterhouseCoopers in New York City. "Companies do come to us to help them develop incident response plans, but lots of times it\u2019s only after they\u2019ve been burned."And getting burned is expensive. Companies have spent billions of dollars recovering from malicious worms such as Nimda and Code Red. And in a 2001 study by the Computer Security Institute and the FBI, respondents who could put a dollar amount on the cost of a security breach averaged more than $2 million in financial losses.Not everyone has to learn the hard way, though. Here are the steps to take to jump-start the process with less time and effort than you might expect. 1 Pull together a team. As long as upper management is serious about security, the first step of incident response planning?pulling together an incident response team?costs next to nothing. "Many companies envision this fully dedicated, highly paid SWAT team doing nothing, waiting for an emergency," Ehrenreich says. "It\u2019s really not that way. It\u2019s almost like a volunteer fire department, where people have other duties, but when there\u2019s an emergency people respond to it."The night Fleet learned of its security breach, Woolsey, COO Susan Gleason (who is no longer with the company), CEO Patrick J. Coll, and representatives from the legal team, communications department and FleetBoston gathered on a preestablished bridge telephone line to discuss how to minimize the damage to Fleet\u2019s customers and its reputation. Meanwhile, the IS team worked to analyze the security breach, find out what information had been accessed and fix the security hole. Based on the IS team\u2019s findings, the fraud department then called customers whose personal information had been compromised.That\u2019s pretty typical of the groups that need to come together after a security breach, although some organizations would have added human resources to the mix. The list of team members can get long, but not everyone needs to be involved with every situation. Also, some people are responsible for fixing the problem; others just need to know what\u2019s happening. The incident response group will be closely aligned with or perhaps even the same as the business continuity or disaster recovery teams.Once team members know their role, a project manager should create a list with multiple ways to contact team members 24\/7. The list should also include contact information for security vendors, Web hosting companies and other relevant technology providers, and it should be available in hard copy at every business location and at people\u2019s homes. The need for that last recommendation was painfully illustrated on Sept. 11.2 Coordinate your efforts. The plan can\u2019t stop with that group. As the Fleet example illustrates, everyone across the organization?right down to the newest person at the call center?needs to know how to react to a potential security breach. That\u2019s why there needs to be a centralized way to report, respond to and track incidents. Customer service representatives need to know who to call about a problem; security employees need to know when to call the CIO; the CIO needs to know when to call the CEO; and someone needs to track what\u2019s going on organizationwide so that different business units can prepare for attacks. "A good bit of incident response is coordination," says David Nelson, deputy CIO at NASA, who\u2019s in charge of the agency\u2019s IT security. "You want to be sure that the right people have the right information at the right time."A security incident at any of NASA\u2019s centers is first reported to an onsite IT security manager. The incident is put into one of seven categories ranging from a legitimate user misusing the system to an access problem in which a hacker has obtained the password of a systems administrator. Anything deemed serious is reported immediately to the centralized NASA Incident Response Center (NASIRC), which shares this information with Nelson. If the incident could be criminal in nature, NASA\u2019s inspector general\u2019s office also gets involved.During large-scale attacks such as the Code Red worm, NASIRC sets up a conference call with IT security managers from all the centers. "We would in real-time go over what\u2019s happening, what\u2019s the damage, what\u2019s being done, and do we have to take any extraordinary measures to protect ourselves," Nelson says. "If we didn\u2019t have [this system] in place, we\u2019d be like sitting ducks without any means of finding out that the hunters are shooting at us. Our experience is that any substantial attack hits a lot of computer systems or IP addresses or places almost simultaneously, and so early awareness can help us take appropriate measures before we\u2019re all dead. That first gun goes off, and all the ducks fly out?maybe one got killed, and we\u2019re sorry about that, but the rest of them are all alive."3 Grant authority. One of the most political parts of incident response planning?but one that can save precious time if an attack is successful?is deciding ahead of time who\u2019s in charge of incident response and which people could pull the plug on the website or network if need be. But the fact that some people high on the corporate food chain have to relinquish control causes friction, says Rebecca Bace, a National Security Agency alum who wrote Intrusion Detection. Unfortunately, there simply might not be time to contact everyone."The velocity of [an intrusion] proceeds in seconds or minutes, rarely hours. If management wants people to be able to respond effectively to quick-moving attacks, they\u2019ve got to empower them to shut certain portions of the system down," says Bace, who is now a faculty member for The Institute for Applied Network Security in Waltham, Mass.The Georgia Student Finance Commission in Tucker, Ga., learned that the hard way?but in slow motion. Bill Spernow, the commission\u2019s new chief information security officer (CISO), was hired after a public security snafu in which personal information about HOPE scholarship recipients was disclosed on the Internet. The organization was painfully slow in responding to the breach, caused by a technical error, because there was no one in-house who could take control of the situation and shut down the website. Now Spernow is in control."You can do this [incident response planning] on the cheap as long as senior management has come to the decision that they\u2019re going to delegate this to one person who can make a decision," says Spernow, a former Gartner analyst who says he took the job because he wanted to apply his advice in the real world. "That will always be the biggest bottleneck whether you\u2019re a Mom and Pop or a Fortune 100. It ends up being a very intense political discussion that brings in lots of fears and control issues. And once you\u2019ve done that, you\u2019ve handled 60 percent of the problem."4 Ready your I.T. resources ahead of time. No doubt about it, the technical aspect of intrusion response is the most complicated part. The IS team or an outsourced monitoring service has to be able to identify problems when they do occur?by examining logs for unusual behavior, looking for vulnerabilities, watching for faulty configurations and monitoring intrusion detection systems, which can generate a tremendous number of false positives. "You have to know when someone\u2019s attacking you because if you don\u2019t know, you can\u2019t do anything. And that\u2019s the hardest part," says Michael Young, CISO and vice principal of State Street Global Advisors in Boston.Once a problem is identified, IT staffers need to be able to look at system logs and analyze exactly what happened. Finding and preserving evidence is the key to fixing a problem and keeping it from happening again. Also, knowing exactly which files were compromised will help a company ensure data integrity and figure out which customers, employees or business partners may be affected. (To learn more about computer forensics, read "IT Autopsy," March 1, 2001, available at www.cio.com\/printlinks.)Whether this can be done in-house or needs to be outsourced depends on the size of the company, how attractive a target it is to hackers and how sensitive its information is. As a general rule, the more crucial and extensive a company\u2019s information assets, the more sense it makes for executives to keep security operations in-house.Fortunately, though, the more a company works to prevent security problems in the first place, the cheaper it will be to deal with them once they do arise. "An ounce of prevention is worth a pound of cure, and that absolutely applies in the cyberintrusion space," says Bruce Moulton, former CISO at Fidelity Investments and a cofounder of the Financial Services Information Sharing and Analysis Center. 5Decide when to involve law enforcement. Reluctant to call the cops and risk having customers and stockholders find out about a problem? That\u2019s normal. But if a situation gets bad enough?a key competitor steals intellectual property, a hacker publishes customer records or a former employee takes down the website?you might change your mind. And in some cases, publicity will ensue anyway. Executives can save time and heartache by discussing ahead of time what situations might cause them to call in law enforcement. For most companies, these situations involve serious financial loss that law enforcement may help recover in damages, or a high-profile case that\u2019s already publicly known.At the University of Washington Academic Medical Center in Seattle, which suffered a security breach that was reported by The Washington Post and other news outlets, CIO Tom Martin initially decided not to call the FBI because he didn\u2019t think the hacker had accessed any patient data. But as soon as he discovered otherwise, he says, he changed his mind."There is a stigma to being a victim because it implies some level of incompetence, but really it\u2019s an inevitability. As we become more Internet-based, it\u2019s only a matter of time before you have problems," Martin says. "To underreport is a disservice to the industry. It\u2019s building in an infrastructure of illegal activity, and we just decided we won\u2019t tolerate illegal activity or accept it as a given."The threat of organized cyberattacks or even cyberterrorism has made this even more of a concern, many experts say, because law enforcement needs to be able to watch for trends or attacks that spread across companies and industries. For research purposes, Carnegie Mellon\u2019s CERT Coordination Center tracks security incidents and asks that companies report even unsuccessful attacks. (For more information, visit www.cert.org.)Increasingly, companies are also being encouraged to contact law enforcement outside of a particular security event. By talking with law enforcement ahead of time, a CIO or CISO is better equipped to evaluate the impact of an investigation on business operations, determine whether law enforcement might be able to help and find out how sensitive information is handled by the authorities."When we get the call [after a security incident] has already happened, it\u2019s difficult for us to help," says Bob Weaver, head of the Secret Service\u2019s New York City Electronic Crimes Task Force, which was made anational model for information sharing between law enforcement and business by the U.S.A. Patriot Act, passed by Congress shortly after Sept. 11. "Call who best serves you. It may not always be federal?you may be best served by a local police department, a state police department, an attorney general. If you don\u2019t have [that relationship] and an incident does arrive, it\u2019s like driving your car down the street and trying to change the tire while you\u2019re doing it. And you don\u2019t want to call 1-800-Law-Enforcement. You don\u2019t want to dial 911. You want to know who you\u2019re bringing in." (For a list of law enforcement contacts, see www.cio.com\/security\/response.) Once in place, incident response plans are morphing, living documents. As Fleet Credit Card Services learned, there are things to be tested and tweaked and learned from?processes that become better and easier to justify to management after they\u2019ve been used. So the next time that organization has a security incident?be it a malicious worm or something more ominous?the company will be that much better prepared. CIO Woolsey is working on making sure customer representatives know how to handle security calls from alarmed customers, and since the security hole was in a vendor\u2019s application, he\u2019s also instituting tighter controls around third-party software. He\u2019s heartened by the fact that to the best of his knowledge, no customers have closed their account because of the incident."I guess the one lesson I would walk away with is never assume that anything is perfect," Woolsey says. "What you need to do is understand that you can\u2019t plan for everything, so you need to have a contingency that\u2019s broad enough and open enough that it can in essence deal with everything."