Tale of a Virus Attack

IN A MAKESHIFT BASEMENT WAR ROOM outside New York City, the CIO of a global retailer was watching things go from bad to worse. Hundreds of computers were infected with a stealth virus not recognized by the latest antivirus software. The primary network at its U.S. headquarters had been shut down since 10 a.m., when she realized there was no other way to stop the virus from spreading. Now it was lunchtime, but the hastily formed IT SWAT team wasn’t eating. They were troubleshooting infected computers and realizing the full extent of the damage.

The virus, Demiurg, was on a rampage?destroying files and programs, eluding SWAT team researchers by mysteriously re-creating itself on each reboot. A sickening thought crept into the CIO’s mind: Her staff might have to rebuild every infected computer.

“We weren’t going to take any chances,” says the CIO, who we’ll call Jane Smith. She acted swiftly: 400 PC users who weren’t running critical programs on the mainframe needed to turn off their computers until further notice. The intercom blasted the order. The SWAT team posted flyers. Managers at a satellite office patrolled hallways delivering the news.

It would be five days before any of those users were back up and weeks before the IT troops were free to work on anything but recovery. A month later, Smith would still be reeling, with business loss estimates of $250,000 to submit to the FBI, a shell-shocked staff worried about credibility and a management team that thought the problem was long over.

Every month, three to seven businesses contact McAfee’s Anti-Virus Emergency Response Team (AVERT) like this company did, reporting a new virus that sometimes takes down hundreds or even thousands of users for days. The Carlsbad, Calif.-based research company Computer Economics reports that in 2000, computer viruses caused $17.1 billion in damage worldwide. None of that helped our CIO, though, because how businesses combat malicious attacks is a dirty, not-so-little secret. Smith, whose company has a policy of not talking to the press, agreed to tell her story on condition of anonymity, in hopes that it might help others when they come under attack.

Tuesday, Jan. 9: First Infection

Something was amiss. About 20 employees at a satellite office were working on a spreadsheet distributed by e-mail. The document was misbehaving on all but one of their computers. Smith was visiting the office on other business, and she identified symptoms typical to a macro virus. To stop it from spreading, she told the infected users to log off the network. She headed home thinking her staff would tell employees to update their virus definition files.

Wednesday, Jan. 10: The Virus Spreads

First thing in the morning, the infection spread from dozens of computers to hundreds. Unknown to the IT staff, when users booted up their computers, the virus executed itself and started generating error messages on most PCs. The staff e-mailed instructions for everyone to look for Demiurg, the virus that was not yet recognized by the latest version of McAfee’s antivirus software. Anyone who found it was to call the help desk.

Smith, who had been out of the office, had not logged on to the network since Friday. In a sequence of events that would later prove critical, she turned on her computer at 8:40 a.m., checked her e-mail, looked at a spreadsheet sent on Tuesday morning to all the company’s executives, searched her hard drive for the virus, found it and turned off her computer by 8:55 a.m.

At the help desk, the phone wouldn’t stop ringing.

By 10 a.m., Smith shut down the network to contain the virus. That meant no e-mail for U.S. employees, no remote access for mobile users, no connection to offices in other countries, and no communication with stores, which could still ring sales and process credit card transactions but could not look up customer data or inventory at other locations. This meant lost sales.

Smith started to assemble an emergency task force of 40 people: the 15 members of her staff who took care of the network, support services and data center, plus trusted consultants. It felt like combat, so she called them a SWAT team and their windowless basement gathering place the war room. “At this point,” she says, “I’m still thinking, ’It’s not that bad.’”

But it was. By noon, when Smith and her team realized how damaging the virus was, they told everyone at headquarters to turn off their computers. The only exceptions were 50 employees on a secondary network that hadn’t been infected and users running critical programs on the AS400 mainframe, which controlled shipping and inventory. (The PCs running off the mainframe were vulnerable, but the mainframe itself was not. AS400s historically have not been harmed by PC viruses.)

Smith’s researchers found some of the virus’s Visual Basic code with the header, “A Win32 virus by Black Jack written in Austria in the year 2000.” They e-mailed the code to AVERT, McAfee’s virus research lab. Still, they kept running into brick walls. “You’d think you’d get it, and then you would reboot and it would come back,” Smith says. Her researchers had a lot to learn about Demiurg (named for the creator of the world in Platonic philosophy), a stealth virus that spreads through Microsoft Excel spreadsheets. When a user opens an infected spreadsheet, the virus infects the Windows Kernel32.dll file, a fundamental part of the operating system. When the computer is rebooted with the infected Kernel32.dll file, the virus spreads to executables and batch files, corrupting so many files that the computer eventually stops working.

These were the worst hours for Smith?when hopes for conquering the virus were dashed again and again. “By 8 [p.m.], it was clear that not only was this going to be a long night but that the next couple of days were going to be quite rough,” she says. “At this point, we’re frantic, because we know we’re down, and we’re down hard, and we know we have no way to bring the users back up because we have no way to block it from spreading.”

It didn’t matter that tables in the hallway were piled with food and sodas. The IT troops were terrified that they might have to rebuild every single machine. “Even revisiting it now, my stomach is all messed up,” says the network services manager. “It was one of the scariest points in my life.”

SWAT team members were desperate to hear from McAfee’s help desk, which according to Smith had not contacted them regarding the virus code sent eight hours earlier. They called the help desk and demanded to speak with a manager, who said Smith’s company needed to wait 24 hours for a response. “As far as we were concerned, we were getting no help from them,” Smith says. McAfee AVERT Director Vincent Gullotto responds that once his lab received infected files or pieces of the code, researchers would have gotten to work. Meanwhile, Smith was keeping management updated. “They knew that we were doing everything we could,” she says. “I was very careful to explain it to them in layman’s terms.” But Black Jack’s header had her thinking, and she asked other company management if she could call the FBI. They approved.

She called information to get the phone number for the nearest FBI office. At first, the person who answered the phone at the FBI wasn’t sure if the case fell under the bureau’s domain, but the more he heard, the more interested he became. Lost business. More than $5,000 damage. Stores affected in more than one state. With Smith’s help, the FBI started building a case, in hopes of bringing the perpetrator to trial.

After midnight, Smith remembered what she had done a long 16 hours earlier. She brought down her laptop, and by walking researchers through her steps, they were able to confirm that she had gotten the virus from the spreadsheet sent to executives. They also found the rest of the Visual Basic code?”the final piece of the puzzle,” she says.

The SWAT team sent McAfee the rest of the code using a Hotmail account on a standalone computer and called it a night. “We all drove home at 2 o’clock that morning listening to the news, and there was nothing,” Smith says. They wondered, wasn’t this as big and bad as the “I Love You” virus?

Thursday, Jan. 11: Containment

By 8:30 a.m. the war room was full of bleary-eyed coffee drinkers with a new hero. Karan Bhagat, a senior engineer from Alliant Technologies in Morristown, N.J., had gone back to his office, slept on the couch for an hour and then got up and found a way to stop the virus. The block, a DOS program beautiful in its simplicity, created three empty directories, each with the same name, where Demiurg wanted to install itself. This prevented the virus from becoming active.

The SWAT team started burning the containment program onto CD-ROMs, handing each fresh copy to somebody new for testing. By avoiding the first impulse to move as quickly as possible, “we were able to find some problems before we had 20 technicians running around the building all with the same questions,” Smith says.

They split into teams, each one responsible for a section of users on the company phone list?a list that would evolve into a spreadsheet with the name, location and computer status of every employee. “Everybody came here and got their marching orders, and we had a plan of attack. We did nothing haphazardly,” Smith says. Each team had a single point of contact back in the war room and at least one staff member who knew the terrain. The troops marked each computer that had the block with a fluorescent green sticker. The block was an important step to recovery, because users running critical programs on the mainframe and those on the secondary network could continue to work without fearing the virus.

That afternoon, McAfee developers sent a first attempt to find Demiurg and clean infected files. The “fix” found the virus but destroyed some files. (McAfee’s Gullotto responds, “I don’t know if that did happen, but there is a possibility that if the cleaning is not done correctly that it could.”) Disheartened, Smith’s researchers came up with step two of what would be called the “big fix”: a way to clean individual spreadsheets. Technicians used Grep, a Unix utility that works in DOS, to search for a text string that identified Demiurg. When they found an infected spreadsheet, they spent from five to 10 minutes manually removing the virus code. Technicians started returning to each desktop, marking each one with a second green sticker, and also began locating and cleaning infected files on the e-mail server.

McAfee’s Gullotto was surprised that Smith and her team took matters into their own hands, but Smith’s beleaguered company had simply lost faith. “We had no idea when McAfee was going to come through for us,” Smith says, exasperated. “What were we going to do?sit around?”

Friday, Jan. 12: Disinfection

The days had blurred into a bad dream. “You have technical people who are driven by a challenge and wanting so much to be involved in something this unbelievable, and being sick to their stomach at the same time,” Smith says. Tensions were escalating outside of IS too. One user locked herself in her office to use her computer. A technician alerted Smith. “I called her and lost it,” Smith recalls. “I said, ’I have a SWAT team in there of 40 people, and you are going to circumvent the process? Don’t you dare.’” The virus did so much damage that SWAT team members had to later rebuild the errant user’s computer. McAfee’s attempts to detect, clean and fix what the virus had done were getting better. “By now, we knew what [Demiurg] did so well that as soon as they would give us a version, we’d do the clean and then we would run X, X, X and X, and we’d say, ’No, it only works up to this point,’ and give them that back. We worked like this,” Smith says, holding up her crossed index and middle fingers. Gullotto says companies hit by new viruses are often involved in testing because they have a larger testing environment or need any kind of fix as soon as possible. Friday afternoon, McAfee sent a fix that worked and the SWAT team skipped the rest of the homegrown clean.

Now that they had the final solution, Smith had a different problem: Should she start letting some users on? She called the president, who desperately needed e-mail, and asked, “What do you want to do? I can let you in only, because the server’s clean and the e-mail’s clean, or we can just say that all users will come up Monday morning.” The president opted to wait. The troops spent the rest of Friday, all day Saturday and half of Sunday returning to all 400 PCs. This time, they taped signs to clean computers warning users not to power up until they got the OK.

The Next Week: Recovery

Monday morning at about 10 a.m., the intercom finally blared good news: Employees could use their computers again. Stores could communicate. European and U.S. locations were no longer an ocean apart. (European locations, which were warned about the virus by fax, only had a few infections.)

For IS, however, the war wasn’t over. A dozen computers were so corrupted that technicians needed to completely reinstall the operating systems and applications. There were details to take care of with the servers and at stores, and 100 mobile users needed to overnight their laptops to headquarters for fixing.

The virus had also damaged morale. Afterward, an IS staff member told CIO that he felt guilty for not somehow preventing the virus. Smith echoed his anguish. “I have a very good staff. If a server goes down, they’ll stay through the night and rebuild it. I mean, the users are up. So for us to have the users down for so long was very difficult on my staff?feeling like, ’Do people really understand?’ And I kept assuring them that they did understand.”

Smith doesn’t think she or her staff lost credibility because they didn’t furrow their brows and hide away. “It was extremely visible every minute of every day that we knew exactly what we had to do.” She had no emergency task list from a consultant, no disaster recovery plan telling her what to do next?only a few pointers from the company’s year-old Y2K plan and her gut instinct. It also helps that no data was lost. “Isn’t that amazing?” Smith asks. “It’s amazing. The [virus clean that McAfee] ended up writing based on our feedback was able to clean everything. Now believe me, we also had backups, but that [would have been] a whole other effort.”

The Aftermath

Forty-one days after the attack, when CIO visited her office, Smith got her first glimpse of the person behind the mystery she was still unraveling?a 19-year-old Austrian Red Cross worker. “Oh my God. It’s like a serial killer,” Smith says, her laugh hollow as she read printouts from Black Jack’s webpage. The FBI had told her less about him than a reporter could find with 30 minutes of Internet searches.

When contacted by CIO, Black Jack took full credit for “the Demiurg” but defended himself, saying he only writes and publishes viruses on the Internet and can’t prevent others from downloading and releasing them into the wild. “I don’t like the fact that someone might have problems because of a virus of mine, but I don’t feel guilty,” he wrote in an e-mail. “The guilty one is the person that downloaded them.” (See “Conversation with a Virus Coder” at www.cio.com/printlinks.)

That distinction doesn’t matter to FBI spokesperson and Special Agent Sandra Carroll, who says the FBI is “hopeful” that an arrest will be made. However, Carroll could not comment on an active case. The conversation always wound back to statements like “I decline to comment,” making it easy to understand why Smith was frustrated by how little the FBI told her. “They’re not big on details, believe me,” Smith says, although she doesn’t regret contacting them.

To the rest of the world, Demiurg was just another virus that never made the evening news, in part because Smith’s company moved quickly to stop it from spreading. Smith still didn’t know how the virus reached her company. She did know she had lots to do. Motioning at a file folder with business loss estimates to complete for the FBI, she says she was still deciding, for instance, how to measure lost productivity of employees who were without computers during the busiest time of year. (See “The Cost of the Virus,” Page 80.) Then all she could do was wait, and try to figure out how her department could have recovered more quickly.

Short of forbidding e-mail attachments, Smith believes she couldn’t have prevented this from happening, and neither can you. “That alone is why I want the story to be told,” she says. “There are things out there you can’t protect yourself from. This could hit you.” Someday, in a makeshift basement war room, you may be the CIO watching things go from bad to worse.