Avi Duvdevani Acting CIO, City of New York NEW YORK?Seventeen days after terrorists attacked New York City, Avi Duvdevani is still under siege."You can\u2019t get near the building," says New York City\u2019s acting commissioner of the Department of Information Technology and Telecommunications (DOITT) when asked if he\u2019s accepting visitors to his Brooklyn office. "I can\u2019t get FedEx packages delivered. Armed guards search my car every morning. Every morning."And this is an improvement. For the first four days after the Sept. 11 disaster, Duvdevani wasn\u2019t driving to Brooklyn; he was waking up on an Army cot in his office after maybe two hours of sleep. He and the 40 staff members in charge of network operations did nothing but work to restore computing and telecommunications services to the dozens of city agencies that were directly and indirectly affected by the attack. Although numb, then grief-stricken, they were luckier than many. They had a job to occupy their hearts and minds?getting the city running again.The project is ongoing, but the commissioner pauses to reflect on how the city\u2019s IT recovery plans were?and weren\u2019t?sufficient for a disaster of such unimaginable proportions.Besides that, Mr. Duvdevani, how do you like your new job?Duvdevani, a 20-year vet-eran of the city\u2019s IT department, had just begun his fifth workday as its acting commissioner when he looked out the third-floor window of his office, saw the Brooklyn Bridge and saw black smoke rising from Lower Manhattan on an otherwise lovely late-summer morning."I thought it was a fire or something," recalls Duvdevani, 54. "So I turned on the TV set, and I was able to see the second plane hit. I realized right away it was terrorist activity. I did two things: I put on the emergency management radio channel so I could follow what was going on, and I ordered that the website be disabled because I thought we had stuff on the site that could be of aid to people doing damage, like online video streaming of traffic conditions. I assumed we were being invaded."Up until that morning, the city of New York, under the reign of Mayor Rudolph Giuliani, was seen as overly ambitious if not utterly paranoid in its planning for the worst. On the 23rd floor of 7 World Trade Center, Giuliani had created the Emergency Operations Center designed to withstand bombs, hurricanes and nuclear assault. Across the river in Brooklyn, DOITT headquarters?which house some emergency resources along with critical mainframes such as those used for tax collection?were built to have redundant sources of water, electricity and telephone services. If destroyed, the data center could be completely replicated in Philadelphia within 24 hours, an expensive service provided by a disaster recovery vendor and tested four times a year.Fortunately, DOITT\u2019s Brooklyn fortress was not hit. In fact, of the department\u2019s 300 employees, only one was injured (in a traffic accident after the attack). But the Emergency Operations Center in the Trade Center was destroyed, and many city office buildings were damaged, including the city administration office at 75 Park Place where, just a week earlier, the previous DOITT commissioner had emptied his office. Debris from the falling towers severely damaged Verizon\u2019s central office at 140 West St., next door to the collapsed 7 World Trade Center. The company provided telephone service for many of the 60 agencies supported by the DOITT, and its central office also housed the Internet connection for the city\u2019s website. At press time, Verizon was still working round-the-clock to restore services. The DOITT\u2019s reliance on that facility created much of the work for Duvdevani and his team in the coming weeks. Always OpenNew York City\u2019s website (www.nyc.gov) promises citizens that it\u2019s "always open," and Duvdevani was determined to keep it that way. After the second plane hit, his staff removed parts of the site that could be useful to attackers and updated the front page to include information about the unfolding crisis.The Web servers were safe in Brooklyn. The Internet connection, however, was located at the damaged Verizon office, and with electricity out in Lower Manhattan, Duvdevani knew the clock for the backup power supply was ticking. By 10:40 p.m., when the batteries gave out, NYC.gov had been migrated to a contingency Web connection. But that raised different technological issues. The backup connection had a different IP address, one that would take three days to fully propagate across name servers on the Internet. During that time, the site would be difficult to access.That simply wouldn\u2019t fly. Switchboards were swamped with incoming phone calls, making connections difficult even for citizens who had dial tones. Many broadcast television stations had relied on antennas atop the fallen towers. NYC.gov was the best way to deliver information that just couldn\u2019t wait: where distraught families could go for help, where dislocated city employees should report to work, which schools were open, which trains were running, which bridges and tunnels were open, how citizens could help.A half-dozen members of Duvdevani\u2019s team worked out a plan for physically rerouting the Internet connection. "That kind of thing would have taken weeks to design, and they worked it out in a few hours," he says. "We sent in four people in overalls and HEPA filter masks Tuesday night and Wednesday morning, and actually rebuilt the connection in another building."By Wednesday NYC.gov was back to normal, except for a few sections kept down for business reasons?for example, online forms where citizens could send messages to officials. "Between Tuesday noon and the following day, the mayor had gotten more than 1,000 messages, mostly of support and prayers," Duvdevani says. "If we\u2019d have kept the [service] up, he would have gotten maybe 50,000 messages, and that\u2019s how many he gets in a year. His office couldn\u2019t possibly keep up." As the world knows, Giuliani had plenty to do.Lessons Learned Like the army cots that duv- devani and his lieutenants slept on for several nights after the attack, the recovery plan for restoring telecommunications was mostly a holdover from Y2K planning. Its name was MARC?Mutual Aid and Restoration Consortium. Each of the 30 telecom providers in the city is a member, including juggernauts like Verizon and smaller companies that install fiber-optic cable or provide terminating equipment.Starting Wednesday, the consortium met once or twice a day, via a teleconference, to figure out how best to restore services across the city, juggling both public and private needs. "I would lay out the city\u2019s priorities and then ask which vendors could possibly support us in those locations," Duvdevani says. "Those decisions were mainly based on where those particular vendors had facilities. If somebody had fiber in the ground within 5 feet of the location, that was an optimum possibility."The city Office of Emergency Management gave Duvdevani the list of which city buildings needed voice and data services restored first. What were the priorities? City Hall was first, of course, and was up by Friday, Sept. 14. Second in line was a large municipal building at 1 Center St., which houses among other things the citywide administration services, the department of finances, the controller\u2019s office and the public advocate. At press time, DOITT has restored phone service to 25 buildings, with still more to go. Network connections will be next after voice lines. "It\u2019s become kind of a game of making some balance judgments between doing something alternative and waiting for Verizon. The question is, Do you want to spend the money to do this, or do you want to wait, if it\u2019s not as critical as some other agencies are," Duvdevani says."The lesson learned here from a technical perspective is that we had too much invested in this one central [Verizon] office," he continues. "What we\u2019re doing now in terms of the restoration is really going to ameliorate that problem because, say, the municipal building has this alternate source for voice. We\u2019re probably going to leave that up forever. That doesn\u2019t mean we won\u2019t use Verizon?whenever they come back up?but now we have a backup. Maybe that\u2019s the silver lining in this whole thing, if there can be one."It will be a long time before things are back to normal, whatever normal looks like. For now, Duvdevani\u2019s team has a daily reminder of the destruction: The 200 employees normally housed at DOITT headquarters have squeezed in an extra 80 IT workers normally stationed at the offices for administrative staff, human resources, and accounts payable and receivable in Manhattan. Like many city employees, their buildings?although still standing?are located in the area of Lower Manhattan that was still off-limits at press time."People are sharing cubicles. The bathroom is a mess," Duvdevani says.As for the IT team, the work has brought the close-knit group even closer together. "It took a few days for the whole thing to sink in. Because we focus so much on the technology issues, we look at everything through a technology lens," Duvdevani says. "People were frustrated they couldn\u2019t help dig through the rubble, so they were very willing to do what they knew how to do. It\u2019s a way to transfer the helplessness into something productive."