IT in the 9/11 Aftermath Difficult

Avi Duvdevani

Acting CIO, City of New York

NEW YORK?Seventeen days after terrorists attacked New York City, Avi Duvdevani is still under siege.

“You can’t get near the building,” says New York City’s acting commissioner of the Department of Information Technology and Telecommunications (DOITT) when asked if he’s accepting visitors to his Brooklyn office. “I can’t get FedEx packages delivered. Armed guards search my car every morning. Every morning.”

And this is an improvement. For the first four days after the Sept. 11 disaster, Duvdevani wasn’t driving to Brooklyn; he was waking up on an Army cot in his office after maybe two hours of sleep. He and the 40 staff members in charge of network operations did nothing but work to restore computing and telecommunications services to the dozens of city agencies that were directly and indirectly affected by the attack. Although numb, then grief-stricken, they were luckier than many. They had a job to occupy their hearts and minds?getting the city running again.

The project is ongoing, but the commissioner pauses to reflect on how the city’s IT recovery plans were?and weren’t?sufficient for a disaster of such unimaginable proportions.

Besides that, Mr. Duvdevani, how do you like your new job?

Duvdevani, a 20-year vet-eran of the city’s IT department, had just begun his fifth workday as its acting commissioner when he looked out the third-floor window of his office, saw the Brooklyn Bridge and saw black smoke rising from Lower Manhattan on an otherwise lovely late-summer morning.

“I thought it was a fire or something,” recalls Duvdevani, 54. “So I turned on the TV set, and I was able to see the second plane hit. I realized right away it was terrorist activity. I did two things: I put on the emergency management radio channel so I could follow what was going on, and I ordered that the website be disabled because I thought we had stuff on the site that could be of aid to people doing damage, like online video streaming of traffic conditions. I assumed we were being invaded.”

Up until that morning, the city of New York, under the reign of Mayor Rudolph Giuliani, was seen as overly ambitious if not utterly paranoid in its planning for the worst. On the 23rd floor of 7 World Trade Center, Giuliani had created the Emergency Operations Center designed to withstand bombs, hurricanes and nuclear assault. Across the river in Brooklyn, DOITT headquarters?which house some emergency resources along with critical mainframes such as those used for tax collection?were built to have redundant sources of water, electricity and telephone services. If destroyed, the data center could be completely replicated in Philadelphia within 24 hours, an expensive service provided by a disaster recovery vendor and tested four times a year.

Fortunately, DOITT’s Brooklyn fortress was not hit. In fact, of the department’s 300 employees, only one was injured (in a traffic accident after the attack). But the Emergency Operations Center in the Trade Center was destroyed, and many city office buildings were damaged, including the city administration office at 75 Park Place where, just a week earlier, the previous DOITT commissioner had emptied his office.

Debris from the falling towers severely damaged Verizon’s central office at 140 West St., next door to the collapsed 7 World Trade Center. The company provided telephone service for many of the 60 agencies supported by the DOITT, and its central office also housed the Internet connection for the city’s website. At press time, Verizon was still working round-the-clock to restore services. The DOITT’s reliance on that facility created much of the work for Duvdevani and his team in the coming weeks.

Always Open

New York City’s website (www.nyc.gov) promises citizens that it’s “always open,” and Duvdevani was determined to keep it that way. After the second plane hit, his staff removed parts of the site that could be useful to attackers and updated the front page to include information about the unfolding crisis.

The Web servers were safe in Brooklyn. The Internet connection, however, was located at the damaged Verizon office, and with electricity out in Lower Manhattan, Duvdevani knew the clock for the backup power supply was ticking. By 10:40 p.m., when the batteries gave out, NYC.gov had been migrated to a contingency Web connection. But that raised different technological issues. The backup connection had a different IP address, one that would take three days to fully propagate across name servers on the Internet. During that time, the site would be difficult to access.

That simply wouldn’t fly. Switchboards were swamped with incoming phone calls, making connections difficult even for citizens who had dial tones. Many broadcast television stations had relied on antennas atop the fallen towers. NYC.gov was the best way to deliver information that just couldn’t wait: where distraught families could go for help, where dislocated city employees should report to work, which schools were open, which trains were running, which bridges and tunnels were open, how citizens could help.

A half-dozen members of Duvdevani’s team worked out a plan for physically rerouting the Internet connection. “That kind of thing would have taken weeks to design, and they worked it out in a few hours,” he says. “We sent in four people in overalls and HEPA filter masks Tuesday night and Wednesday morning, and actually rebuilt the connection in another building.”

By Wednesday NYC.gov was back to normal, except for a few sections kept down for business reasons?for example, online forms where citizens could send messages to officials. “Between Tuesday noon and the following day, the mayor had gotten more than 1,000 messages, mostly of support and prayers,” Duvdevani says. “If we’d have kept the [service] up, he would have gotten maybe 50,000 messages, and that’s how many he gets in a year. His office couldn’t possibly keep up.” As the world knows, Giuliani had plenty to do.

Lessons Learned

Like the army cots that duv- devani and his lieutenants slept on for several nights after the attack, the recovery plan for restoring telecommunications was mostly a holdover from Y2K planning. Its name was MARC?Mutual Aid and Restoration Consortium. Each of the 30 telecom providers in the city is a member, including juggernauts like Verizon and smaller companies that install fiber-optic cable or provide terminating equipment.

Starting Wednesday, the consortium met once or twice a day, via a teleconference, to figure out how best to restore services across the city, juggling both public and private needs. “I would lay out the city’s priorities and then ask which vendors could possibly support us in those locations,” Duvdevani says. “Those decisions were mainly based on where those particular vendors had facilities. If somebody had fiber in the ground within 5 feet of the location, that was an optimum possibility.”

The city Office of Emergency Management gave Duvdevani the list of which city buildings needed voice and data services restored first. What were the priorities? City Hall was first, of course, and was up by Friday, Sept. 14. Second in line was a large municipal building at 1 Center St., which houses among other things the citywide administration services, the department of finances, the controller’s office and the public advocate.

At press time, DOITT has restored phone service to 25 buildings, with still more to go. Network connections will be next after voice lines. “It’s become kind of a game of making some balance judgments between doing something alternative and waiting for Verizon. The question is, Do you want to spend the money to do this, or do you want to wait, if it’s not as critical as some other agencies are,” Duvdevani says.

“The lesson learned here from a technical perspective is that we had too much invested in this one central [Verizon] office,” he continues. “What we’re doing now in terms of the restoration is really going to ameliorate that problem because, say, the municipal building has this alternate source for voice. We’re probably going to leave that up forever. That doesn’t mean we won’t use Verizon?whenever they come back up?but now we have a backup. Maybe that’s the silver lining in this whole thing, if there can be one.”

It will be a long time before things are back to normal, whatever normal looks like. For now, Duvdevani’s team has a daily reminder of the destruction: The 200 employees normally housed at DOITT headquarters have squeezed in an extra 80 IT workers normally stationed at the offices for administrative staff, human resources, and accounts payable and receivable in Manhattan. Like many city employees, their buildings?although still standing?are located in the area of Lower Manhattan that was still off-limits at press time.

“People are sharing cubicles. The bathroom is a mess,” Duvdevani says.

As for the IT team, the work has brought the close-knit group even closer together. “It took a few days for the whole thing to sink in. Because we focus so much on the technology issues, we look at everything through a technology lens,” Duvdevani says. “People were frustrated they couldn’t help dig through the rubble, so they were very willing to do what they knew how to do. It’s a way to transfer the helplessness into something productive.”