San Diego Wildfires: Business and Government Systems Keep Running Despite the Disaster

For Qualcomm CIO Norm Fjeldheim, it’s been anything but a typical working week. On Monday, Fjeldheim became one of the more than 250,000 San Diego area residents driven from their homes by the fiercest set of wildfires to ravage Southern California since 2003.

“I became concerned on Sunday night,” says Fjeldheim. “I was watching the news closely and monitoring e-mail. I was scheduled to fly to San Francisco yesterday, which I canceled. My neighborhood was evacuated at about 10:00 [on Monday].”

Fjeldheim and his family—four people and two dogs—are now camped out in a downtown San Diego hotel, watching the news and following the latest developments in the fire, which has been burning for three days and has destroyed at least 1,000 homes and 100 businesses, according to news reports.

Yet despite the fire, Fjeldheim and Qualcomm were both on the job yesterday. The CIO says all systems are up and running at the San Diego-based maker of wireless communications devices and software. “There have been some intermittent power issues because of the fires, but so far none of our systems have been impacted due to the backup systems we have in place,” he says.

In part, that’s because Qualcomm has an “Operations Readiness” team in place to respond to emergencies and IT is a part of that, says Fjeldheim. The team has been in place for many years: it was originally formed to deal with Y2K in the mid-1990s. Its scope was later expanded to handle all disaster preparedness and responses for Qualcomm.

While Qualcomm’s main IT facilities were not threatened by the fires as of Tuesday morning, Fjeldheim said a number of outlying buildings in the evacuation zones were being closed down as a precaution. All Qualcomm’s systems and buildings can be monitored remotely, which Fjeldheim and his team have been doing for the past 36 hours.

But with media outlets putting the number of area evacuations to as many as 500,000, the ability to communicate with employees is critical, especially when it comes to keeping workers alerted to office closings in potential danger zones. To deal with such an event, Qualcomm has multiple communications systems in place to notify its employees. “We have extensive Remote Access Systems (RAS) in place,” says Fjeldheim, “enough for up to 10,000 employees.” And with staff needing to work remotely due to the blazes, “we can also support a very large number of conference calls for employees,” he adds.

Four years earlier another conflagration swept over the region, and Fjeldheim says the company learned from that experience. “This is the second time we have had to deal with San Diego wildfires,” he says. “After the last fires, we increased our RAS capacity, along with our conference call capacity.”

And while Qualcomm Stadium is housing those displaced by the fire, the company is also helping fight the fire on another front: Fjeldheim says its cell phones are being used by some emergency personnel.

San Diego County CIO William Crowell says he was sitting on Torrey Pines Beach when the first fires broke out on Sunday morning. He says he smelled the smoke when he got home but “I thought it was the neighbor burning something.” However, by late Sunday night, he says, “we knew we had a major issue on our hands.”

At that point Crowell, who had been monitoring the fires on the news, instituted his continuity of operations plan for the IT department as the county activated its Emergency Operations Center. For Crowell and San Diego officials, the good news is that the county’s two data centers are far from the flames: one is in Tulsa and the other is in Plano, Texas, both outsourced to Northrop Grumman.

Still, Crowell has been working 12-hour shifts with other IT team members ever since Monday to support and enhance the county’s website and its 211 emergency hotline system. While the fires and ensuing evacuations have meant some of his staff have been unable to report to work, “We’ve been OK. I have a core team of key people who are coordinating various activities.”

From Crowell’s perspective, “our two biggest issues are to support the dissemination of information to the public through our Web and 211 services.”

The County’s website, which got about 400 visitors a day before the fires, saw its volume swell to more than 500,000 visits on Monday as residents sought emergency information about the spreading blazes. To support the increase in visitors and to speed performance, Crowell’s team made technical changes on the fly to separate the emergency part of the county’s website and put it on a separate server. IT had originally planned to re-architect the site in December to improve its ability to deal with dramatic volume increases; Crowell says it he may just do it now. “When you outsource, you can marshal the resources to get something like this done,” he says.

Demand also initially overwhelmed the county’s “2-1-1” service, an emergency hotline for fire and related information, says Crowell. However, Crowell’s team was able to quickly scale the system up from handling 24 concurrent callers to taking 237 simultaneous calls. “Wait times are becoming reasonable again,” he says, noting that they had been as long as 15 to 20 minutes during the recent peak.

“The principal issue for IT is the ability to leverage up your capacities. You test your assumptions and you see what works and what didn’t. And we have done a fairly good job,” says Crowell, who notes that others, including AT&T and some ISPs, have also been initially overwhelmed by the public’s use of their infrastructure in the wake of the fires.

In a disaster, “everything get overwhelmed,” he says. “So everyone is facing the same issues.”

Related reading

• Lessons from hurricane Katrina: It pays to have a disaster recovery plan in place
• Business continuity and disaster recovery planning: The basics
• MIT’s Yossi Sheffi: How to recover from nearly any disaster
• Three Wall Street IT execs assess life after 9/11