Lost in all the drama is what actually happened. How could a large city government apparently lose control of its network, and how could its own characterizations of the system be so questionable?

[ Follow the Terry Childs saga blow by blow in InfoWorld's special report. | Read the actual court documents. ]

I've been covering this case in my blog almost since day one, and have been trying to figure out exactly what happened, reading between the lines of published articles, and reading court documents until the wee hours of the morning. Here's what seems to be true, what is clearly open for question, and what lessons business IT should draw from this saga.

First, despite the many news reports claiming that Childs had shut down all or part of the city and county of San Francisco's network, what actually happened was that Childs refused to provide his superiors the passwords to the city's core FiberWAN network, effectively preventing them from administering the network. The network continued to function, and no city applications, data, or resources were lost, unavailable, or otherwise inaccessible.

Just who is Terry Childs, and why was he so powerful?

Terry Childs, a Cisco Certified Internetworking Engineer (certification number 14018), was a member of the San Francisco DTIS, the city's IT department, for the past five years. As a CCIE, Childs shares this distinction with only 16,000 or so others across the globe. He was part of the group that built and managed the city's networks, and in the past several years had been tasked with bringing together the many disparate networks that ran the city. As the city's most experienced and advanced network administrator, he essentially single-handedly designed and built the FiberWAN, a city-wide network built on fiber interconnects and MPLS. This network is complex, and forms the core of all city services.

Following the completion of the FiberWAN, Childs looked upon his creation as art—so much so that he applied and was granted a copyright for the network design as technical artistry. Skeptical of his colleagues' abilities, Childs became the sole administrator of the FiberWAN, and the only person with the passwords to the routers and switches that comprised the network. This state of affairs was widely known throughout DTIS, and Childs was the only point of contact for changes, troubleshooting, and overall management of this network.

Sources have stated that not only was Childs the only admin, he was always on call, 24 hours a day, 7 days a week, 365 days a year. As the only admin with the knowledge and access to the FiberWAN, he had no help. During the past few years, the DTIS staff has been significantly reduced due to budget cuts, keeping the city dependent on a sole admin for its core network.