Even More Tales of Technology Terror: Personal Stories of Tech Disaster

CIO — Last year, we thrilled you with our Terrifying Tales of IT Horror, which showcased the blood-curdling things that can happen to companies when their IT systems go wrong. This year, we decided to get a bit more personal and ask tech veterans near and far for tales of personal woe and destruction at the hands of IT run amok.

Read on, and feel glad that nothing like these has ever happened to you…or has it? If you can relate, feel free to share your own tale.

First: Nightmare at Napster

In 2000, Michael T. Halligan, (now CTO at BitPusher) was an overworked, underpaid grunt at Napster. “How cool!” you think. But remember, Napster’s day had waned by the new millennium, and Halligan’s horror story had just begun.

We had no money, and everybody in the music industry was suing us. Suffice it to say, we had very few technical resources. The day I started there was the last day of a sysadmin who had literally gone crazy working at Napster (but that's another story). That brought us down to a very senior admin and me. A week went by, and the other sysadmin called in sick for a few days—then he stopped answering phone calls, leaving me on my own with no documentation whatsoever for approximately 350 servers.

After about six months of this, the stress started taking its toll on my health, and I caught a flu that took a month to kick. Unable to stay home to recover, I brought a beanbag and some blankets so I could sleep under my desk. One morning during this, I awoke at about 4:00 to a ringing phone and a beeping pager. The voice on the other end of the phone was someone from CNET asking if Napster had shuttered its doors. I groggily said, "No comment," and started to go back to sleep when the question struck a chord. Shuttered our doors? I got online, and sure enough, we were completely down.

Insanely sick as I was, I called our service network service provider—Above.net—and found out that their network was fine. But they had noticed a 300Mb/s drop on our circuit. I crawled into my Jetta and sped the 30-mile trip to Above.net to investigate the problem.

What I found out was that our supervisor cards in our 6509 were apparently not functioning. This was a bit disconcerting since we didn't have access to them. Our former system administrator hadn't left it to us. I called him a dozen times to no avail. (I admit that I did it mostly because it was 4 a.m. and I wanted to annoy him for causing me this misery.)

After an hour of creativity, I found a backup of a config file from an office router, ran it through a Cisco password crack utility and discovered it had the same password as our production routers (joy!). During this time, I called a friend who owns a Cisco consulting shop, I started running through commands and found out that the first Supervisor module had failed, but the second one was fine. Unfortunately, the second one wasn't configured.

My friend had Cisco dispatch a technician in record time, and he got us back up in about three hours.

The world was abuzz with news that Napster had been down most of the day. For eight hours, teenagers couldn't steal their Green Day, housewives couldn't download Rod Stewart and programmers couldn't trade the latest Amon Tobin MP3s. Two hours after we went back up, we hit a record of a million concurrent users, pushing 1.2GB/s of text searches—and promptly went back down about a half hour after I had fallen asleep in my car.

Next: The Business Process That Wouldn’t Die>>

or...

Continue Reading