Last year, we thrilled you with our Terrifying
Tales of IT Horror, which showcased
the blood-curdling things that can happen to companies when
their IT systems go wrong. This year, we decided to get a bit
more personal and ask tech veterans near and far for tales of
personal woe and destruction at the hands of IT run amok.
Read on, and feel glad that nothing like these has ever
happened to you…or has it? If you can relate, feel free
to share your own tale.
First: Nightmare at Napster
In 2000, Michael T. Halligan, (now CTO at BitPusher) was an overworked, underpaid grunt at Napster. “How
cool!” you think. But remember, Napster’s day had
waned by the new millennium, and Halligan’s horror story
had just begun.
Ghostly Gear: Technology Tools for Paranormal Investigations
25 Terrifying Information Technology Horror Stories
Even More Tales: Personal Stories of Tech Disaster
Talking Tombstones Bring Words from Beyond the Grave
Real Monsters That Frighten the Daylights out of CIOs
We had no money, and everybody in the music industry was
suing us. Suffice it to say, we had very few technical
resources. The day I started there was the last day of a
sysadmin who had literally gone crazy working at Napster (but
that’s another story). That brought us down to a very senior
admin and me. A week went by, and the other sysadmin called in
sick for a few days—then he stopped answering phone
calls, leaving me on my own with no documentation whatsoever
for approximately 350 servers.
After about six months of this, the stress started taking
its toll on my health, and I caught a flu that took a month to
kick. Unable to stay home to recover, I brought a beanbag and
some blankets so I could sleep under my desk. One morning
during this, I awoke at about 4:00 to a ringing phone and a
beeping pager. The voice on the other end of the phone was
someone from CNET asking if Napster had shuttered its doors. I
groggily said, “No comment,” and started to go back to sleep
when the question struck a chord. Shuttered our doors? I got
online, and sure enough, we were completely down.
Insanely sick as I was, I called our service network service
provider—Above.net—and found out that their network
was fine. But they had noticed a 300Mb/s
drop on our circuit. I crawled into my Jetta and sped the
30-mile trip to Above.net to investigate the problem.
What I found out was that our supervisor cards in our 6509
were apparently not functioning. This was a bit disconcerting
since we didn’t have access to them. Our former system
administrator hadn’t left it to us. I called him a dozen times
to no avail. (I admit that I did it mostly because it was 4
a.m. and I wanted to annoy him for causing me this misery.)
After an hour of creativity, I found a backup of a config
file from an office router, ran it through a Cisco password
crack utility and discovered it had the same password as our
production routers (joy!). During this time, I called a friend
who owns a Cisco consulting shop, I started running through
commands and found out that the first Supervisor module had
failed, but the second one was fine. Unfortunately, the second
one wasn’t configured.
My friend had Cisco dispatch a technician in record time,
and he got us back up in about three hours.
The world was abuzz with news that Napster had been down
most of the day. For eight hours, teenagers couldn’t steal
their Green Day, housewives couldn’t download Rod Stewart and
programmers couldn’t trade the latest Amon Tobin MP3s. Two
hours after we went back up, we hit a record of a million
concurrent users, pushing 1.2GB/s of text searches—and
promptly went back down about a half hour after I had fallen
asleep in my car.
Next: The Business Process That Wouldn’t Die>>
The Business Process That Wouldn’t Die
Back when he was still working in IT, Ray Wang
suffered one of those classic “How did they let that
happen?” moments. It’s no wonder he went into
analysis (as a principal analyst with Forrester Research, of
course…what did you think we meant?)
I was working at a company that was set to roll out a $15
million ERP implementation as part of a multichannel order
management project. Our CIO was fired two weeks before the
project was supposed go live. The new CIO had us come in to
review architecture. His current-state processes were mapped to
future-state systems. Without going into details, it looked
like the team had done a detailed mapping. We’d cut off
legacy systems at the same time go-live occurred. Teams were
nonstop testing for weeks, making sure that any current
functionality wasn’t lost.
On the day we went live, we flicked on the switch and
everything was humming along. We breathed a sigh of relief as
orders kept coming in with no issues. Fulfillment looked
But sometime around midday, we discovered that the staff
couldn’t keep up with the orders, and we couldn’t
figure out why. In the background we heard that the printers
were running out of paper, which seemed like a normal issue.
Otherwise, all the systems were chugging along. The website was
getting good hits and orders seemed to be coming in OK. The
goal of the rollout was to move most of the orders to customer
self-service, and so far it seemed that the self-service Web
orders were coming into the system with no problems.
However, we found out later that the system was
automatically taking the data and pumping out a fax, which then
printed out via the company’s printers for someone on the
team to reenter the data into the system! The process
consultants replicated existing business processes without
optimizing them and so the company ended up having to hire 20
more order entry clerks to keep up with the 400 percent
increase in volume of faxes!
Next: Shaken, Not Stirred…
Shaken, Not Stirred…
In 1989, consultant Elizabeth Zwicky was working
her way through another fall day. Then the earth started to
My scariest moment came just after the Loma Prieta earthquake.
I was working on the third story of a three-story building,
which had never seemed particularly tall to me until the very
second that I could tell that the earth was standing still and
the building wasn’t.
We evacuated the building, and it became clear that the
structure was still standing—as was civilization. But
thanks to bridge and road damage, few people could actually get
home. As the first shock wore off, we became a crowd of bored
people standing around in a parking lot watching the asphalt
ripple, and the novelty wears off that pretty quickly.
The company I worked for had its own generator, and the big
ol’ Sun systems we ran had so much capacitance that it didn’t
even crash when we flipped to generator power, so we had all
sorts of computers and lights and whatnot still running when we
left the building. Eventually somebody from management came
around and said, “You know how in 1908 San Francisco burned
down after the earthquake? We really need to do our part to
avoid recreating that, so who’d like to volunteer to go turn
stuff off in case floating clouds of gas come by?” Everybody
volunteered, to which they said, “I gotta tell you, the
structural engineers haven’t been by, so technically, the
building might fall down at any moment.” We contemplated it. It
didn’t look like it was going to fall down. And by golly, it
was our job! To save Menlo Park from fire! And it was not
boring! So we kept on volunteering, and they carefully wrote
down all our names, counted us off, regretted that they had no
hard hats and advised us to leap for the middle of the building
if the outside started to peel off. Then we moved inside to
turn off everything electrical we could find.
I didn’t get the most interesting wing, it turned out. That
would be the one where they found the researcher still working.
The people who found him had to explain that we were going to
turn off the servers so there was really no point sticking it
out. But I did get the wing with the most damage, forcing me at
one point to climb over a downed filing cabinet to get to a
computer located in one of the offices. It was an outside
office, in the bit they had said might peel off. Worrisomely,
it was also a low, stable, two-drawer file cabinet, not some
tipsy high thing. But I climbed over it and started looking for
power switches on the computer.
None on the front. Reached for the back. Couldn’t reach. Lay
on my stomach on the desk with my head under the bookshelves,
groping for the power switch and…felt things shift underneath
me. Immediate thought: “I am about to have my head crushed
because I am an idiot. And if not, I am going to fall to my
death when the building crumbles.” Then I thought up a number
of obscenities inappropriate for publication, directed in
approximately equal proportions at myself and at the loose leg
on the desk on which I was lying. The leg, as it turned out,
was the only thing moving.
I got out from under the bookshelf, ripped all the power
cords I could find out of the wall and moved on, though much
faster—if more carefully.
The next day, we moved back into the building.
The week after that, we reevacuated the wing I’d been in
because the structural engineers declared it unsafe.
Next: The Day the E-Mail Stood Still, and the Man Who Caught
The Day the E-Mail Stood Still, and the Man Who Caught
Back in the mid-90s, Brad Knowles was senior
Internet mail adminstrator at America Online, at the time the
largest online service provider in the world. But with great
power comes great responsibility…
It’s known as Black Wednesday, August 10, 1996, the
day all of AOL’s routers went down, and no one could get any
packets to our systems—they all just got thrown away. But
computers could still contact our backup name servers at ANS (a
subsidiary of AOL that ran all of our external WAN
connections), so they knew who all of our mail servers were and
how many IP addresses we had listed.
Now, it’s important to know that the Internet
RFCs requires that that you wait at least two minutes
when you start to set up a standard TCP/IP connection before
you finally declare the other end to be dead. The standard
practice is also to attempt to connect to each of the IP
addresses you know for a given name, usually in the sequence in
which you received them. At the time, the standard practice for
mail servers was that you contacted all listed mail servers for
a given domain before you gave up.
Now, step back and do the math for seven names with seven IP
addresses each, and two minutes per IP address:
7 x 7 x 2 = 49 x 2 = 98
So, just making one delivery attempt to a single user at AOL
was taking 98 minutes to time out. Then another 98 minutes to
time out for the next user or the next message for a user at
At the time, most sites were running Sendmail. They were set
to rerun their queue once an hour, and many sites would
typically have just the one queue runner process. Each time
you’d start up a queue runner, if you had even a single message
queued up to a single person at AOL, that process would sit
there and spin its wheels for at least 98 minutes trying to
talk to the AOL mail servers before giving up—and it
would block and not do anything else while it was spinning its
wheels. But less than 60 minutes after that happened, another
queue runner would get fired up—and would almost
certainly hang on the same message going to AOL, or on another
message going to AOL.
Do that often enough, and you get enough queue runners hung
up to AOL that your queue is clogged and you’re not getting
mail through to anywhere else in the world. Do
that long enough, and you’ve got so many
queue runners hung up to AOL that you run out of RAM and swap
space and your mail servers crash.
Well, that’s what happened, and I was personally
blamed for taking out all Internet e-mail across the entire
world. As a result, angry spammers publicly handed out my
private telephone numbers and people were asked to complain
directly to me. I was also told about at least one business
that went bankrupt because it was waiting on a time-critical
RFP to come in and it didn’t get its bid into the system in
time, so it lost the contract.
Once we finally did come back up, it literally took days for
us to recover and to catch up to all the backlog that was
created for us on the Internet—and it took the rest of
the world a few more days beyond that to recover from the rest
of their backlog.
Next: Dear Mom…>>
You’ll have to forgive us for leaving this
one anonymous—but read it all and you’ll understand
why. Let’s just say that the fellow who submitted it was
kind enough to supply evidence that his tale is terrifyingly
I have a website. It has a mixture of stuff on it, mostly
out of date, because being a company director I don’t often
have time to keep it up to date. One day I had just finished a
new webpage with my most recent photos and had decided to send
the link to interested friends so they could have a look at
them (and tell me if any of them were particularly
So there I am, going through my address book selecting those
people who are good enough friends to be interested, when the
phone rings. It’s my mother and she’s lost the installation
disk for her printer. So being the dutiful son, I go to Epson’s
website, download the drivers for her and attach them to an
e-mail to send to her. The phone rings again and it’s a
customer with a load of questions about his service that if he
had read the instructions he would have known all the answers
anyway—except he’s lost the instructions—again. So
I click on “Create Mail” and start a third mail going. I then
remember I haven’t yet sent the e-mail to my mother so I go
back to the other one, add her address to the “To” section and
Problem: I have sent her the wrong
e-mail. I’ve sent her the one about the website. Blood running
cold time. Mad panic. The reason: I am a
transvestite and the new page was of me
dressed up as the female me—and she
didn’t have a clue. OMG! What can I do? I start trying to think
of excuses—you can imagine the blind panic. Anyway, five
minutes of absolute mayhem ensues, she is almost 70 and would
NOT understand, I am close to tears, then suddenly I hear a
noise. Incoming mail. I am expecting it to be from her.
Instead, it was: “This is the automated sender verification
system at XYZ.com. Your e-mail will held in a queue for 7 days
until you verify you are a real person by
clicking on the following link. After that we will assume your
e-mail was spam and it will be deleted without being
Normally I slag off Challenge Response systems (from the
antispam point of view)—but in this
Next: School Days, School Daze…
School Days, School Daze…
It sounds like a tale from the Dark Ages, but it
happened this year! ZaReason
CEO Cathy Malmrose was helping upgrade computers at a number of
schools. The experience taught her more than she’d ever
At a school district the size of a college campus, I worked
my way up the ladder until I finally got to talk to the head
honcho in charge of IT for the school. He explained that they
just updated “all the school’s computers.” I asked him what
they had updated since I know that most classrooms currently
have old, nonfunctional systems. His reply: “We got that new
thing, you know with the wires?” His hands were making big
gestures, so I ask, “You mean a network?” His reply: “Yeah,
that.” This guy was in charge of all IT decisions. I screamed
But it didn’t stop there. When I was working with a
small, poor inner-city school, the head honcho told me they had
three new computers that are “great, but they don’t work.” I
know she won’t know the specs off the top of her head so I ask
if I can see them. She leads me to a room with three Dell
monitors sitting on the floor. “Here they are.” I had to
desperately find a way to explain that they wouldn’t
“work” without a little more hardware.
Next: “What’s His Name”
“What’s His Name”
Earlier in his career, Cognos CIO Claudio Silvestri experienced a classic horror story moment: stage fright during a major presentation in front of his new boss. It doesn’t get much scarier than that.
While working as a junior executive at a large multi-national IT hardware manufacturer, we were working on an Oracle Financials implementation and just wrapping up the testing phase when the company replaced its CIO.
The prior CIO was very much in favor of a distributed IT model; however, as you might guess, the new CIO was very much in favor of a centralized model. The project was stopped and new direction set to implement SAP in a highly centralized manner.
In order to respond quickly, I hired a very large and reputable consulting and system integration company. We spent the next two months recalibrating and putting together our plans. After a very intense period, we were set to present the new direction to our executive management and the new CIO. It was going to be quite a show, with people from around the world set to hear from me on the plan to move forward.
With the worldwide teams gathered in a large auditorium and me up front behind a podium, I began my presentation. I started by introducing everyone involved and thanking them for the tremendous effort over the prior weeks to prepare what we believed was a winning plan to adjust to the new approach. The senior Partner from our SI was seated directly to my left, and so I started with him.
As I turned to introduce him, my horror story began: I could not for the life of me remember his name. After many weeks of working very closely with this person, through some intense and energetic dialog and more than a few beers at a local establishment, I simply drew a complete blank and could not remember his name. I’m sure I must have looked stunned on stage to all in the audience and to our new CIO, who was sitting front and center.
The Partner, being much more polished than I, realized my predicament. He simply came up to the podium and introduced himself as “John Doe”.
This got a laugh, broke the tension, and turned the situation completely around. The plan was well received after a scary start.
Next: The IT Man Cometh…Thankfully
The IT Man Cometh…Thankfully
Typing in anger leads to careless
mistakes—especially if you hit “Enter” too
soon. Just ask one of our own, Senior Editor Kim Nash.
We’re happy to say her friend survived the scare, but he
needed some serious help….
I remember when a friend of mine in California was spewing
vitriol about his boss to me, in Massachusetts, via e-mail. Yet
he mistakenly addressed the message to said boss—which he
realized only upon hitting the “Enter” key. It was
after-hours on the East Coast, and my friend, now panicky and
more than a bit sweaty, paged his IT manager at home for
Can you say, “Desperate to keep my job? Please,
please, please?” In those days, the IT manager
couldn’t do a thing from his house, so he bundled up, got
in his car and motored into the office to obliterate the
offending e-mail from the system before his boss opened it the
That act of human kindness got the IT manager a bottle of
fine, fine California wine FedExed to him the very next
Don’t you wish all these stories could have such happy