Avoid a Tech Catastrophe and Step Away From the Button

It's only a matter of time before all hell breaks loose when fiddling with switches and power cords

Gunnar Pippel

Admit it: We can't keep our hands off tech and gizmos, even when we know better. We give in to the irresistible urge to see what happens if a button is pushed or innocently believe it couldn't hurt anything. Then chaos breaks loose and IT spends hours repairing the damage. Oops.

The InfoWorld Off the Record blog is where IT pros share these real-life tales of the job. In this collection, rookies and veterans alike are caught up when the harmonious hum of work comes to a sudden halt.

Share your experience about managing IT, developing apps, supporting users, or a humbling moment. Send your story to offtherecord@infoworld.com. If we publish it we'll keep you anonymous -- and send you a $50 American Express gift cheque.

Robert Acocella

The circuit breaker vs. the network admin

At budget time, a senior network admin showed off the data center to the managers, especially two newish servers. The admin bragged about the uptime and waved away questions about a UPS, saying there's no need. To prove his point, he touched the server's dedicated circuit breaker -- and 500 users instantly lost their server connection. The second server proved just as cooperative, severing the connection for the rest of the employees with merely a touch of the circuit breaker.

Somehow, the admin does kept his job, if not his pride, and management allotted more money for server infrastructure.

Katja Govorushchenko

The wrong button vs. the newbie

The good news: You caught a mistake before damage was done. The bad news: You're not out of the danger zone.

A junior techie went to the server room to turn off a noncritical machine. The techie found it and pushed the button -- then realized he had the wrong server, one housing the files in use by 600-plus employees. Letting go of the button would mean chaos.

With no cell service and nobody within earshot, his only hope was a phone standing 10 feet away. A desperate search yielded an unusual solution: The techie removed his pants, threw them around the phone, and finally pulled the phone in. Help eventually arrived -- along with several other eyewitnesses who wouldn't forget the sight soon.

Andrew Lewis

The power off button vs. the tinkerer

The data center operator only wanted to expedite tasks, ease workflows, and clean up nagging details. But one day the zeal for improvement went a little too far.

The efficiency bug bit back when the operator noticed a loose ring on a piece of glass -- that happened to cover an emergency power-off button linked to the data center. A tiny turn of the ring set off a chain of events, including a systems shutdown and a blackout. No long-term damage was done, but the operator learned to investigate thoroughly before making "improvements" -- and when to keep his hands to himself.

Ron Chapple Stock

The fire suppression system vs. the boss

Sometimes, an assuming desktop item can unleash havoc at the office.

This  tale involves the classic computer room: three mainframes bearing multiple attached tape drives, a row of disc packs the size of dishwashers, three line printers, one high-speed laser printer -- and a massive Halon fire suppression system. The boss and the operators can't agree on how to handle a shutdown in case of fire, with the boss insisting the operators stay behind to handle the mainframes. To settle the matter, they gather for a test run.

Before they can start, a stray paperclip drops into the control box, creates a short circuit, and triggers the Halon. Guess who's the first one out the door? And the operators prove their point handily.


Servers vs. execs

Here's a simple rule for admins: Do not let the execs touch the servers. Repeat: Do not let the execs touch the servers.

Otherwise, you may suffer the same fate as two tech departments in the midst of an arduous data center consolidation after a merger, complete with change notices, VP requests, and so on. You wouldn't have known it, though, on the day workers couldn't get email and some remote sites couldn't connect. The staff eventually traced the problem to server failures and wondered why it all happened at once.

Coincidence? No. The VPs had visited the acquired company earlier that day and turned off "unused" servers they determined had no impact on productions systems.

Repeat: Do not let the execs touch the servers.


Servers vs. techs

Stupid users, clueless execs, bumbling bosses -- you expect trouble from them. But what happens when a colleague within the IT ranks is responsible for that fateful flick of the wrist? In this case, a near disaster.

The tech pro should've known better when tasked with determining which servers were to be decommissioned in a large, bustling data center. Be it bad judgement, brain fart, or simple ignorance, he honed in on a critical management server -- then unplugged it, took it to his desk, and reformatted the hard drive.

This major mistake unleashed a tidal wave of problems related to the firewalls, backups, and that database. Unsurprisingly, the techie was transferred before he could strike again.


Have an IT story to share?

Write up your own crazy-but-true tale about managing IT, developing apps, supporting users, a humbling moment, or a time when something went very right.

Send your submission to offtherecord@infoworld.com. If we publish your story -- anonymously, of course -- you'll receive a $50 American Express gift cheque.

Copyright © 2014 IDG Communications, Inc.

Related Slideshows