Debunking the Threat to Water Utilities

?Consider this: An operator at a water treatment facility presses a button to add a certain measurement of chemicals to untreated water. Instead of doing so, the computer dumps twice the amount of chemicals, an amount way above the maximum safety zone. The resulting excess causes poisonous toxins in the water and when distributed to individual homes, entire communities fall ill. Investigators and the public are left asking, ?How did this occur?? The answer: a computer bug known as a Trojan horse.??From Hardening America?s Public Utilities Against the Threat of Cyberterrorism, by Jason B. Lee and Steven E. Roberts.

Jason Lee and Steven Roberts, risk mitigation and security experts, postulate that the simple Trojan horse hack can result in cyberterrorism. Is this credible or simple fear mongering? CIO went to the Massachusetts Water Resource Authority (MWRA)?s operations center in Chelsea, Mass., to find out.

What we found was a system that would be difficult to hack into and any number of best practices for securing systems against cyberterrorism. Here?s the skinny:First, a hacker would need access to the MWRA computers. We are in a locked room accessible by key card and manned 24/7. To get in, you must check in at the facility?s front desk (and then check out later), offer your credentials, wear a temporary badge and be with an escort at all times. After you leave, your host will send a memo to senior management detailing the visit for the record.

The computers we?re looking at distribute water throughout much of eastern Massachusetts. An hour or so west?near the Wachusett Reservoir?is an identical crescent of computers that monitor water quality and control the chemicals that enter the water, according to Marcus Kempe, director of operations support at the MWRA.

Together, these two banks form the MWRA?s Scada system. Scada (pronounced ?scay-da?) stands for supervisory control and data acquisition; most public utilities rely on a highly customized Scada system. No two are the same, so hacking them requires specific knowledge?in this case, knowledge of the MWRA?s design and access to that customized software.

Scada is not networked, except in two places. One, a dial-up modem, is offline. Only one person has clearance to use it. Turning it on must be done manually by someone with clearance at the facility. And two, there is a link to the MWRA?s general IT infrastructure through a program called Plant Information (PI). PI gives a small set of supervisors with the highest clearance a one-way view of data about the water system. They can look, but they can?t touch. This data can also be piped into a war room around the corner from us in the operations center, which is used for incident response.

If a hacker somehow got into Scada, he would need user names and passwords to gain control of the command and control computers; he would need a way to either make changes undetected?though someone is watching the system around the clock?or hide the fact that he is making changes. And he would need to work fast: Systems lock after a few minutes of inactivity and can?t be reactivated without a password.

Scada connects through a private line (soon, via microwave) to Programmable Logic Controllers, or PLCs, at the water facilities, which churn 250 million gallons of water per day from the reservoir to faucets. PLCs are dumb, rugged chips that basically never fail. They follow the lowest level, most basic instructions (such as turn on and turn off), and report them to Scada (?I just turned on.?). If something is wrong, the PLC says, ?Help me? in the form of an alarm. The alarm sounds at the water site and at the Scada operations centers. The alarm also flashes on the computers, and it can?t be shut off until a formal acknowledgement of the alarm is made and physically logged by a human being.Every month, about 1,700 samples of the water are tested for unusual characteristics. ?Rolling crews? periodically go to MWRA pump stations and storage sites, and check the integrity of the facilities, and the electronics at the facilities such as the chlorine monitoring devices. Most of the water facilities are under surveillance and, currently, under the watch of the National Guard.

But suppose a hacker got by all this and, through the use of a computer either at the operations facility or remotely, planted a Trojan horse that at some point ordered the system to dump too many chemicals in the water.

That water, chlorinated, leaves the reservoir and enters the pipes, where it will receive PhpH adjustment and fluoridation.

Scada receives data about the water 10 minutes after it enters the pipes. It?s checking for wild fluctuations in chlorine levels, which would indicate a reaction with some bacteria or foreign agent. There are several more chlorine checkpoints, at two hours downstream, three hours, and so forth. If the Wachusett Reservoir were in one endzone of a football field and your faucet were in the other, your water would be checked at its own one yard line, its 20, its 40, your 40, and then it would be stored at another facility at your 20 yard line and tested there too. It also receives a goal-line chlorine treatment as an extra safety measure. It would take your water anywhere from 12 hours to three days to go endzone to endzone.

If, after all of this, toxic water made it to faucets because of a computer hack, and people got ill, the MWRA would convene in its war room, and proceed with a detailed emergency incident response plan that includes shutting down pumping facilities, and sending out emergency broadcasts, among other steps.