The Harvard Network Operations Center has a database with between 1,500 and 2,000 registered system and network managers. When the IDS detects a "hit," the system tries to correlate the hit with other hits. If enough tests pass, the system auto alerts and sends a missive to the responsible manager. Last year roughly 10,000 such messages went out. "We want people to treat the auto alerts as gospel," says Network Security Manager David LaPorte, who works for Tumas.

Real-time alerts are an important part of network surveillance, but without the ability to look back in time, alerts are of limited use. It's important to find systems that have been compromised. But once you've found these systems, it's equally important to evaluate the damage that's been done. For example, says Tumas, Harvard's IDS system recently discovered a Microsoft Active Directory domain controller that had been hacked. Not surprising, none of the system's logs had been turned on.

To find out what had happened to the system, Tumas and his team turned to QRadar, a security monitoring system sold by Q1 Labs. QRadar monitors multiple sources of information, including packet traces, network flows and security events; builds a model of the network; uses the real-time information to update the model; and archives information as necessary to permit event reconstruction at some future time.

Just as every packet in and out of Harvard gets evaluated by the IDS systems, every packet also gets processed by QRadar. The system analyzes the packets, reconstructs the UDP and TCP streams, decodes the protocols, determines whether protocols are running on the correct port and updates a database of what it's learned in real-time. The system can also be programmed to record part or all of every packet that it sees, although doing so obviously requires a significant amount of storage for a network the size of Harvard's.

"We data-mined every single connection that this system created across the border, then went through and picked out the things that were not typical command-and-control bot traffic—anything that we couldn't identify," Tumas says.

It turned out that the compromised system had participated in a 350-megabyte file transfer with a computer system at another university. This was a matter of great concern. So Harvard contacted the other university and had it look at the other compromised system. The administrators at the other school found the files—350 megabytes of French music. "They weren't in [the system] long enough to discover the value of what they had," Tumas surmises.