Inside a Network Operations Center

CSO — I recently had a chance to visit Harvard University's network surveillance center. One doesn't normally see the words university and network surveillance in the same sentence, because surveillance of any kind is usually seen as being at odds with the tradition of academic freedom present at most universities. Unfortunately, higher education has long been associated with Internet-related computer crime—both as victims and as the home institution of many perpetrators. As a result, many universities have had to make significant investment in various kinds of network monitoring.

What makes Harvard's network surveillance notable is not the fact that Crimson engages in network surveillance but the scale and technical sophistication of those monitoring operations. Harvard has 6-gigabit connections to both Tier 1 Internet providers and Internet2. Between 10 and 20 terabytes of data moves across Harvard's border every day. What's more, traffic frequently undergoes asymmetric routing, which means that packets travel across different border routers depending on whether they are leaving Harvard or returning—one of the unfortunate consequences of something known as "hot potato routing."

Yet despite this complexity, Harvard manages to categorize and record information about practically every packet crossing its borders.

To find out how Harvard works this magic, I met with Jay Tumas, Harvard’s network operations manager. It wasn’t a long walk: Jay's office at University Information Systems is just a block down the street from my office at the School of Engineering and Applied Science.

No Packet Left Behind
Harvard's connections to the Internet and Internet2 take place in three physical locations: two in Boston and one in Cambridge. But rather than deploy intrusion and anomaly-detection systems at the border, Tumas has built a dedicated monitoring system that takes all critical traffic, makes a copy of every packet and sends those copies to the network surveillance center on 10-gigabit optical fibers. There the flows are reassembled using Cisco switches and sorted according to protocol family using a cluster of Top Layer 4508 IDS Balancers.

This architecture both lets Harvard split the load among multiple systems—it’s too much data for one IDS—and lets each IDS be configured with only the signatures that it actually needs, which makes each IDS run faster than it would if it were responsible for the full protocol suite.

"Last year we had over 10 million IDS hits," says Tumas. But instead of sending out an alert for each hit or just tabulating them in some log file that nobody ever really reads, Harvard has built a reactive system that rates the severity of each IDS hit, judges the chance of a false positive and then automatically alerts the responsible security manager.

Continue Reading