IT Strategy: IT Versus Terror

Preventing a terror attack is invaluable. But even invaluable IT projects need realistic business case analysis to succeed.

On the evening of Sept. 27, 2001, Howard Rubin, a computer science professor at City University of New York who had advised the Clinton administration on technology issues, was home observing Yom Kippur, the holiest day on the Hebrew calendar.

Observant Jews don’t work, drive or use appliances on Yom Kippur, but Rubin had a strong feeling he should pick up the phone when it rang that night.“My wife didn’t want me to answer it,” he recalls. But he did.

On the other end of the line was one of the most senior members of the previous administration. He wanted to know if Rubin knew of any technologies the government could use to help catch terrorists.

Rubin’s answer has since become a technology mantra among members of the intelligence community: data mining, he told the official.

Data mining is a relatively new field within computer science. In the broadest sense, it combines statistical models, power¿ful processors, and artificial intelligence to find and retrieve valuable information that might otherwise remain buried inside vast volumes of data. Retailers use it to predict consumer buying patterns, and credit card companies use it to detect fraud. In the aftermath of September 11, the government concluded that data mining could help it prevent future terrorist attacks.

A Proliferation of Projects

Experts say that the government, and in particular the intelligence community, has come to rely heavily on data mining. A 2004 Government Accountability Office report found that federal agencies were actively engaged in or planning 199 data mining projects. Of these, 14 focused explicitly on catching terrorists and preventing attacks, a total that does not include projects at seven agencies (such as the CIA and the National Security Agency) that did not respond to the GAO survey. Over the past year, The New York Times, USA Today and other media outlets have uncovered top-secret programs within those agencies that collect and look for patterns in phone records, e-mail headers and other personal information (see "What to Do When the Government Wants Your Data"). When these programs were made public, the president and other members of his administration defended them as critical to the war on terrorism.

Given the administration’s commitment to programs using these data mining tools and the pressure on everyone to prevent another attack, it comes as no surprise that these projects are being approved by agency heads almost as fast as they are being conceived, experts say. "There is a real fear of not going down this path, because if there is value you don’t want to be on the side that opposed [a data mining project]," says Robert Popp, who was deputy director of the Information Awareness Office at the Defense Advanced Research Projects Agency. Of course, government officials also have a straightforward reason for pursuing data mining projects, says Robert Gourley, CTO of the Defense Intelligence Agency: "We want to protect our country and our way of life."

No Scope, No Budget, No End

But some experts are beginning to question whether an IT strategy of unlimited scope, budget and schedule will best serve that end. It’s a conundrum CIOs face every day. IT projects, no matter how vital, tend to fail when controls don’t exist or those controls fall away in the face of a time crunch or crisis. Lack of oversight is the chief cause of project failures, according to the Standish Group, an analyst firm that tracks IT success rates. It leads to overly ambitious projects, an unwillingness to change the original vision and inattention to signs that something isn’t working. "It doesn’t matter if it is a supply chain project, an ERP system or data mining—those things need to be considered," says Jim Johnson, the Standish Group’s chairman.

"No one [in the government] has looked at data mining from an IT value perspective," says Steve Cooper, former CIO of the Department of Homeland Security. "I couldn’t figure out [the value of data mining] when I was in DHS, and I can’t figure it out now. But that didn’t stop us from using it."

In other words, according to Cooper, no one has done a business case analysis to determine whether the government is getting a return on its investment. Instead, a rationalization is usually sufficient: If a project has a chance to catch just one terrorist, then it is worth it.

Given that the government’s track record on IT project management is particularly poor (see "Federal IT Flunks Out," www.cio.com/051506), a lack of typical IT project analysis, prioritization and management controls could backfire. Badly. Experts worry that projects could drag on for years and that good projects could be thrown out with the bad because of privacy and civil liberties issues. (In fact, Congress has already halted a number of data mining projects, including the Department of Defense’s Total Information Awareness project, an ambitious 2003 attempt to create a massive database containing just about everything and anything that could be used to identify possible terrorists. See "Poindexter Comes in from the Cold," www.cio.com/080104.)

Experts are also concerned that in its zeal to apply technology to antiterrorism, the government could disrupt the crime-fighting processes of the agencies that are charged with finding and stopping terrorists before they act. As any good CIO knows, if users see a system as an obstacle to getting their jobs done effectively, they will rebel or simply ignore it—in this case, with potentially disastrous consequences.

Among data mining experts, there is a growing sense that the government needs to apply the same kind of analysis to its antiterrorism IT strategy that CIOs in the private sector use to keep their projects from spinning out of control. "These projects have perfectly reasonable goals," says Fred Cate, director of the Center for Applied Cybersecurity Research at the University of Indiana. (Cate was counsel for the Technology and Privacy Advisory Committee created in 2003 by Donald Rumsfeld to study his agency’s use of data mining.) "But there’s no oversight procedure," he says.

Data Mining: The State of the Art

The government’s data mining projects fall into two broad categories: subject-based systems that retrieve data that could help an analyst follow a lead, and pattern-based systems that look for suspicious behaviors across a spread of activities. Most data mining experts consider the former a version of traditional police work—chasing down leads—but instead of a police officer examining a list of phone numbers a suspect calls, a computer does it.

One subject-based data mining technique gaining traction among government practitioners and academics is called link analysis. Link analysis uses data to make connections between seemingly unconnected people or events. If you know someone is a terrorist, you can use link analysis software to uncover other people with whom the suspect may be interacting. For example, a suspicious link could be a spike in the number of e-mail exchanges between two parties (one of which is a suspect), checks written by different people to the same third party, or plane tickets bought to the same destination on the same departing date. Many experts believe that the NSA project analyzing millions of domestic phone records is this kind of link analysis system.

Finding the Hidden Linkages

However, link analysis projects are useful only if they have a narrow scope, says Valdis Krebs, an IT consultant who famously developed a map showing the connections among the 9/11 hijackers—after the fact. Successful link analysis requires a reliable starting point—a known terrorist, for example, or a phone number associated with one. Link analysis becomes less effective when it’s used in an attempt to spot anomalous behavior. "If you’re just looking at the ocean, you’ll find a lot of fish that look different," says Krebs. "Are they terrorists or just some species you don’t know about?" If the government searched for only the activities mentioned above—e-mails, checks and plane tickets—without the added insight that one of the network’s members was a terrorist, investigators would be more likely to uncover a high school reunion than a terrorist plot, says Krebs. If the government casts the net too wide, he adds, the projects could cost more, take longer and raise the risk of "false positives," such as the high school reunion example.

One example of the government applying a more realistic scope to a data mining project is a system the DoD is currently testing that sifts through the data the agency has on everyone with a security clearance, looking for patterns that could identify spies. These patterns might include purchases that are out of line with someone’s pay grade, unreported foreign travel or e-mail exchanges with a person known to work for a foreign government, says a counterintelligence official involved with the project who requested anonymity. The parameters for these searches are developed by counterintelligence officers, based on their experience of what suspicious activity looks like. As the technology improves, the DoD hopes to rely on artificial intelligence to decide which patterns warrant attention and which do not.

However, even systems that have more limited scope, such as the DoD’s security clearance system, are sending out mixed signals. "Right now, it’s information overload," says the counterintelligence official. "With the rules we have now, we would have a ton of false positives." His goal is to refine the system and eventually show that the concept works. This, he hopes, will encourage people to share more data.

His project isn’t yet a success, nor has it been deemed a failure. He doesn’t anticipate getting usable results for three or four years. The factors that will determine its future are the same as with any IT project: how well the technology performs, the problems the DoD uses the system to solve and what it does with the results it gets.

Projects Get the Ax

If antiterrorism data mining is going to improve, the business rules aren’t the only aspect that needs to change. After all, a system is nothing without good data. Sometimes law enforcement has a detailed profile of a terrorist suspect. But in other cases all they have is a name. "Names alone are not a helpful way to match people," says Jeff Jonas, data mining’s acknowledged superstar, who made his name protecting Las Vegas casinos from cheats. Jonas, for example, shares his name with at least 30 other Americans. This is one of the reasons why Yusuf Islam (a.k.a. folk singer Cat Stevens) was detained in a Maine airport in 2004.

After 9/11 the government began replacing the Computer Assisted Passenger Pre-Screening (Capps) system—which only tracked passenger data collected from the airlines (names, credit card numbers, addresses)—with Capps II, which would add information culled from data brokers such as ChoicePoint and LexisNexis. Capps II first gained notoriety in 2003, when reports surfaced that Northwest Airlines and JetBlue gave passenger records to the Transportation Security Administration so it could test the new system. Critics asked about privacy safeguards, which were virtually nonexistent according to public records, and in response to the outcry Congress withheld funds for Capps II until the GAO completed a study on how exactly the TSA intended to protect privacy.

In August 2004, the TSA pulled the plug on its $100 million-plus investment in Capps II in favor of a new system called Secure Flight. Secure Flight and its predecessor share many characteristics, most notably combining passenger records with data purchased from commercial databases. (According to a recent government audit, DHS and the Department of Justice spent more than $25 million in 2005 buying data for fighting crime and preventing terrorism.)

In September 2005, the Secure Flight Working Group, a collection of data mining and privacy experts who the TSA asked to review the project, completed a nine-month analysis and filed a confidential report that was highly critical of the system. Within a week, the report was on the Internet. It read, "First and foremost, TSA has not articulated what the specific goals of Secure Flight are." It went on to say, "Based on the limited test results presented to us, we cannot assess whether even the general goal of evaluating passengers for the risk they represent to aviation security is a realistic or feasible one or how TSA proposes to achieve it."

Bruce Schneier, a security expert who was a member of the working group, sees Capps II and Secure Flight as primary examples of how the lack of proper scope has damaged antiterror IT efforts. Even if you managed to design a data mining system that could comb through phone records or credit card transaction and spot terrorists with a 99 percent success rate, it still would not be a good use of investigative resources, argues ¿Schneier. For example, if the approximately 300 million Americans make just 10 phone calls, purchases or other quantifiable events per day, that would produce 1 trillion pieces of data a year for the government to mine. Even 99 percent accuracy would produce a billion false positives a year, or about 27 million a day. And

1 2 Page 1
Page 1 of 2
7 secrets of successful remote IT teams