By Ben Worthen

Ed note: Ron Suskind’s new book, The One Percent Doctrine, is named after the Bush administration’s guiding principle that if a terror threat has a one percent chance of being real, it needs to treat it like an absolute certainty. Similar thinking has influenced the government’s use of data mining as the prevailing technology strategy for fighting terror. Instead of requiring data mining program leaders to develop a business case demonstrating a project’s value, agency heads approve these programs with the rationalization that if it has a chance—however remote—to catch a terrorist, then it is worth it—despite issues such as cost, project delays due to unlimited project scope and a clearly demonstrated public sensitivity to encroachments on civil liberties. This in an excerpt from an article, which will be published in CIO magazine on Aug. 1, by Senior Writer Ben Worthen detailing the issues at stake in preventing terror with IT.

In the aftermath of Sept. 11, the government concluded that data mining could help it prevent future terror attacks.

Data mining is a relatively new field within computer science. In the broadest sense, it combines statistical models, powerful processors and artificial intelligence to find and retrieve valuable information that might otherwise remain buried inside vast volumes of data. Retailers use it to predict consumer buying patterns, and credit card companies use it to detect fraud.

Experts say that the government, and in particular the intelligence community, has come to rely heavily on data mining. A 2004 Government Accountability Office report found that federal agencies were actively engaged in or planning 14 data mining projects that focused explicitly on catching terrorists and preventing attacks, a total that does not include projects at seven agencies, including the CIA and the National Security Agency. Over the past year, The New York Times, USA Today and other media outlets have uncovered top-secret programs within those agencies that collect and look for patterns in phone records, e-mail headers and other personal information. When these programs were made public, the president and other members of his administration defended them as critical to the war on terrorism.

Given the administration’s commitment to programs using these data mining tools and the pressure on everyone to prevent another attack, it comes as no surprise that these projects are being approved by agency heads almost as fast as they are being conceived, experts say. “There is a real fear of not going down this path because if there is value, you don’t want to be on the side that opposed [a data mining project],” says Robert Popp, who was deputy director of the Information Awareness Office at the Defense Advanced Research Projects Agency.