On the evening of Sept. 27, 2001, Howard Rubin, a computer science professor at City University of New York who had advised the Clinton administration on technology issues, was home observing Yom Kippur, the holiest day on the Hebrew calendar.

Observant Jews don’t work, drive or use appliances on Yom Kippur, but Rubin had a strong feeling he should pick up the phone when it rang that night.“My wife didn’t want me to answer it,” he recalls. But he did.

On the other end of the line was one of the most senior members of the previous administration. He wanted to know if Rubin knew of any technologies the government could use to help catch terrorists.

Rubin’s answer has since become a technology mantra among members of the intelligence community: data mining, he told the official.

Data mining is a relatively new field within computer science. In the broadest sense, it combines statistical models, power¿ful processors, and artificial intelligence to find and retrieve valuable information that might otherwise remain buried inside vast volumes of data. Retailers use it to predict consumer buying patterns, and credit card companies use it to detect fraud. In the aftermath of September 11, the government concluded that data mining could help it prevent future terrorist attacks.

A Proliferation of Projects

Experts say that the government, and in particular the intelligence community, has come to rely heavily on data mining. A 2004 Government Accountability Office report found that federal agencies were actively engaged in or planning 199 data mining projects. Of these, 14 focused explicitly on catching terrorists and preventing attacks, a total that does not include projects at seven agencies (such as the CIA and the National Security Agency) that did not respond to the GAO survey. Over the past year, The New York Times, USA Today and other media outlets have uncovered top-secret programs within those agencies that collect and look for patterns in phone records, e-mail headers and other personal information (see "What to Do When the Government Wants Your Data"). When these programs were made public, the president and other members of his administration defended them as critical to the war on terrorism.

Given the administration’s commitment to programs using these data mining tools and the pressure on everyone to prevent another attack, it comes as no surprise that these projects are being approved by agency heads almost as fast as they are being conceived, experts say. "There is a real fear of not going down this path, because if there is value you don’t want to be on the side that opposed [a data mining project]," says Robert Popp, who was deputy director of the Information Awareness Office at the Defense Advanced Research Projects Agency. Of course, government officials also have a straightforward reason for pursuing data mining projects, says Robert Gourley, CTO of the Defense Intelligence Agency: "We want to protect our country and our way of life."