Why The G-Men Aren’t I.T. Men

In the past few months, FBI CIO Zalmai Azmi has been very careful not to say, “I told you so.” After the FBI was forced to scrap its $170 million virtual case file (VCF), a case management system, because of numerous delays, cost overruns and incompatible software, Azmi was finally given full authority over the agency’s IT budget and encouraged to centralize much of the IT decision making at FBI headquarters in Washington, D.C. He and other top executives at the FBI recognize that they must radically change the agency’s culture if the Bureau is ever going to get the high-tech analysis and surveillance tools it needs to effectively fight terrorism. The FBI, they say, must move from a decentralized amalgam of 56 field offices that are deeply distrustful of technology, outsiders and each other to a seamlessly integrated global intelligence operation capable of sharing information and preventing crimes in real-time.

Former IT managers at the FBI say that sharing information has never been standard operating procedure for the nation’s top law enforcement agency. In fact, FBI agents intent on solving crime are accustomed to holding information close to their bulletproof vests. Many agents scorn the idea of sharing information within the Bureau—and with other federal, state and local law enforcement agencies. Even when agents are promoted to management positions at headquarters, they bring with them the same secretive modus operandi to the day-to-day management philosophy in Washington, D.C.

FBI CIO ZalmaI Azmi is working hard to win buy-in from agents in the field so that the next multimillion-dollar case management system for the FBI doesn’t run aground like the last one did.

“They work under the idea that everything needs to be kept secret,” says Sherry Higgins, the former project manager for the FBI’s $600 million IT modernization project. “But everything doesn’t have to be kept secret. To do this right, you have to share information.”

In addition, FBI officials have long marginalized the role that IT could play in connecting the dots between seemingly unrelated intelligence, evidence and field notes. (Consider this: Former FBI Director Louis Freeh didn’t even have a computer on his desk.) The result is a history of troubled IT projects, including a system to automate 35 million fingerprint files, which ran over budget by $170 million and was delivered years behind schedule. An updated system created to give local police and sheriffs the capability to check for stolen guns, stolen cars and other crime-related data also fell years behind schedule and ran over budget by more than $103 million in the 1990s.

But the VCF’s failure caught the public’s attention more than any other FBI IT boondoggle, primarily because leaders in the FBI and Congress labeled it as the government’s primary weapon to fight terrorism. Without it, the FBI’s ability to stop terrorist acts before they happen was seriously jeopardized. “This program has been a train wreck in slow motion, at a cost of $170 million to American taxpayers and an unknown cost to public safety,” said Sen. Patrick Leahy (D-Vt.), a ranking member on the appropriations subcommittee for Commerce, Justice, State and the Judiciary, at a February hearing.

Changing the culture at the FBI will be a gargantuan task, Azmi acknowledges. The job has been so frustrating that many top executives left after only short stints. Between 2002 and 2003 alone, four CIOs came and went. And the $170 million VCF system ground through 10 program managers before it was killed. To stop this merry-go-round of failure, top officials say, the FBI not only has to learn to share information, which means communicating more honestly and more frequently with executives and field agents, it needs to establish basic IT management disciplines. As the agency’s newly appointed CIO, Azmi is working to win buy-in from agents in the field so that the next case management system does not run aground. His team has almost completed an enterprise architecture that will lay out standards for a Bureauwide information system.

But like other large global organizations that have endured huge IT failures, the FBI must first understand and learn from the failure of the VCF project. The story of what happened with VCF and how an organization’s culture affects the success of IT development has lessons for both private- and public-sector CIOs who strive to make their organizations more competitive and responsive to their constituents.

Azmi likens his work at the FBI to “running a marathon when you are out of shape. Every mile is an uphill battle and an accomplishment at the same time. The race is painful and the 26th-mile marker far away.”

A New Mission for the FBI

VCF was the key component in Trilogy, the FBI’s effort to modernize its infrastructure. (Trilogy also called for providing agents with 30,000 powerful desktop PCs and a high-bandwidth network to connect FBI locations worldwide; both of those projects have been completed.) VCF was supposed to enable agents to more quickly and easily share information with each other worldwide. As FBI Director Robert Mueller recently explained to Congress, VCF would provide an electronic means for agents to globally send field notes, documents, pieces of intelligence and other evidence so that they could hopefully act faster on leads. A congressional investigation after 9/11 identified poor communication as one of the reasons that the FBI didn’t act fast enough on information about Middle Eastern men taking flying lessons. So agency executives hoped VCF would help move the balkanized structure of the FBI, in which agents rarely shared information on cases, to a more unified organization that would trade off leads in a real-time fashion. VCF would be “the first real change in the FBI’s workflow and processes since the 1950s,” according to a Justice Department Inspector General report.

At its core, VCF was designed to replace a paper-laden process with a Web-based one so that agents could attach to digital reports evidence such as photos, audio files, and possibly news and video surveillance clips. Agents could perform keyword file searches and control access on a need-to-know basis. Currently, agents use WordPerfect to write reports, notes and official documents, print them out, and then fax or send them via overnight mail to supervisors in other locations for approval. Once approved, the documents are faxed or sent back overnight. Clerks then key the reports into the system, a decades-old, Ada-based mainframe called the Automated Case Support System. The process takes days; VCF was designed to cut that time down to hours.

The VCF contract was awarded to Science Applications International Corp. (SAIC) in June 2001. The contract called for a Web-based front end that would link to legacy systems. Even when the contract was awarded, the FBI had yet to identify all of the requirements, Mueller told a Senate appropriations subcommittee in February.

But three months later, terrorists flew jets into the World Trade Center and the Pentagon, and the FBI was given a new mission: Stop terrorist attacks before they happen. The FBI scrapped the original VCF contract and determined that the new case management system must involve a complete replacement of its legacy systems to give the agency a more robust technology solution that would analyze clues, evidence and leads. In February 2002, the FBI asked SAIC to develop a much more ambitious VCF system that could manage millions of case files in a variety of formats, absorb hundreds of thousands of new case files every year and provide agents with a three-second response time to specific queries. Under pressure to respond to the terrorist attacks, the FBI asked SAIC to deliver the system in 22 months in a “flash cutover” strategy, which called for ditching the decades-old system for VCF overnight.

Yet even by then, the FBI had established few requirements for VCF. “Here is where SAIC made honest mistakes,” said Arnold Punaro, executive vice president of SAIC, in testimony submitted Feb. 3 to a Senate appropriations subcommittee. “We should have known that this approach was too ambitious…. It was here that SAIC should have made its concerns known to the director,” particularly concerning the flash cutover approach.

Mark Hughes, president of SAIC’s system and networks solutions group, which oversaw VCF development, says his team did express concerns about the drastically changing scope of the project to program managers at the FBI. But he suspects those concerns never reached Mueller. “We took for granted the message was being moved up the management chain,” Hughes said.

They shouldn’t have.

An Atmosphere of Secrecy

In late spring 2002, just three months into her new job as Trilogy program manager, Sherry Higgins wanted to know the answer to a simple question: Why did the Bureau’s top executives and the project’s contractor, SAIC, insist on using IBM mainframes as the platform for the VCF project, rather than cheaper and more efficient thin clients or Web-based technology?

Higgins couldn’t get a straight answer, so she called a meeting with executives from SAIC. She recalls that the answers she got to her questions were vague and evasive. At another meeting a few months later, Higgins asked point-blank why SAIC was dead set on mainframe technology. Finally, “they said that that was what the FBI directed them to do,” Higgins recalls.

Higgins then tried to find out why the Bureau wanted to pursue mainframe technology. She called another meeting with about 15 FBI IT experts and asked each attendee why the FBI had directed SAIC to use a mainframe platform. No one offered a reason, until Higgins called on Fred Dexter, then deputy assistant director for information resources management. As Higgins recalls, Dexter said, “It’s that way because the leadership just spent $10 million on a mainframe, and they don’t want it to go to waste.” (Dexter, who has since left the FBI, could not be reached for comment.)

To Higgins, his answer illustrates one reason why the FBI had so much trouble equipping its agents with high-tech systems and software to fight crime and terrorism. “They’ll throw good money after bad to cover up a bad decision,” says Higgins, who is now a senior consultant for the International Institute for Learning.

Higgins, a seasoned program manager who has handled major projects for AT&T, Lucent and the Olympics, quickly learned that the culture of the FBI was dominated by a secretive G-Man mentality. Secrecy may be useful in investigating crimes, but it created an atmosphere of distrust among field agents and managers in headquarters, Higgins says. The secrecy also bred a culture of intimidation, say former FBI and U.S. Justice Department managers. Many current FBI and DoJ IT employees, former FBI executives and Washington IT experts familiar with the FBI either declined to be interviewed for this article or requested anonymity. “There always has been a culture of intimidation at the FBI,” says one former IT manager. “It doesn’t surprise me one bit that people don’t want to talk.”

Underlying the machismo was an even deeper problem: VCF, if it ever worked as envisioned, would shake up the agency’s work processes and collide with the widely held perception among FBI agents that technology wasn’t a central part of helping them do their jobs. Agents view gathering human intelligence as one of the primary means of working cases, with computers providing a supporting role, says Frederick Bragg, a special agent in Syracuse, N.Y., and president of the FBI Agents Association. “You can sit at your computer all day and look at what other people already know,” he says. “Most agents would rather go out in the field.”

Bragg acknowledged some resistance to technology; however, he said agents are not opposed to anything that can help them in their job. Agents do not fear technology, he adds. But having seen the VCF failure, and experiencing unfulfilled promises of other technology advances to help solve crimes, agents have become wary. “They are now more circumspect. They have an attitude: We’ll believe it when we see it,” Bragg says.

Ken Orr, head of the Ken Orr Institute and a former member of the National Research Council (NRC) committee, which advised the FBI on the VCF project, agrees. “The problem at the FBI appears to be mostly cultural,” Orr says. “And that’s orders of magnitude harder to correct than a project management problem.”

The FBI’s dismissive attitude toward IT was embodied by former FBI Director Freeh, who ran the Bureau from 1993 to just before 9/11. “[Freeh] was not an IT person,” says a former DoJ IT manager familiar with the FBI IT culture. He and the businesspeople around him were uncomfortable within technology. “They only see IT as a back-office support function,” adds the manager, who asked not to be identified. (Freeh did not return phone calls to his office at MBNA America, where he is in charge of government affairs for the bank.)

Robert Mueller came in as the new FBI director in September 2001, promising a fresh start. He identified upgrading FBI technology as one of his top 10 priorities. Yet two years later, not much appeared to have changed, according to several people familiar with the agency. In the summer of 2003, while Mueller and then acting CIO Wilson Lowery were interviewing Jerry Gregoire for the CIO post, Gregoire, who had been CIO for Dell Computer and Pepsi Cola, says they asked him what he would do to get VCF back on track. In response, Gregoire asked whether the FBI had considered killing VCF and starting over. The answer he recalls is, “We can’t because that’s not how things are done here, because you have to take the heat and go to Congress.”

He adds, “I came out of that meeting saying, Holy smokes, I don’t know if I can help them.” Gregoire was offered and accepted the CIO job in September 2003. But within days, Lowery called him at his ranch in Austin, Texas, asking him to send a letter declining the offer, according to Gregoire. (Despite persistent requests for an explanation of why the FBI was withdrawing the offer, Gregoire did not receive any reason for the reversal.)

Gregoire was interviewed a few months after the quick departure of Darwin John, the former CIO for the Mormon Church and Scott Paper. John lasted only 10 months as FBI CIO before leaving due to what he would only describe as a disagreement on “a matter of principle” with Mueller. Looking back on his tenure as CIO, John describes the job of transforming the FBI IT operations as “the Mount Everest of IT challenges.”

A Train Wreck in Slow Motion

That was the culture Azmi inherited in December 2003, when he left his CIO post at the U.S. Attorneys’ Office to accept the acting CIO position at the FBI. (He was officially named CIO in May 2004.) When Azmi came aboard, one of the first questions he asked FBI management was how big his IT budget was. The answer Azmi received, delivered with a straight face, was $5,800. The reason: Nearly the entire FBI IT budget was controlled by field offices and other divisions in the Bureau, not the headquarters CIO. “Nothing really shocked me more,” Azmi says.

Azmi also learned that the FBI had not fully complied with a law—signed by President Clinton seven years earlier—that mandated the responsibilities and competencies for federal CIOs. For instance, the FBI hadn’t even developed basic IT policies, an enterprise architecture, a portfolio management strategy or a strategic plan. “I was surprised to see that an organization as large as the FBI, and with such a critical mission, [did not have in place] some of these basic things,” Azmi says, shaking his head.

As for VCF, Azmi says he noticed immediately that there were too few FBI technicians compared with the number of SAIC contractors. As a result, the FBI could not keep up with communicating the ever-expanding requirements for VCF to SAIC developers. SAIC was filling in the blanks and making too many key development decisions, he says.

To meet the December 2003 deadline, SAIC created eight software teams, which included 250 full-time positions, to write software and develop applications to meet ever-evolving requirements. Turnover at the FBI caused the requirements to change frequently, both Hughes and Punaro say. Since November 2001, the FBI had 19 IT management changes. “Each change brought new directions, a different perspective on priorities and new interpretations to requirements,” Punaro wrote in his testimony to Congress. In all, the FBI asked for 36 contract modifications, an average of one and a half change orders per day. The numerous requirement changes were “the most damaging aspect,” Punaro testified. As a result, the price for the entire Trilogy project ballooned from an original estimate of $379.8 million in 2000 to $596 million by 2004, according to the U.S. Government Accountability Office.

In December 2003, SAIC delivered what it later described as a “snapshot in time” of the software for VCF. An independent evaluator, Aerospace Corp., tested the system as if the copy were a finished product and found numerous deficiencies. The NRC also delivered a report in May 2004 to the FBI, concluding that “the FBI’s IT modernization program is not currently on a path to success.” The report recommended the Bureau abandon its big-bang implementation in favor of an incremental approach and develop better program management skills and personnel.

These reports convinced the FBI that its flash cutover strategy was too ambitious. In June 2004, the agency changed its approach, requiring only that an initial operating system focusing on workflow processes be delivered by December 2004. The full operating VCF system would be delivered in 2005. SAIC delivered an initial VCF workflow process system on time, and in January, the initial VCF system was delivered to the FBI’s New Orleans office for agents to test. Azmi traveled to New Orleans in March to meet with the agents and collect their reviews. He came back with bad news for Mueller. Two months after the system was deployed in New Orleans, with the FBI having spent 1,800 hours training 240 agents to use the system, Azmi reported to Mueller that the program was too complicated for agents to use.

A week later, Mueller finally informed Congress that he was going to do what IT experts say should have been done years earlier: Kill VCF.

Airing Out the FBI

Now a year into his job as the FBI’s fifth CIO, Azmi has made some progress in building a firmer foundation for IT. In February, Mueller finally gave Azmi budget authority over the Bureau’s IT budget—a rare occurrence for the federal government, in which most budget power resides with federal CFOs or with IT managers in divisions deeper down the agency’s organizational chart. The budget authority will give Azmi the power to more easily consolidate IT applications and systems, and dictate standards across the agency. Azmi says he also recently solved some issues with the Bureau’s enterprise architecture, and now he can consolidate systems and develop new ones based on the agency’s mission to combat crime and terrorism. Azmi is close to identifying all of the FBI’s IT projects to fulfill his goal of establishing a portfolio management plan. He is in the process of creating a five-year strategic plan to guide IT development and spending.

Azmi is also trying to change the agency’s traditional skepticism of IT. He meets regularly with key stakeholders, such as the special agents advisory group, to solicit advice on functions for the new system to replace VCF. He also has appointed a visiting special agent (who rotates in every six months) to his staff to inform Azmi and the IT department about which technologies field agents need. In turn, the visiting agent learns how the CIO office and IT operate, information that he can take back to fellow agents. And every six months, Azmi calls an “all-hands meeting” with the entire 500-member IT staff to discuss how things are going.

In March 2005, Azmi delivered on the first promise he made when he was named CIO. The help desk typically closed at 7 p.m. EST, after which technicians were reachable only by pager. Last year, Azmi promised agents worldwide that they would have access to a round-the-clock help desk. In March, Azmi told agents that the help desk was open 24/7. I want them to know that “if they’re on the job, we’re on the job,” Azmi says.

In late May, the FBI announced it would build a new case management system called the Sentinel in four phases. The agency is expected to issue an RFP for the project this summer. Azmi says he plans to deploy several capabilities, including workflow, document management, record management, access control, audit trails, single sign-on and PKI applications. Azmi realizes that the transition to the new system will require winning over agents first. To manage expectations, he plans to communicate often and pour lots of resources into training agents.

“We want to automate those things that are the most manually cumbersome for the agents so that they can see that technology can actually enhance their productivity,” he says. “That is how to change their attitudes.”

So far, Azmi has received high marks for what he has been able to accomplish. “He’s made an improvement over time,” says Randy Hite, the GAO’s director for IT architecture and systems issues. “There are a lot of things contributing to the challenges for the organization, and some are being dealt with now.”

But Congress and other government and private watchdog groups continue to cast a critical eye on the FBI’s efforts to join the 21st century. In a recent report, the National Academy of Public Administration, an independent government advisory group in Washington, D.C., raised the specter that Congress may have to pass a law requiring the FBI to develop a comprehensive information-sharing process if the Bureau does not immediately improve.

Hite says the Bureau still has a long way to climb toward elevating IT to the level in which it supports the FBI in meeting its mission of fighting crime and analyzing intelligence to fight terrorism. “It’s a quantum leap from defining IT policies to making sure they are followed by the people out there managing projects,” he says. “A kind way to describe the FBI is to call it a challenged organization.”

Azmi is aware of the mountain that faces him—not to mention the consequences if he fails to deliver the support systems the agents need to fight against high-tech crime and terrorism. “Looking at the mission of the FBI and how critical it is, I will tell you that we are at war,” he says. “And the best tool we have is information, and if information doesn’t get to agents on the street in time, then we haven’t done our job properly.”