Five years and $405 million into the FBI\u2019s effort to create an all-digital case-management system for its special agents, it\u2019s clear how far the Sentinel project has slid off the rails. According to the inspector general at the Department of Justice, many promised functions remain undelivered and defective code inhibits the use of some of the pieces of the system that are already in the FBI\u2019s hands.\n \n But FBI CIO Chad Fulgham insists Sentinel, which he inherited when he joined the bureau in December 2008, isn\u2019t doomed\u2014even though independent reviewers estimate it could take another eight years to complete, at an additional cost of $453 million. To save it, the former Lehman Brothers IT executive is battling what he calls \u201cold guard\u201d thinking and a federal bureaucracy that he says is \u201cnuts.\u201d\n \n Sentinel is a big problem, but it\u2019s just one of several Fulgham faced when he arrived. He was surprised to also find outdated technology, 200 open IT positions and a convoluted federal purchasing process.\n \n Like any new CIO, in government or the private sector, Fulgham had to assess what was working and what wasn\u2019t, triage immediate problems and build relationships with colleagues who could support his long-term agenda. Congress and various government oversight groups want action, just as a corporate CEO and board of directors would, observes Charlie Feld, CEO of the Feld Group, an IT turnaround consultancy.\n \nFeld, who has been CIO at Delta Airlines, Burlington Northern-Santa Fe railroad and Frito-Lay, says that upon starting a new job, a CIO carries the most political capital he\u2019ll have for the next two years. And he needs to spend it \u201ceven if it pisses people off,\u201d Feld advises. \u201cSenior executives can take bad news. They just can\u2019t take not knowing or continuous streams of bad news that build uncertainty.\u201d\n \n In the past two years, Fulgham has drawn heavily on his political capital, using his corporate experience to bring the latest IT-management techniques and new technologies to the FBI. This includes collaboration software, mobile applications and a modern network infrastructure. Nowhere has Fulgham spent more of his political capital, however, than on his remediation plan for Sentinel. With support from FBI Director Robert Mueller, Fulgham has restructured parts of the original project agenda. He has, for example, traded traditional sequential software development for agile methods and will replace a Sun Solaris platform with virtualized Red Hat Linux servers. \n \n He also hopes that throttling further participation by contractor Lockheed Martin will get Sentinel done, within budget, by September. That\u2019s when Mueller\u2019s 10-year term ends. \u201cWhen we release this, FBI will declare victory on Sentinel,\u201d Fulgham says.\n \n The inspector general, however, calls Fulgham\u2019s plan optimistic, noting the FBI\u2019s inexperience with agile development. Delays, these overseers warn, are apt to pile up as the bureau changes course. The most difficult phases of the project are ahead, they say.\n \n Fulgham, 36, doesn\u2019t flinch. FBI agents and staff, he says, \u201clove what we\u2019re doing. But the old guard that\u2019s used to doing things one way,\u201d as Fulgham calls his critics, \u201cthey\u2019re not exactly happy with the Chadster.\u201d\n No More BoondogglesFBI technology hasn't ever been much to brag about. The agency has failed to build a fully electronic case-management system since coming up with the idea 11 years ago, with efforts so far costing taxpayers at least $575 million, plus an inestimable toll in less-efficient crime fighting. After the 9\/11 terrorist attacks, critics faulted the FBI and intelligence agencies for not sharing information effectively. Technology failures were a core problem.\n \n \n \n There have been some improvements: FBI employees can process some forms digitally, for example, and the CIOs of the largest intelligence agencies meet regularly to devise ways to share information and trade project advice. But lacking a fully digital case-management system means FBI agents still spend a lot of time shuffling paper. Sentinel\u2019s electronic workflow would speed up and improve analysis of case notes and evidence, as well as facilitate better coordination with fellow law enforcers.\n \n \u201cWe must make sure agents and analysts have the technology tools they need to do their jobs. I won\u2019t accept any more techno-boondoggles,\u201d Sen. Barbara Mikulski (D-Md.) said in an interview. Mikulski chairs the Senate subcommittee that oversees the FBI\u2019s $8.3 billion budget.\n \n In 1995, while corporate CIOs were building distributed applications running on PC servers and diving into Web technology, the FBI launched a system for managing case files and evidence\u2014the Automated Case System (ACS)\u2014that used green-screen, mainframe software. By 2000, the FBI was planning to replace ACS as part of a larger modernization project called Trilogy. Within Trilogy, the Virtual Case File was to have been a set of applications that eliminated paper forms that agents had to fill out by hand, route around the office in file folders and store in metal drawers.\n \n After 9\/11, Mueller accelerated Trilogy and got approval from Congress to create a CIO position. But in the next three years the bureau went through four CIOs, seeking the right fit in terms of experience and management style. In 2005, the FBI canceled the Virtual Case File amid reports from the Government Accountability Office that the bureau wasn\u2019t watching its contractors well enough and couldn\u2019t account for 1,200 pieces of missing equipment worth $7.5 million. Worse, the GAO said, the unfinished project, which to that point had cost $170 million, was poorly designed and relied on custom code that would be difficult to maintain.\n \n The FBI began planning Sentinel in 2006, awarding a six-year, $451 million contract to Lockheed Martin. In Senate hearings last April to review the FBI\u2019s fiscal 2011 budget request, Mikulski recalled the agency\u2019s checkered IT past, telling Mueller she was concerned that now Sentinel was in jeopardy of becoming \u201ctechnojunk,\u201d just as Virtual Case File had.\n \n \u201cWhen you have a project that goes over four or five years,\u201d Mueller responded, \u201csome forms of delays are, I wouldn\u2019t say inevitable, but need to be identified and addressed and contained. I think we\u2019ve done it here.\u201d\n Fix Something FastBefore joining the FBI, Fulgham was a senior vice president at Lehman Brothers, directing its worldwide identity management, messaging and unified communications teams. He helped to manage projects using well-known agile principles, such as short development cycles with frequent prototyping. He also worked directly with vendor developers to speed projects along.\n \n \n \n In 2008, as Lehman collapsed into bankruptcy, Fulgham interviewed for the FBI CIO position. Then-CIO Zalmai Azmi was set to leave in November after five years in the job. Fulgham had doubts about Sentinel. Corporate IT executives learn to avoid such megaprojects because they have high failure rates, he says. \u201cFrom the outside,\u201d he recalls telling Mueller, \u201cwe don\u2019t think you can pull it off.\u201d \n \n He took the job anyway. Fulgham spent his first few months at the bureau examining its IT procedures and infrastructure, although it didn\u2019t take him long to find out that more than Sentinel needed fixing. On his first day, his PC wouldn\u2019t boot. Nor was his BlackBerry configured. On his second day, he turned on his PC to reveal that the standard desktop image at the FBI was Windows 2000 with Corel WordPerfect 8. \u201cI really was disheartened,\u201d he says.\n \n But he saw the chance to make fast improvements that played to his strengths. At Lehman, he had planned and implemented unified communications tools from Microsoft for 28,000 employees worldwide. One project integrated Microsoft Communicator instant messaging, voice-over-IP (VoIP) and desktop videoconferencing tools with Cisco\u2019s Call Manager and Unity to give Lehman employees one dashboard for managing their communications. At IBM Global Services and JPMorganChase before that, he had led teams responsible for securing networks and mitigating security risks.\n \n Accordingly, he spent much of his first year replacing the FBI\u2019s 1990s-era ATM and frame relay network with revamped cryptography and new Cisco gear that supplies 45 times the bandwidth to 800 local offices nationwide. More than 27,000 agents, analysts and other employees got BlackBerry smartphones. RSA identity-management tools from EMC let authorized FBI employees use those BlackBerrys to access the secure National Crime Information Center database of criminal records while working in the field.\n \n Meanwhile, 30,177 desktops and laptops were replaced and almost 100,000 are being upgraded to Office 2007. Unified communications software, also from Microsoft, includes SharePoint for collaboration via instant messaging and document sharing. The so-called next-generation workspace sports 24-inch monitors and advanced cameras and headsets for peer-to-peer VoIP videoconferencing.\n \n Helping to smooth the rollout of these new technologies was a reorganized IT staff. Fulgham filled the open positions, embedded IT staff in user groups and established a chief marketing officer position to advertise new features and capabilities to FBI employees.\n \n The modern infrastructure has already improved coordination and workflow, he says. In October, the bureau arrested 133 public officials and others involved in a massive corruption case in San Juan, Puerto Rico. One thousand agents and staff fanned out across the island in the largest single operation in FBI history. The takedown was coordinated in real time using SharePoint and the other new UC and mobile tools, he says.\n \n After the raids, Mueller demonstrated the new technology to Attorney General Eric Holder. \u201cI had smile on my face like the Cheshire cat,\u201d Fulgham recalls.\n Tackling TroubleSuch quick IT wins can supply a new CIO with the momentum needed to set a troubled project right, says Brad Brown, director in the IT practice at McKinsey. And when it\u2019s time to tackle the problem project, Brown advises CIOs not to halt the whole thing, if possible. Instead, find one piece that can be finished right away while senior leaders work with the CIO to dissect the larger project plan.\n \n \n \n That approach accomplishes two things, he says. First, keeping at a clearly defined task that is important to the project prevents the organization from falling into chaos. Second, delivering that piece improves morale and gives the team a success to build on. \u201cYou\u2019re trying to build trust at the same time you\u2019re making change,\u201d he says.\n \n The CIO must emit self-confidence and confidence in the organization, he adds. \u201cIf a project\u2019s in trouble, there\u2019s angst. People look for a lifeline and a way forward.\u201d \n \n Fulgham\u2019s call to action came in late 2009, when Lockheed missed its first major deadline under his watch, then had trouble resolving problems with electronic forms it had built. Immediately, Fulgham demanded regular demonstrations of the in-progress code for senior executives from both Lockheed and FBI. He also began quarterly meetings with Lockheed CEO Robert Stevens and Linda Gooden, the executive vice president of Lockheed Martin Information Systems and Global Services, along with Mueller and other FBI leaders. Having top executives in those meetings escalated the issues, he says, \u201cso they could see firsthand what my customers would see.\u201d\n \n When trouble continued, Fulgham issued a stop-work order to curtail Lockheed\u2019s remaining work on Sentinel. He also directed Lockheed to correct defects in the pieces of Sentinel it delivered in 2009 and 2010. Among them, according to the inspector general, was a security setup that could allow a user to create a fake identity to electronically sign documents, which resulted from a disagreement between Lockheed and the FBI about how to interpret a digital signature requirement.\n \n Meanwhile, Fulgham formed plans to switch to agile development, which required a smaller team than the group of 200-odd Lockheed personnel assigned to Sentinel. Fulgham instead wanted developers who were intimately familiar with the key products Sentinel would use, including Microsoft search, Oracle databases, EMC document-management software and Adobe document-creation tools.\n \n Fulgham dialed the vendor CEOs, including Steve Ballmer at Microsoft and Joe Tucci at EMC, to explain his plight and enlist their help. \u201cNot many people turn down phone calls from the FBI,\u201d Fulgham notes. \u201cThey were ecstatic because they weren\u2019t able to make their products successful\u201d in Sentinel the way the project had been managed.\n \n A fresh team of about 50 vendor developers and support staff, along with the bureau\u2019s own IT staff, now deliver a piece of Sentinel every two weeks. Fulgham and others at the FBI see demos every other Friday. Fulgham\u2019s point man on Sentinel, CTO Jeff Johnson (with whom Fulgham worked at Lehman for four years) sits with the developers. Some pieces of Sentinel now have been successfully rolled out, such as a module for writing interview reports and sharing case leads.\n A New CourseA new CIO has two years to prove himself, according to Feld, the turnaround expert. That\u2019s enough time to have completed at least one big innovative project and have corrected the trajectory of inherited problems, he says. \u201cBy three years, he\u2019s either really solid or gone.\u201d\n \n \n \n As Fulgham enters year three, he counts several successes and has set a new course for the mess he now owns. This year he plans to finish Mobile FBI Net, a project to allow secure access to the bureau\u2019s secret network from outside FBI facilities. For example, a field agent who finishes an interview related to an open case would be able to hold a secure videoconference about it with the agent managing the case at headquarters.\n \n Meanwhile, some of the most substantive parts of Sentinel remain to be done. For example, only four of the 18 forms routinely used by FBI employees have been recreated digitally. Yet fully computerized workflow for investigation approvals, evidence handling and other tasks can\u2019t be deployed until all the right forms are available, Fulgham says.\n \n Perhaps most critical is making the 8.3 million FBI case histories that reside in the ACS mainframe system fully accessible to special agents using their new workstations. The original plan called for Lockheed to migrate ACS data to Sentinel. Now Fulgham has decided Sentinel users will tap that data using enterprise search technology from Microsoft. The FAST search product will index and search the FBI\u2019s proprietary mainframe data, as well as newer Oracle databases and the FBI\u2019s new EMC Documentum document-management system.\n \n Fulgham thinks that enterprise search and analytical tools will allow special agents to look at relationships among different kinds of data, including the FBI\u2019s case histories, geospacial information, link analysis and repositories from other intelligence and law-enforcement sources.\n \n The inspector general is doubtful Sentinel will be done by September, within budget, as Fulgham has promised. An October report notes that two of the four original project phases have yet to be built, and the parts of Sentinel deployed so far are used by only one-third as many employees as regularly use the old ACS mainframe system. Of Fulgham\u2019s decision to reduce the size of the development team, the report notes wryly: \u201cwhile useful in keeping costs down, [the smaller staff] may not be sufficient to complete the system.\u201d\n \n In a written response, FBI Associate Deputy Director Thomas Harrington said the report doesn\u2019t accurately reflect how the FBI is handling Sentinel now. It includes, he says, \u201can inflated cost estimate for completing Sentinel that is based on a worst case scenario for a plan that we are no longer using.\u201d\n \n The inspector general\u2019s criticism is misplaced but understandable, Fulgham says. \u201cThe agile method is very new to the federal government. The IG is going to look at any new strategy with skepticism.\u201d He has offered to train auditors in agile \u201cso they can better understand what [CTO] Jeff [Johnson] and I and others take for granted.\u201d\n An End to Old Ways?Troubled projects are difficult enough to mend in the corporate realm, but would-be fixers at the federal government face many more obstacles, says Feld. Once expensive, multiyear endeavors get underway, federal procurement rules and other regulations make them tough to stop. For example, the FBI doesn\u2019t yet know how it will resolve its agreements with Lockheed, an FBI spokesman says. And the bureau hasn\u2019t revealed any estimates for the cost of getting out of the contracts, the inspector general says.\n \n \n \n Meanwhile, circuitous procedures govern everything from how products can be bought to how staff can be organized. Fulgham was shocked to learn that he can\u2019t negotiate contracts with vendors directly: That\u2019s done by contracting officers. \u201cI thought, \u2018Are you nuts? Why did you hire me?\u2019\u201d he says. \u201cYou pay executives to make these decisions. I know what things are worth and can play one vendor against another.\u201d\n \n John Reece, former CIO of the Internal Revenue Service and now a consultant to the federal government, sympathizes with Fulgham. Reece joined the IRS in 2001, having recently retired as CIO of Time Warner, to straighten out an IT modernization project begun two years earlier. \u201cI was facing exactly the same barriers,\u201d he says. \u201cProcesses take forever.\u201d\n \n Fulgham, Reece says, has every reason to be frustrated. He cites one recent study by the Department of Homeland Security which found it takes 250 days, on average, to get an IT procurement request approved.\n \n Fulgham says he\u2019s shared his views with federal CIO Vivek Kundra, who has pledged to reform IT management processes, including procurement, so government IT projects can proceed faster and make better use of emerging technology. Changes announced in December call for Congress to work with federal CIOs and the Office of Management and Budget to overhaul budgeting models for large projects to allow for modular development\u2014just the approach Fulgham is now taking with Sentinel. Kundra also plans to work with federal CIOs to redefine their roles, so they have, for example, more control of commodity IT purchases.\n \n Despite the shivers they produce for many CIOs, megaprojects will never go away, even in the private sector, says McKinsey\u2019s Brown. At some point, every organization must undertake a major platform or architecture upgrade. \u201cBut constructing these large projects well,\u201d he says, \u201cis the hardest thing we do as technology professionals.\u201d\n \n Fulgham\u2019s five-year strategic plan doesn\u2019t include any such monsters; important projects are broken down into manageable pieces. He intends to increase the FBI\u2019s use of small-team agile development. Other goals: consolidate data centers, replace a proprietary network security system with an off-the-shelf product, and redesign the unclassified, secret and top-secret networks for a single look and feel girded with multiple levels of access controls.\n \n \u201cIT is all about helping us conduct investigations faster,\u201d Fulgham says, and successful deployment of Sentinel will go a long way toward that goal.\n \n But September is coming up fast.\n Follow Senior Editor Kim S. Nash on Twitter: @knash99.