The U.S. Department of Health and Human Services (HHS) has a trillion-dollar budget — and a lot of legacy software in need of modernization.
John O’Grady, acting director of the digital service division in the department’s Office of the Inspector General, is responsible for delivering the technology used by OIG staff to oversee Medicare, Medicaid and 100 other federal programs.
“Over the past four years, we’ve been leading a large-scale initiative to modernize all of our agency’s legacy applications. It’s a pretty big task: We have more than a hundred,” he says, adding that the project aims to save taxpayers’ money and help auditors and investigators more effectively promote patient safety, prevent prescription drug abuse, and strengthen protections against Medicaid fraud.
One of the first steps in dealing with 30 years of accumulated technology debt was making a shift to agile development and DevOps. To do so, O’Grady brought in external resources to coach the team in agile and lean methodologies.
“We’re training people up on agile and lean, getting them coached up, explaining what a good product owner is and what we’re going to need from our customer base,” he says.
But there was a catch. “Following agile tenets and trying to get 80 percent of a product owner’s time to actually focus on a project was very tough.”
Of the 20 or so staff driving one of the bigger modernization projects currently in progress, three are working solely on organizational change management, promoting the benefits of the new development methodology and, more broadly, the changes to the application.
Low-code, outsourced development
With the change in development mindset came a move to new tools. “We started off trying Python development, and we’ve built a few applications in that that still exist to this day and are on a fairly modern stack,” O’Grady says.
In the end, HHS decided to take a low-code approach, centered on ServiceNow. Best known for its IT service management platform, ServiceNow has repositioned itself as a cloud-based workflow management tools provider under the leadership of former SAP CEO Bill McDermott.
O’Grady says ServiceNow fits well with the HHS’s hybrid cloud environment — a mix of AWS, servers in an Equinix data center, and applications running in the Office365 and DocuSign clouds.
In 2019, 13 applications built using ServiceNow reached user acceptance testing or were deployed as minimum viable products, says O’Grady, who heads up project management, with much of the programming outsourced to ServiceNow partner ITG, which was recently acquired by another government contractor, ICF International.
Across the department, O’Grady manages around 110 contractors developing software. Two things have made that easier: “We’ve put Scrum masters on each of the projects. And we’ve got very good project directors over each of the various contract teams,” he says.
His top tip: Establish solid relationships and trust early in the process.
Breaking up the monolith
Having tested the waters with smaller projects, last year the team set about modernizing one of the agency’s largest applications, its investigation management system, a mainframe port dating back to July 2003 — before business intelligence, data mining, mobility and social media were common concerns.
Not surprisingly, the system has accumulated an exhaustive list of requirements to make it more effective and innovative — such as roles and permissions, and task routing. On top of that, the existing system is case-centric rather than subject-centric.
“That doesn’t really help you when you’ve got a hundred John Smiths all at the same address, with the same NPI number, realizing, ‘Hey, this is really one guy related to 120 cases,’” he says.
Moving to a subject-centric data model in an all-or-nothing rewrite would have involved a massive effort to migrate data from the old Oracle database. Instead, O’Grady is taking a different approach.
“The way we’ve tackled it, it’s not going to be a big monolithic investigative management system anymore,” O’Grady says. “It’s going to be a suite of investigative tools that all interlink and talk to each other that investigators can use.”
In phase one, beginning late last year, the team developed enhancements to the application as a learning exercise — and to get a handle on the task ahead of them, which O’Grady says is “absolutely huge.”
Phase two, now under way, will tackle the subject-centric model and build up the data intake process.
However quickly they code, the legacy system will be around for a while yet: Some of the cases it holds have been open for five or ten years. The new tools they are developing continue to exchange data with the old app in a way transparent to the user, who should be able to access all functions in either system.
Federal funding: The biggest challenge
“The biggest challenge is really a governmental challenge,” O’Grady says. The nature of federal funding makes it difficult to budget for such a long project. “It’s really hard to know what you’re going to have from one year to the next, never mind five years down the road. And these things really are multi-year efforts.”
Although application modernization will benefit the department and save money in the long run, management has to be convinced it’s worth spending a little more right now. “That’s the tricky bit, especially with the turnover in government,” he says.
It’s important, therefore, to seize the opportunity presented by periods of relative stability to make big changes in the code.
“We’ve been very lucky in that we have an excellent executive management team that’s stable, but my previous years at OIG, it was a revolving door,” says O’Grady. “I think my first two years we went through six CIOs or something like that. It’s incredibly hard to get any traction and continue on along a plan, especially a five-year plan, when that’s happening.”