17 Steps to Implement a Public Sector Big Data Project

In the private sector, companies with massive volumes of data at their disposal, like Amazon and Facebook, have made millions of dollars leveraging that data with analytics. Public sector agencies, too, are data-intensive, and increasingly agencies at every level of government are looking to leverage their data to better serve citizens.

In a recent report for the IBM Center for The Business of Government, Kevin C. Desouza of Arizona State University, laid out the following public sector big data implementation plan, drawing on interviews with CIOs from every level of government.

Big data projects are complex undertakings at best. This holds especially true in the public sector, where such projects often require large infrastructure changes, program designs and agreements across agencies and departments. The first and biggest stage of any big data initiative is planning your project. Attention to detail can make or break your project before it even begins.

“The planning phase includes conceptualization of the project, which is vital for establishing a platform for success and ensuring that stakeholders are properly informed,” Desouza says. “This is an opportunity to lay the foundation of a quality project.”

First, determine what big data can and cannot do for your organization. You need to learn how big data can benefit your organization and what the risks and challenges are. Think through the complexities of governance and policies in place around data, processes and systems — especially if you have outdated policies that don’t account for current technologies. Understanding the policies of other agencies to identify shared constituents and minimize duplication of effort can pay big dividends.

“Dig into examples and look at what has worked and what has not and even contact individuals who have been featured in press stories,” Desouza says. “If CIOs do not have time to do their homework on big data, they should probably not commission a big data project.”

Articles and whitepapers rarely talk about big data project failures. Professional networks are essential to getting that information. Talk to peers at other agencies, academic institutions, think tanks and the private sector.

Build an advisory group within your organization to both extend your influence while also helping you place big data within the context of your working environment.

“Coalitions can go a long way in furthering agendas and creating inroads to new partnerships or information,” Desouza says. “CIOs will need to perfect their ‘elevator pitch’ for big data to engage people in a coalition. The elevator pitch should explain how an investment in data management will allow the agency to tackle an existing problem more effectively and efficiently or take advantage of a new opportunity.”

For your first big data project, focus on something that directly benefits citizens and stakeholders. This will draw attention and critical thought. Get down to specifics later; for now, it’s about the broad opportunity.

“Keeping the opportunity broad at the start allows CIOs more flexibility to engage other stakeholders and give them an opportunity to shape the program,” Desouza says. “A common strategy employed by CIOs is to outline the broad opportunity in the form of a working paper or position paper. This paper looks at the opportunities that exist within an agency for superior data management. The working paper then becomes the platform for having strategic discussions and deliberations.”

The best place to begin is with the easiest opportunities. Begin a project by tackling public data rather than getting involved with private data. Modernize existing technologies and processes for efficiency before creating new processes.

“CIOs that have witnessed success with their big data efforts note that they began by addressing problems that were simple, yet were visible pain points for an agency,” Desouza says. “Choosing the visible pain points and building a data-driven solution helps win support for the overall program.”

This is also a time to build a map of data elements and their interconnections. These maps can help you uncover data dependencies, interactions among data elements and organizational and political elements.

Build alignment between your big data project and other organizational efforts or risk having your project perceived as a distraction from core efforts that pulls away valuable resources. One way is to embed phases of a big data project into existing IT efforts. For instance, weave data governance issues into every IT project.

It is also essential to line up a sponsor from senior management.

“These projects need a sponsor — someone who is willing to champion the project during moments of controversy or discomfort,” Desouza says. “It is important that someone with clout is willing to weather the proverbial storms that often accompany the initiation of big data efforts.

It is easy to overlook or even undermine privacy with a big data project. CIOs need to adopt the role of privacy advocates when undertaking these projects, especially since existing privacy laws may require updating as a result of new technologies.

“CIOs should be acutely aware of privacy and security considerations as discussions on data are taking place,” Desouza says. “This will be critical for project success. Ultimately, if CIOs are aware of these issues and advocate for care in their handling, this will be reflected positively in how the project proceeds and is perceived by stakeholders. Privacy and ethical considerations around data collection, integration, analysis and dissemination should be discussed openly and sincerely. Seeking clarity from legal counsel is essential.”

Build a taskforce with both technical and organizational expertise to oversee the project. Ideally the taskforce will include representatives from the IT team who understand the technology, representatives from the business side who perform the tasks that generate or use the data being managed, and representatives who understand the legal and governance restrictions on the data in question.

“Each of these perspectives is valuable and must be included so as to ensure that the big data project does not run into any major surprises,” Desouza says. “One of the critical roles to assign to the taskforce is that of the spokesperson. Ideally, there should be one individual to give regular updates to stakeholders and keep the senior sponsor apprised of any issues.”

Expect resistance from parts of your organization. The best way to overcome it is to determine the likely sticky areas ahead of time.

“One CIO interviewed for this study notes that his city’s open access program caused internal strife because it gave city employees access to other city employees’ information, resulting in discomfort throughout the organization,” Desouza says. “There will be political repercussions for analyzing data that was never looked at before. This is especially true if the big data project has anything to do with increasing efficiency of operations. Outlining the various sources and types of resistance upfront can help CIOs build an educated campaign and pitch for the project.”

You need to develop key performance indicators (KPIs) around your big data project that focus on both process and outcome measures. Process measures are about improving efficiency; they capture gains in quicker completion times, lower costs of operations and so on. Outcome measures are about customers’ perception of the service; these measures include improved customer service, increased customer value and so on.

“Baseline data on organizational processes should be captured before the project begins. This will allow meaningful comparisons of outcomes, both before and after project commencement,” Desouza says. “Performance indicators should make sense to the business units involved and offer information on what the unit actually needs (not useless esoteric measures).”

Public sector databases contain citizens’ data, making them valuable targets. You must assesses the potential impact of compromised data and develop a risk mitigation plan with processes for reducing the risks.

“It is important to consider who has access to data, how much sensitive information is returned when database queries are made and what the physical security surrounding server rooms is,” Desouza says.

He notes that you should also develop a communications plan alongside the risk mitigation plan to ensure that messages are accurate and advance the goals of your agency or program. The communications plan should include dealing with press, academia and other agencies.

With the planning stage complete, it’s time to put the gears in motion. The effectiveness of your planning in the previous stage will play a big role in your success, but good project management at this stage is equally important.

“Executing a big data project requires ongoing attention from the project’s advisory group and the staff managing the project,” Desouza says. “Learning and establishing best practices for project management is important. Organizational proficiencies or inefficiencies can bring about the success or failure of the project.”

There’s no way around it: CIOs need to consistently monitor the project status to get in front of major problems and allow for the development of creative solutions. Desouza says many CIOs use formal or informal dashboards for their projects that leverage the KPIs developed in the planning stage.

“CIOs say they need to regularly check the pulse of the program both from a process and outcome perspective,” Desouza says. “In addition, they need to constantly gauge the conditions in the environment, especially in terms of any sentiment toward the project. Appropriate and timely communications, along with other interventions, can help address the issues and nip potential problems in the bud.”

Communication is vital at every step of your big data project. Success in big data requires breaking down silos of data and information, and that makes sharing the information you have essential.

“Communication about milestones, inefficiencies, successes and failures will help an agency and peers gain a better understanding of big data,” Desouza says. “In instances where data is shared between agencies, constant coordination, communication and feedback is necessary to ensure mission success.”

Executing a big data project is difficult enough without the inevitable scope creep that comes as stakeholders see progress and think of additional ways to use the data. You don’t want to continuously adjust the project plan and deliverables.

“It is critical that CIOs keep a watchful eye for scope creep and be clear on the boundaries of the current effort and how future revisions and additions will be made,” Desouza says. “One approach might be to take the model that Google follows and release products in beta.” By doing so, you can capture new ideas and work them into the next release or update.

Big data technologies are evolving at an exceptional pace. But your project may not need all-new technology. Maybe you can repurpose existing technology assets. Stay focused on managing your data. With a clear view of data management issues from an organizational and policy viewpoint, it should be relatively easy to choose the appropriate technology.

“Multiple CIOs report that the minute their agency announces an effort on big data (or any other major data activity), they are bombarded with calls from sales consultants who inquire and try to sell products and services,” Desouza says. “Having a clear focus on the goal of the project, which is to leverage data and manage it more effectively toward a business outcome, helps keep everyone focused.”

Sometimes big data projects fail. But it can be hard to pull the plug, given sunk costs. In these situations, failure to call it quits will not only make the situation worse, it could hurt the state of your entire IT department.

“One strategy suggested by a CIO is to outline clearly at the project’s beginning the conditions under which the project would be stopped,” Desouza says. “Thinking through these upfront not only helps in setting realistic expectations of the project, but also will sensitize the team to look for signals of trouble and discuss them openly during the team meetings.”

Once your big data project is up and running, you’re not done. It’s time to review what the agency accomplished—including what went well, what failed and what could have been done better—and to plan for the next project.

It is a good idea to document the entire project, including lessons learned from all stages, to retain the institutional knowledge you gained as a result. This information can also be shared with peers.

“One important element of conducting a postmortem is that it should not be used for evaluation or to point figures at individuals or events,” Desouza says. “Unless people are protected to share their true experiences and learning episodes, the postmortem exercise will not be of any value.” Also, conduct a thorough impact analysis to convey the value of the project, accounting for improvements in both process measures and organizational value measures. Once it’s done, publicize it.

The results of your big data project and the lessons learned from it should help you identify opportunities for new big data projects. You’ll be able to build on the practices and processes you established with your first project. That said, give your team a little time to recover before plunging into the next project.

“One additional benefit of waiting before launching the next effort is that it gives CIOs more time to collect evidence on the performance and benefit of the first project,” Desouza says. “This information will help CIOs make a stronger case for the next project.”