Big Data Problem Plagues Government Agencies
Government agencies are collecting vast amounts of data, but they're struggling just to store it, let alone analyze it to improve efficiency, accuracy and forecasts.
Mon, June 11, 2012
CIO — Big Data has the potential to transform the work of government agencies, unlocking advancements in efficiency, the speed and accuracy of decisions and the capability to forecast. But despite the potential benefits, most federal government agencies are struggling to leverage Big Data.
These agencies lack the data storage/access, computational power and personnel they need to make use of Big Data, according to a recent report, "The Big Data Gap," by MeriTalk. MeriTalk is a community network for government IT developed as a partnership by the Federal Business Council, Federal Employee Defense Services, Federal Managers Association, GovLoop, National Treasury Employees Union, USO and WTOP/WFED radio.
"Government has a gold mine of data at its fingertips," says Mark Weber, president of U.S. Public Sector for NetApp, underwriter of MeriTalk's report. "The key is turning that data into high-quality information that can increase efficiencies and inform decisions. Agencies need to look at Big Data solutions that can help them efficiently process, analyze, manage and access data, enabling them to more effectively execute their missions."
The government is collecting data: 87 percent of government IT professionals say their stored data grew in the last two years and 96 percent expect their data to grow in the next two years (by an average of 64 percent). Unstructured data makes up 31 percent of data held by the government, and the percentage is on the rise.
On average, government agencies store 1.61 petabytes of data, but expect to be storing 2.63 petabytes within the next two years. These data include: reports from other government agencies at various levels, reports generated by field staff, transactional business data, scientific research, imagery/video, Web interaction data and reports filed by non-government agencies.
The majority of IT professionals (64 percent) say their agency's data management system could be expanded or upgraded to cope with this growth, but they estimate it will take an average of 10 months to double their short- to medium-term capacity.
While government agencies collect massive amounts of data, MeriTalk's report found that only 60 percent of IT professionals say their agency analyzes the data collected, and less than 40 percent say their agencies use the data to make strategic decisions. That includes U.S. Department of Defense and intelligence agencies, which on average are even farther behind than civilian agencies when it comes to Big Data. While 60 percent of civilian agencies are exploring how Big Data could be brought to bear on their work, only 42 percent of DoD/intel agencies are doing the same.