Federal Government's Big Data Efforts Lagging
Facing contracting budgets and soaring data volumes and varieties, the feds should take a cue from the private sector and embrace big data technologies to better serve constituents.
Thu, November 29, 2012
CIO — WASHINGTON -- As agencies in the federal government consider new IT initiatives to handle the challenges presented by soaring volumes of structured and unstructured data, they are playing catch-up with their industry counterparts. But a big data strategy is essential to ensure that federal IT departments glean valuable information from far-flung sources and, most importantly, effectively execute their mission to serve constituents, according to Sameer Kalbag, CTO at the federal division of HP's Autonomy.
In remarks to federal workers and contractors at a government IT conference, Kalbag described the twin pressures that government agencies are facing from contracting budgets and the surge in data--not just in volume, but from an influx of content in new forms and from new sources, such as social media, sensor technology and audio and video, amounting to what he called an "untenable situation."
"It's not a surprise to anyone that data is growing by exponential rates," Kalbag says. "What's different today is really the variation and the types of data out there."
Data Is Not Just BiggerPart of that shift stems from what might be described as the democratization of content production. Kalbag points out that not so many years ago, even when businesses were dealing with enormous data sets, that information came from relatively few sources.
Fast-forward to a time when blogging already seems a quaint technology and the platforms for expression online are innumerable, and the challenges of gleaning useful insights from vast stores of disparate data are far greater than in the early days of the Web.
"The way content is being created is fundamentally different," Kalbag says. "Extracting value out of a video is a lot different than looking at rows and columns in a database."
And just by the numbers, federal IT shops have their hands full. Earlier this year, MeriTalk, a research group and networking community for government IT workers, surveyed departments and agencies and found that 87 percent of federal IT workers reported that the volume of data they deal with had increased over the past two years. Then 96 percent of respondents said they expect their data volumes to rise over the next two years, by an average of 64 percent. Nearly one-third of federal data was described as unstructured.
The survey also reveals that federal IT workers are still grappling with their approach to big data, with nine in 10 reporting challenges with their implementations. On average, survey respondents say that they don't expect to take full advantage of big data solutions for another three years.