NPR Deploys Splunk for Web Analytics
Using Splunk, NPR has found a faster way to build up customer information
Thu, March 24, 2011
IDG News Service — In a novel use of the software, National Public Radio is using the Splunk log search engine to analyze Web traffic for its audio streams and downloads.
NPR metrics analyst Sondra Russell described the setup at GigaOm's Big Data conference, held Wednesday in New York.
Splunk offers what it calls a search engine for machine data. It was originally built to parse log files, or the files programs and hardware generate to document their transactions, errors and other operational information. By coordinating the timestamps of messages from different applications and hardware, Splunk allows system administrators to pinpoint difficult-to-locate system problems.
In recent years, however, customers have been expanding their uses of Splunk to other duties, explained Splunk Chief Technology Officer Erik Swan, also speaking at the event. Web traffic analysis and business intelligence are two such ancillary uses.
For much of its Web traffic monitoring, NPR uses standard Web traffic analytic software, which can deliver reports on how many people visit each Web page. Such software usually generates these counts by using cookies or by embedding each page with a small script that alerts the software when the page is rendered in a browser.
The media organization, however, found it difficult to get reliable usage summaries for a number of aspects of its service. For instance, the organization needed to get an accurate count of how many listeners tuned into their streamed audio and video programs.
To get this data, NPR had prepared a PHP script that would parse the server log files and translate the results into a form that could be digested by Adobe's (ADBE) Omniture, a Web analytic tool. Getting information back, however, could take up to 24 hours, and Russell still didn't trust that the results provided an accurate count.
In the cases of streaming usage, many users might start a stream, then pause it, and restart it. Or perhaps a user would restart a stream after a failed Internet connection. In the server log files, all these events were logged as separate events, not a linear sequence of actions by a single user. As a result, there was no way of determining how many connections were from different listeners, and how many were multiple streams to a single user.
"With our PHP scripts, we could not get that level of sophistication. So our numbers could be off," Russell said."It was hard to make rational decisions based on this."
By working with Splunk, NPR could derive listener numbers and information directly from its servers' log files. The software allows users to script search results and then graph the results, or show them on a dashboard.