Anomaly Detection Lets You Find Patterns in Log Data
Generating insight from log data traditionally requires writing a search. But that means you need to know which questions to ask. How do you get insight from data you know nothing about? Sumo Logic's answer combines machine learning and pattern recognition to detect anomalous events in your data.
Tue, September 10, 2013
CIO — Organizations typically generate tremendous volumes of data from their infrastructure on a regular basis, much of it machine data in the form of logs. Turning those logs into insight is a difficult challenge and represents one of the more intriguing promises of big data analytics.
Traditional security and log-management tools attempt to provide insight into the chaos, but they typically require users to write rules to detect anomalies. Writing those rules requires pre-existing understanding of the data—you need to know what you're looking for before you can perform a search to find it.
It's Humanly Impossible to Know Everything About Your Data
"The first challenge is not just that there's a vast amount of data, but the fact that typical analysis of machine data typically relies on search as the fundamental mechanism to investigate what's going on," says Sanjay Sarathy, chief marketing officer (CMO) of machine data analytics specialist Sumo Logic.
"The challenge with search is that you fundamentally need to know what you're searching on. Given the explosion of data, it's humanly impossible to know everything about your data," says Sarathy.
"CIOs don't care about the logs. CIOs care about the events those logs represent," he adds. "They care about anomalies. The traditional way of getting to those anomalies and events is writing rules. But the challenge you have is actually to write those rules. Given the amount of data, it's impossible to write rules for every event."
Anomaly Detection Uses Machine Learning, Statistical Analysis to Detect Events
Sumo Logic's answer is Anomaly Detection, a major architectural enhancement to its Log Management and Analytics service based on its LogReduce technology.
Anomaly Detection combines machine learning, statistical analysis and human knowledge from your domain experts to analyze streams of machine data, detect events in the stream and provide alerts on those events, allowing you to remediate issues before they affect business services.
"Basically, we reduce log lines into a set of patterns," Sarathy says. "That allows us to figure out the root cause of issues. We don't need to know anything about that data in advance to be able to come up with any of those patterns. You, as the domain expert, help us understand which patterns are relevant and which aren't. We're building on that pattern recognition technology to provide an automated way to do anomaly detection."