Using Metrics to Improve Your Process

Metrics are mere curiosities unless they tell a story that supports or encourages change or improvement.

When you’re looking at producing metrics for your security program, you’ll almost always be trying to measure whether you’re doing something better or worse. However, to quantify “better” or “worse,” you need to determine what’s quantifiable. And in order to figure out what is quantifiable, you need to understand your process.

In fact, one of the most valuable parts of setting up a metrics program is that it forces you to do some process analysis, and analyze what it is that you do in order to measure it. You can then focus on how to improve it.

Analyze Your Process

There are plenty of formal ways of analyzing your process, but – unless it’s insanely complicated – you can typically do an effective analysis by looking at what you do from the top down, breaking it into chunks that represent major steps. You’ll build something like a flow chart, except instead of “if” and “else” logic, you’ll capture the decision inputs that go into the transitions, the resources required, and the clock-time typically spent.

Scope your process into steps, and then review what you’ve captured to see if there are points where your process enters into a wait-state. Your analysis may capture cases where your operation seems fairly efficient but gets deadlocked by a broken process in your department or in another business unit.

Identify Areas of Improvement

Once you’ve done some basic process analysis, you may conclude you’re already perfect. More likely, however, you’ll make some recommendations for improvement.

Also, once you’ve broken your process down, you can project whether there will be a major or minor impact to changing one part of the process or another.

For example, you might realize that your process won’t improve if you install a trouble ticketing system, but might improve dramatically if you can boost the search speed against your system logs. Here, you see that we are quickly into the land of “it depends,” which is exactly what your metrics program is going to help you understand better and clear up!

It also might turn out that your process runs fairly well until it bumps up against another business unit’s processes – where it grinds to a halt for a couple days – before jerking back into smooth motion. If this occurs, congratulations! You’ve just measured inefficiency in how your business unit works with another, and you now know what to focus in on.

Use Metrics to Help Tell Your Story

Remember: a metric is a bunch of data and a means of reducing it to tell a story. When designing your metrics, always ask “what story am I trying to tell?” Then go into it honestly with a spirit of discovery. Make your metrics program part of your learning process, and you’ll be able to explain it to others when you need to. Read more from Marcus Ranum here.

Marcus J. Ranum works for Tenable Security, Inc. and is a world-renowned expert on security system design and implementation. He has been involved in every level of the security industry from product coder to CEO of a successful start-up. He is an ISSA fellow and holds achievement and service awards from several industry groups.