7 ways to avoid alert fatigue

security alerts
Thinkstock

So much noise

As a company grows, more tools are required, and with more tools come more alerts and often a breakdown of processes and procedures to handle them. Soon enough, the alerts coming from each of your systems and tools sound like an obnoxiously loud cocktail party, everyone having different conversations about different things. As a result, Security and DevOps teams become so desensitized to these alerts that even when the system flags a truly anomalous activity, it may get ignored due to burnout.

What you want is for alerts to sound off like a harmonious choir, all working together and only hitting the high notes when a real issue arises. So how do we get there? Rather than sit by the sidelines waiting for the next team member to hit this negative inflection point, Chris Gervais, vice president of engineering at Threat Stack, and VictorOps, offers seven ways teams can avoid alert fatigue.

threat alerts
Thinkstock

Make all alerts contextual and actionable

It’s a tiring workday sifting through alerts that have no meaning and no context from which to determine a course of action. Alerts need two key things in order to be effective:

  • Context that comes from pairing data points from across the system to paint a complete picture, including runbooks, graphs, logs, notes, and any other details relevant to resolving the issue.
  • Source details that indicate exactly where the issue originated and any other areas of your system that were impacted, so you can fix the problem at the root.
threat alerts
Thinkstock

Reduce redundant alerts

Plain and simple, it’s inefficient to be paged on the same issue over and over — especially if it’s a non-issue. This is one of the biggest factors leading to alert fatigue. It doesn’t matter whether it’s an alert triggered by regular engineering work, or a third-party app setting off an unnecessary alert: these instances can all lead to alert fatigue. Reducing and consolidating alerts can be done by either fine-tuning the alerting protocol for each tool, or even better, by combining all security functions into a single platform to unify alert configurations and origination.

threat alerts
Thinkstock

Designate alerts to a single source or timeline

With each tool sending off its own alerts (most often directly into your email inbox), it becomes difficult to connect the dots and uncover real issues — that is, if you even pay attention to these alerts among the clutter of your email. You should never rely on email alerts as your single source of truth. It’s far better to use an open communication channel like Slack to stream alerts, provide team-wide visibility, and allow for open discussions to resolve issues.

Streamlining security functions (threat intelligence, vulnerability management, CloudTrail, etc.) into a single place can also go a long way in unifying security alerting.

threat alerts
Thinkstock

Adjust anomaly detection thresholds

Caught up in the day-to-day hustle, many teams forget to fine tune baselines on a regular basis. This results in more alerts about nothing, further adding to fatigue. A good place to start is by addressing your noisiest alerts, but an even better solution is using a tool that can learn from your system’s baselines over time, adjusting as you scale so you don’t have to do this manually.

threat alerts
Thinkstock

Ensure that correct individuals/teams are alerted

Another problem that crops up as teams grow is ensuring that everyone on the team has the right access to the right alerts in order to take action on them. As part of your continuous improvement processes, allow each team member to decide how, how often, and on what topics they should be alerted.

threat alerts
Thinkstock

Customize personal notifications/page

There are countless stories of engineers and ops folks being woken up during the night by non-severe alerts. Not only will your team be sleeping less, they may even stop trusting the daytime alerts! Instead, ensure that only high-severity alerts trigger a “wake me up in the middle of the night” scenario. All others can wait until the morning.

threat alerts
Thinkstock

Revisit and adjust regularly

The preceding six recommendations shouldn’t be a one-time effort; you need to revisit them regularly to ensure the system is working as it should be. Below are several questions you should post to your team during postmortem exercises and regular team-wide meetings:

  • Is alert “Signal:Noise” tuning owned by the entire team?
  • Is alert tuning part of your continuous improvement processes?
  • Are your teams empowered to prioritize work and address the factors that contribute to alert fatigue?
  • Are the escalation processes sane and effective?
  • Can more data be integrated into alerts to provide the proper context to make decisions?