Identifying the Metrics That Matter (and Who They Matter to)

Effective cybersecurity relies on selecting the right metrics to inform and guide decision-making, but determining the right metrics is not always clear. Metrics that matter are tailored to the needs of all stakeholders, from analysts to board members—and provide insights into actual threats and the effectiveness of security measures. 

By focusing on relevant metrics, organizations can strengthen defense mechanisms and align cybersecurity efforts with business objectives far more effectively. 

What Are “Metrics that Matter?”

Metrics that Matter are the measurements and data points that provide a clear understanding of threat levels, risk assessment, impact analysis, and operational effectiveness in the context of Managed Detection & Response (MDR). Instead of focusing on traditional volume-based metrics exclusively, these metrics delve deeper into how threats evolve within an organization’s ecosystem and the effectiveness of countermeasures. 

Metrics that Matter shift focus from vanity metrics to information that drives security posture and enablement. 

Metrics at Different Levels: From Analysts to the Board of Directors 

In today’s interconnected digital environment, Security Operations Center (SOC) teams, CISOs, and the board of directors all need data to make informed decisions, which is why metrics should be customized for each decision-making level. Ideally, it’s about presenting the right information in a context that is relevant and actionable for everyone. 

Below is a list of security stakeholder roles and what metrics are typically relevant for each role. 

Analysts

• Alert Metrics 
  • Volume of alerts, alerts being worked and their priorities, alert tuning, and spikes in alerts 
• Incident Details
  • Specifics of each security incident. 
• Operational Efficiency
  • Metrics that drive efficiency and responses to threats. 
• Detection and Escalation
  • Mean time to detect and escalate.

SOC Managers

• Trend Analysis 
  • Evaluation of tactical and strategic trends over time. 
• Pattern Identification
  • Recognition of patterns in security alerts. 
• SLA Tracking
  • Adherence to service level agreements, aiming for targets like 99% SLA performance. 
• Utilization Rates
  • Team investigation load and hours consumed by Analysis on Demand (AoD).  

CISOs

• Risk Profile Reduction 
  • Strategies to decrease organizational risk. 
• Effectiveness of Controls 
  • Validity and optimization of security controls. 
• Cyber Kill Chain Analysis
  • Distribution of incidents across the kill chain and identification of gaps. 
• Solution Evaluation
  • Effectiveness of security solutions like email gateways. 
• Policy Informing
  • How alerts shape broader security postures and policies. 

Board of Directors

• Strategic Alignment
  • Connection between security metrics and business objectives. 
• Risk Management
  • Understanding of risk reduction and return on investment. 
• Investment Validation
  • Justification of security-related spending. 
• Maturity Measurement
  • Progress in security maturity. 
• Security Efficacy
  • Effectiveness of the security team in enhancing overall security. 

Setting the Stage for Metrics-Fueled Decision Making  

Metrics are more than historical data points. By measuring and analyzing the right metrics, companies can identify areas of improvement outside of operations, such as architecture and engineering. 

Understanding where threats are stopped within the cyber kill chain is crucial. It enables the understanding of current security posture and how effective it is at thwarting attacks at different stages—from reconnaissance to actions on objectives. Pinpointing the stage where most threats are neutralized can spotlight strong security measures. Identifying stages where threats slip through can reveal weaknesses that require immediate attention. This influences strategic decisions on where to invest in security resources. 

Also, metrics have to be actionable. They need to recount what happened and inform what steps to take next. For example: If an increased number of threats are detected at the ‘delivery’ phase of the kill chain, the metric should spur an investigation into email gateway defenses, prompt questions about effectiveness configuration, and lead to specific actions like updates, patches, or configuration changes to reduce susceptibility to phishing attacks. 

The importance of doing the right things with the right metrics cannot be overstated. A methodical approach to measurement and analysis empowers not just the analysts and SOC directors but scales up to inform strategic decisions at the director or board level.  

Metrics that Matter don’t just tell a story about the past. They help chart a course for a more secure future. 

The Role of the Client in Applying Metrics That Matter 

True engagement means more than just receiving alerts. In reality, metrics are the most useful when clients engage as active partners, which means investigating incidents on both sides and collaborating on outcomes.  

Two-way communication is crucial for precise incident resolution and tailoring defense mechanisms to specific threats. Active client involvement, such as highlighting noteworthy incidents, enriches the process, allowing for detailed examination and enhanced understanding of security events. Proactive participation ensures that metrics reflect and drive the continuous improvement of security postures. 

Tying Numbers to Business Rhythms

Metrics play a critical role in keeping a finger on the pulse of any business’ cybersecurity posture, but what about timing? 

Metrics that Matter aren’t just about producing valuable reports, but should also weave security narrative into the fabric of business operations. Every discussion, number, and review should be aligned. That means tackling the right things during weekly, monthly, or quarterly meetings.  

Here’s what should be reviewed with your MDR partner: 

Weekly/Bi-Weekly Tactical Meetings 

Objective: Deep dive into the operational efficiency of security measures (with an emphasis on actionable insights)  

Weekly or bi-weekly tactical meetings are where the rubber meets the road in service delivery. Here, the focus is on discussing the operational heartbeat of cybersecurity.  

Analysts and technical teams dissect service delivery topics, sifting through the minutiae of day-to-day operations, alerts, and incident responses. These discussions are vital, as they allow for the analysis of alert trends, identification of any deviation from the norm, and exploration of tuning opportunities based on real-time data.  

Monthly Operational Meetings

Objective: Checkpoint for aligning technical operations with business objectives 

Monthly operational meetings elevate the conversation to a metrics and reporting package delivery. This is where analysts and directors come together to review and make sense of the data compiled over the past month. It’s a comprehensive look at alert volumes, priority breakdowns, and alarm tuning—and ensuring that every stakeholder understands the impact of cybersecurity measures on the broader business landscape. 

By examining trends in investigations and escalations, the team can pinpoint deviations and adjust strategies accordingly. Metrics like SLA performance, mean time to detect, and mean time to escalate become the lenses through which service delivery is scrutinized.  

The Quarterly Business Review

Objective: Strategic forum for providing a panoramic view of cybersecurity posture and connecting granular data to business goals 

QBRs should be a holistic analysis of how the service contributed to success over the past quarter. This is where strategic overviews paint a picture of performance, enabling executive sponsors and senior leaders to understand their security stance in the context of business success. It’s about ensuring that the cybersecurity strategy is not only responsive but proactive in securing the environment and success. 

Real-World Example: Binary Defense Customer

Let’s zoom in on a real-world Metrics that Matter example focusing on a Binary Defense customer, a power supply company. 

In a collaboration with this customer, Metrics that Matter were used to affirm security investments and drive security practices forward. This approach surpassed basic SLA fulfillment and reduced volume alerts, aiming instead to evolve the company’s security defenses.

For example: During a Priority 1 event, Binary Defense analysis identified a command-and-control incident through Microsoft Sentinel. This was linked to the Cyber Kill Chain framework, revealing the network beacon’s effectiveness. When signs of a Cobalt Strike Beacon (indicative of a severe cyber threat) were discovered, the situation was quickly contained and the client was advised on improving security hygiene to bolster future defense. 

Metrics that Matter proved invaluable again when intercepting a phishing attack at the delivery stage, identified through Sentinel. The immediate response, which was deleting suspicious emails and blocking malicious domains, went beyond addressing the immediate issue. It also set a new benchmark for the company’s email gateway security strength. Proactive measures resolved the threat and demonstrated the power of timely, precise metrics in evaluating and improving security controls. 

Evaluating an MDR Provider Through Metrics 

Evaluating an MDR vendor’s metrics package is a critical step in understanding how services will align with and enhance  security posture.  

It is important to ask questions that go beyond surface-level data. Specifically, ask about the metrics that illuminate overall environment’s health, especially the accuracy of threat detection. This metric reveals the number of alerts received, how many were escalated, and which were true positives, which is a direct reflection of the SOC’s detection quality and efficacy. This will prove more about the MDR vendor’s operational effectiveness than volume or time-based metrics. 

Service Level Agreements (SLAs) and response times are important, but they shouldn’t be the only benchmarks for performance. There’s a danger in relying solely on these figures because they can be superficial, and not fully indicative of the value an MDR provider brings to the table.  

A key consideration is how the MDR provider’s metrics can drive proactive security measures and strategies. Metrics should be about preventing incidents and strengthening the security fabric just as much as they’re about response. The Metrics that Matter are those that highlight environmental health and the Security Operations Center’s (SOC) performance in a meaningful way. 

Lastly, the success of these metrics is not just in their reporting, but the actions they prompt and the collaborative relationship they foster between the client and the provider. An MDR provider should serve as an extension of the client’s security team, helping to enhance security measures. Aim for a partnership that’s characterized by regular feedback, engagement, and mutual efforts to improve.