Measure for Measure

by Hank Berkley
Contributing Editor

Measuring success in cyber security is difficult. In the simplest case it can be seen as a binary value – you had a security issue or you did not, but that is not particularly useful and certainly won’t help you to improve your overall security posture. In a world in which metrics drive just about everything, why are security metrics so difficult to identify?

The values in cyber security that are most meaningful cannot be measured because they are unknown. We have seen companies report on how many SPAM emails are detected or how many viruses are identified. Those numbers tell you that your systems are doing something and might tell you that you are under attack when the numbers increase substantially. They don’t say anything about the effectiveness of your systems. Numbers are only meaningful when they are put into a context and examined relative to other numbers.

We can learn a bit more if we know how many SPAM emails are detected relative to the total number of emails received. Still better is to view this ratio over time. But we don’t really know how well we are doing unless we put a number on the volume of SPAM items that get through our filters. Or we measure how many items we blocked that should not have been blocked. Those numbers are difficult to derive.

Attaining meaningful cyber statistics has the added challenge that there is a limited amount of public information with which to compare numbers. Cyber spending and results are often treated as confidential information in businesses. When we examine financial results, we put them into context by comparing them to similar organizations. What is their inventory turn rate? What is the stock’s PE? But it is rare to know how much your competitor is spending on security or how effective particular actions are. Are we seeing more DDoS attacks or receiving more email per employee than other firms?

Perhaps it is because there is a dearth of information that there are no standard measures in information security. We have seen numbers such as the percentage of total IT spending or the amount spent per employee, but these have been based on limited sample sizes and there is no science to indicate that they are anything more than measurements without meaning. Is spending 6% of your IT budget on security the right amount? No one seems to know.

While the stigma of having suffered a security breach seems to be lessening and companies are often required to publicly disclose their losses, few companies provide details about how much they spend or how effective that spending is. Without that information it is impossible to measure our own results and to optimize our own results.

Drucker’s “you can’t manage what you can’t measure” is every bit as applicable to cyber security as it is to a shop floor. But just measuring changes in your own situation can only tell you if you are getting better or worse. It doesn’t provide a real scorecard unless you have those numbers from others. For this you need to establish an open relationship with others in the cyber security field. Even if you compete in the rest of your business.

While they are not focused exclusively on metrics, there are numerous Information Sharing and Analysis Centers (ISACs) that are worth investigating for just this purpose. Some are specific to an industry group such as the FS-ISAC for financial services or the AUTO-ISAC for the automotive industry while others such as the IT-ISAC have a broader membership. The groups have both local and (inter)national gatherings that are informative and unbiased. They don’t have all the answers but becoming an active member can be a starting place to obtain meaningful cyber metrics.

If you are unaware of a group in your industry you can start with the National Council of ISACs. They provide a great service that is unavailable through commercial sources.