Types of indicators
There's a new report that's available from NIST that has some interesting ideas in it. This report is the draft of National Institute of Standards and Technology Interagency Report (NISTIR) 7564, Directions in Security Metrics Research.
The part of this report that I found particularly interesting is this:
Analogous to economic indicators, security metrics may be potentially leading, coincident, or lagging indicators of the actual security state of the system. The distinction is significant. A coincident indicator reflects security conditions happening concurrently, while leading and lagging indicators reflect security conditions that exist respectively before or after a shift in security. If a lagging indicator is treated as a leading or coincident indicator, the consequences due to misinterpretation and reaction can be serious. The longer the latency period is for a lagging indicator, the greater the likelihood for problems. That is, a lagging security metric with a short latency period or lag time is preferred over one with a long latency period, since any needed response to an observed change can take place earlier. It is important to recognize lagging indicators and, if they are used, to be prepared to handle the intrinsic delay and associated limitations.
That's obvious when you hear it, but I hadn't thought of that before.
Leading indicators are ones that tell you what's going to happen in the future. Coincident indicators tell you what's happening right then. Lagging indicators tell you what happened in the past. What you'd like to find is a leading indicator of the security that your systems have. If that indicator starts to drop, you have a chance to address the source of the lower level of security before it becomes a problem. If you have a lagging indicator, by the time that you find out that you once had a lower level of security, you may already have been compromised.
The problem is that it's not clear what a good leading indicator of information security is. If you can find one, you can probably have the basis for a good service or product.