Measuring effectiveness of security solutions to help users make cost benefit decisions

Dr Jeremy Ward,  Effectiveness Special Interest Group

Cyber security suffers from a fundamental lack of clear, objective, universally agreed measurements for the intensity of Internet-mediated threats and the effectiveness of information security systems in reducing these.

In the absence of such measurements, it is difficult to determine the relative effectiveness of information security systems and to make objective decisions about how the threat is changing. Most importantly it reduces the effectiveness of information security management, because, as is well known: "you can't manage what you don't measure."

Most vendors of information security products use an estimate of Internet threat level that is based on the concept of potential harm to systems and the intensity of measures that need to be put in place to reduce that potential.

The obvious draw-back to such systems of measurement is that they are essentially subjective and (intentionally) predictive; yet no regular, objective analysis of the accuracy of their predictions is published by the vendors.

In an attempt to introduce objectivity into their measurement of the Internet threat, a number of organisations use 'honey pots' to attract and trap malicious code attacks. The drawback to such systems is that they are specifically designed to be attractive to attackers and are therefore not necessarily truly representative of 'real' systems.

Also, because their function is primarily to attract attacks, they will be constantly changed and updated - making it difficult to use them to provide comparative measurements over time. Some vendors have also established Internet threat measurement systems that attempt to correlate data from a large number of agents on existing security devices in order to detect developing threats.

Such systems more nearly approach the requirement for objective measurement of threat. However, by their nature they must rely on agents installed on existing security devices placed in existing networks. This makes for a very large number of variables and the greater the number of variables the more difficult it is to generate results that are susceptible to good comparative analysis.

It follows that, since there is no clear, objective methodology for measuring and comparing the size and intensity of the Internet-mediated threat, there is no clear and objective methodology for measuring the effectiveness of the countermeasures against it. This lack of ability to quantify the effectiveness of solutions makes it difficult for users to plan cost-effective security and it makes it hard for suppliers to demonstrate the value of security.

The Cyber Security KTN is addressing this challenge via a Special Interest Group whose objective is to remedy this capability gap by developing prototype metrics and specify a test-bed for validating them. The group is chaired by Jeremy Ward of Symantec.