Security Pie

When I’m reading an article or an analysis about the effectiveness of a specific security tool I often get upset. The main reason is that security researchers (at least the one that I’m reading) are taking an “all or nothing” approach to security products analysis – it either works or not. Even when talking about defense in depth in conjunctions with Information/data/network security some people have a tendency to write about this “all or nothing approach”.

When I read Rich’s post on Web Application Firewalls I thought that it would be very useful if anyone could come with detailed analysis on the effectiveness of different products. Similar to the way that the National Highway Traffic Safety Administration (NHTSA) is measuring the effectiveness of seat belts. Even when using amperic data, researches will argue about the effectiveness of data:

A recently publicized claim by one analyst that seat belts reduce vehicle occupant deaths 70-80 per cent is based on studies found to contain fundamental systematic error. Deaths occur only 50 per cent less often to belted compared to nonbelted vehicle occupants in crashes, according to previously unanalyzed data from three U.S. states during recent years.

I’d like to see more information providing statistical analysis like used in this US Roads 1997 article, showing the effectiveness of security solutions under different circumstances:

• Mode of deployment
• Attack vector
• Policy used
• Other combined security tools/methods used

This will allow to measure the effectiveness of security tools and provide proper analysis that will allow an organization to perform proper risk analysis.