In Search of Evidence-Based IT-Security

From Next-Generation APT-Defense to Machine Learning and Artificial Intelligence: The promises of IT security product vendors are often bold. Some marketing promises are simply impossible, because they violate a fundamental theorem of computer science, the halting problem. Many IT security professionals are skeptical of security appliances, antivirus software and other IT security products and call them snake oil. Furthermore security products often have security vulnerabilities themselves, which has lately been shown by the impressive work done by Tavis Ormandy from Google's Project Zero. When there's disagreement about the effectiveness of an approach then rational people should ask for scientific evidence. However, surprisingly this evidence largely doesn't exist. While there obviously is a lot of scientific research in IT security it rarely tries to answer practical questions most relevant to users. Decisions are made in an ad-hoc way and are usually based on opinions rather than rigorous scientific evidence. It is quite ironic that given the medical analogies this field likes to use (viruses, infections etc.), nobody is looking how medicine solves these problems. The gold standard of scientific evidence in medicine (and many other fields) is to do randomized controlled trials (RCTs) and meta-analyses of those trials. An RCT divides patients in groups and a treatment – for example a new drug – is compared against a placebo treatment or against the current best practice. Single trials are usually not considered sufficient, therefore meta-analyses pool together the results of all trials done on a particular question. There's no reason RCTs couldn't be applied to the question whether a particular security product works. Evidence-based medicine is undoubtedly the right approach, but these methods aren't without problems. Publication Bias skews results, many studies cannot be replicated and the scientific publishing and career system is often supporting poor scientific practices. But this doesn't question the scientific approach itself, it just means that more rigorous scientific practices need to be implemented. Unfortunately, in the few cases where controlled studies are done in the Infosec world they often suffer from the most basic methodological problems like being underpowered (too few participants), never being independently replicated or not measuring relevant outcomes. (There are a few studies on password security and similar questions.) Applying rigorous science to IT security could provide a way out of the security nihilism that dominates the debate so often these days - “Everything is broken, everyone's going to get hacked eventually”. And by learning from other fields Evidence-Based IT Security could skip the flaws that rife other fields of science.