What is the Null Hypothesis? 🧐

In scientific research, the Null Hypothesis (H₀) is the default assumption that there is no relationship between two measured phenomena. It’s the “statement of no effect.” For a new drug to be deemed effective, researchers must first disprove the null hypothesis that the drug has no effect. This principle is akin to the legal concept of “innocent until proven guilty.” You start with the assumption of no guilt (or no effect) and then gather evidence to the contrary.

In cybersecurity, the process of gathering evidence is critically important. The ISC2 emphasizes that without robust digital forensics practices, many cyberattacks would go unnoticed or remain misunderstood. While it is common for cybersecurity professionals to engage in this, too often the diagnosis of a cybersecurity issues begins and ends at the first step; the vulnerability scan.

The Vulnerability Scan: A Flawed Premise?

Vulnerability scanning operates on a principle that is, in many ways, the opposite of the Null Hypothesis. It essentially assumes guilt. A scanner probes a network or application and reports on potential weaknesses. The problem is that these scans often lack context and can be rife with false positives. A scanner might flag a vulnerability that, in the specific context of the system, is not actually exploitable. This creates a “guilty until proven innocent” scenario, where security teams are sent scrambling to patch issues that may not pose a genuine threat.

• Example: a notification of a patching vulnerability appears around a service (SSH), stating a device is not-patched.  It may well have been  successfully patched through an automated update although the specific documented release number (v1.153) wasn’t updated by the vendor. This is a quite common occurrence, and shows how easily scanning alone can fail to provide the full picture.

Similarly, in cybersecurity, the “legit purchased ticket” is a validated, secure configuration. A vulnerability scan, in its typical form, is like that cursory glance at the perimeter. It might easily spot some obvious issues, but it doesn’t come close to validating the security of each individual component from the inside out. And so, it may be useful for an expert to take a quick, cursory glance at the crowd to make some educated assumptions,  but nowhere close to validating the entire audience nor the validity of each presented ticket. In fact, it may even create additional risks to communicate those observations and create a false sense of security. Is this how your business identifies risk? 

What kind of reliability am I getting from this?

You may wonder about this, and you absolutely should. After testing a null hypothesis (H₀), there are basically four possible outcomes based on the reality of the hypothesis and the decision(s) made from the test results.

1. Correctly Accept H₀: The null hypothesis is true, and the statistical test correctly leads to its acceptance (or failure to reject). This is a correct decision.

   • • Example: Open Ports in a scan show that Telnet (TCP Port 23) is open. This is usually reliable, because it should never be open. 

2. Type I Error (False Positive): The null hypothesis is true, but the statistical test leads to its rejection. An example is concluding a treatment has an effect when it does not. (see patching example above.)
3. Type II Error (False Negative): The null hypothesis is false, but the statistical test leads to its acceptance (or failure to reject). An example is concluding a treatment has no effect when it does.

   • • Example: The scan results appears clean, but you are not protecting your business in any useful way (using MFA, EDR, Encryption, etc.), and without evidence to the contrary you conclude there is no vulnerability. This is very common, and leads to complacency and overconfidence. 

4. Correctly Reject H₀: The null hypothesis is false, and the statistical test correctly leads to its rejection. This is also a correct decision.

   • • Example: The scan reports are clean, but fail to identify the absence of a password policy nor use of password management tools, a major weakness that makes your business an easy target.