After spending years in cybersecurity education, I’ve seen the same claims about phishing awareness training repeated like gospel: “Users are the Human Firewall,” “Training reduces risk,” “Engaged employees are your first line of defense,” “Interactive learning drives change…” but does it?
We just finished the largest known enterprise-scale field trial assessing phishing training effectiveness using the NIST Phish Scale, and the results were sobering. Across 12,511 employees at a fintech firm, we tested passive and interactive training methods, followed by controlled phishing simulations.
The results? Training didn’t matter.
Our findings align with growing evidence questioning training effectiveness. Ho et al.’s study (2025) found similar results—training had minimal impact on phishing susceptibility. Likewise, Jampen et al.’s 2020 meta-analysis concluded that while some short-term improvements are possible, the evidence for lasting behavioral change remains weak.
What We Found: Training vs. Phish Difficulty
Let’s get to the point:
- Employees who received no training were about as likely to click or report phishing emails as those who completed vendor-provided modules.
- The difficulty of the phishing email—as defined by the NIST Phish Scale—was the only meaningful predictor of success or failure.
- “Easy” phish had a 7% click rate.
- “Hard” ones jumped to 15%.
- Neither lecture-based training nor interactive quizzes produced statistically significant improvements in detection or reporting.
- Effect sizes for training were below 0.01—too small to justify the enormous cost and compliance burden.
This reinforces a now-familiar refrain: people don’t need more training—they need better shields.
Multiple studies across different sectors have reached similar conclusions.
The Regulatory Elephant in the Room
Despite the mounting evidence, training remains mandated by HIPAA, PCI DSS, ISO 27001, and others. This creates a compliance paradox: organizations continue to invest heavily in programs that demonstrably don’t move the needle.
While regulators may have good intentions, it’s time to ask hard questions:
- Should we redefine compliance to prioritize effectiveness over box-checking?
- Can we incentivize measurable outcomes rather than relying on outdated training requirements?
When Training Hurts More Than Helps
One of the more unsettling findings: training might actually increase risk in some cases. We saw higher click rates on “easy” phishing emails from trained users, suggesting that training may:
- Shift attention toward more sophisticated cues
- Desensitize users to “obvious” threats
- Create a false sense of confidence
This phenomenon has been observed elsewhere. Caputo et al.’s 2014 study tested embedded training delivered immediately after users clicked phishing links. Despite receiving contextual education about the specific phishing indicators they missed, participants were not significantly less likely to click in subsequent trials—even with just-in-time training at the moment of peak learning opportunity.
In short: some users overfit to training examples and miss the forest for the trees, while others simply don’t benefit from training regardless of timing or context.
A Word About the Coming Storm: LLMs and AI
What happens when phishing emails are no longer riddled with typos and poor grammar? What happens when attackers can:
- Clone organizational voices,
- Autogenerate context-rich lures,
- And tailor attacks to personal social graphs—at scale?
This is not speculation—it’s happening now. Tools like ChatGPT, WormGPT, and open-source LLMs are already being weaponized.
Training won’t stop that. Instead, we should be investing in:
Cryptographic sender verification
Deploying SPF, DKIM, and DMARC ensures emails are authenticated at the domain level before reaching inboxes. SPF verifies authorized IPs, DKIM uses cryptographic signatures to confirm message integrity, and DMARC ties SPF/DKIM together—allowing domain owners to specify handling of failed authentication while generating feedback reports. Together, these protocols create a powerful defense against spoofing and phishing .
Secure PROCESSES for critical functions
High-risk actions—such as approving invoices, changing passwords, or resetting accounts—shouldn’t rely on a single email click. Instead, implement workflows with out-of-band confirmations (e.g. SMS or voice), multi-person approvals, and temporary tokens tied to user/device-specific factors. Embedding such verification steps dramatically reduces the risk of phishing-based breaches.
Passwordless authentication
Eliminating passwords removes credentials that are often targeted by phishing. Modern solutions leverage FIDO2/WebAuthn, enabling authentication via hardware tokens or platform-based biometric systems. The private keys stay on the user’s device, preventing attackers from capturing shared secrets—making these methods inherently phishing-resistant. Major standards bodies and platforms (FIDO Alliance, W3C, Microsoft, etc.) recommend and support this approach.
Zero‑trust network designs
With zero‑trust, no device or user—inside or outside the perimeter—is automatically trusted. Every access request is re‑authenticated, authorized based on least-privilege principles, and continuously validated. This limits the impact of stolen credentials or successful phishing clicks, preventing lateral movement within the network.
User behavior anomaly detection
Automated monitoring solutions—such as Network Access Control (NAC), User and Entity Behavior Analytics (UEBA), and AI‑driven systems—track deviations from established user patterns. Sudden spikes in file access, off-hours logins, or anomalous data transfers trigger alerts or automatic contingencies. These tools detect suspicious activity even if a user unknowingly engages with phishing content.
From Punishment to Pragmatism
One troubling trend in many organizations is the use of punitive responses to failed phishing simulations—mandatory punitive training, disciplinary actions, even public shaming. Our findings suggest this is not only unfair but counterproductive.
Users aren’t the weak link; they’re the last line of defense in a chain that should have blocked the phish long before it reached their inbox. Blaming them for failures in that system does nothing to strengthen it.
Where Do We Go From Here?
Let’s be clear: this study doesn’t mean we should abandon training entirely. But it does mean we should:
- Reframe awareness efforts as culture-building, not threat mitigation.
- Use the NIST Phish Scale to design better benchmarking for simulated campaigns.
- Focus compliance reporting on technical controls and layered defense rather than superficial training metrics.
- Reserve training for high-risk roles or post-incident remediation—not blanket deployment.
Final Thought
It’s time we stop asking “How do we make humans better at catching phish?” and start asking: “Why are we still relying on them to do so much catching in the first place?”
An earlier version of this post, including the poster session and an early draft of the paper, was published on my blog at:
https://roze.ma/2025/03/07/phish-scale/
Leave a Reply