The value of phishing simulation in a strong security program

August 17, 2022
Justin Hall
Director Security Services

One of the more fiery topics of discussion amongst security practitioners and luminaries in 2022 is the role of phishing simulation and assessment in an enterprise security control strategy.

woman at computer working on phishing simulation
What role does phishing simulation play in your security program?

It has long been gospel that security awareness training is an essential practice for an organization taking security seriously. We need to continually remind our employees about the threats they face, and the responsibilities they carry to protect themselves and their employer from those threats. Training should be:

  • Consistently delivered, in a regular “drip” throughout a year, so that the message stays top-of-mind.
  • Current and relevant, covering recent attack trends in detail (and even using examples of tactics that have been successful against the organization) and focused on the behaviors and actions expected of employees.
  • Nontechnical, delivered “in their language” and in a way that they can understand.
  • Engaging, produced and executed with content that draws in the audience and impacts them.

The last point is particularly relevant in this discussion about phishing simulation.

Why we do phishing simulation

We characterize phishing simulation as the practice of delivering simulated phishing attacks to employees—along with associated training material—in an effort to teach them to recognize and respond to the real thing, but in a safe and educational setting. This practice is the manifestation of the principle of “experiential learning”. Since the 1970s educators have considered this to be a formal field of education, and have explored its value as a part of a larger educational strategy. Our man Confucius said it well: “I hear, I know; I see, I remember; I do, I understand.”

Also, people remember best when they experience something rather than just read about it or watch a video on their computer.

Teaching your employees the “how-not-tos” of phishing. 

Applied to security awareness training, our goal is to have users experience the practice of receiving a phishing email that was unexpected, and then measure their response. Do they report it? Do they poke at it a bit before doing so? Do they fall for the fraudulent claims that come from the sender? Through this effort we determine their susceptibility—or their resilience—to this attack vector.

When our Security Consulting team does phishing simulation for our customers, we carefully craft content in coordination with their security team, identifying scenarios and approaches that are particularly troublesome for their users. We use tools to deliver the e-mail and web content that allows us to measure the responses from the targets: simply opening the message and reading the content; clicking the links or opening the attachments; or submitting data to a form built to steal credentials.

By developing several different campaigns with varied scenarios and content, sent to many different groups of employees, we can start to pinpoint weaknesses in their awareness of threats, and adjust the training to match. We also direct the users who engage with the content to training material on the subject immediately. We find those who have been told “you just failed a phishing test” are paying quite a bit more attention and are more ready learners. When organizations perform these exercises regularly with targeted training in between, we see improvement in the reporting metrics. Users are more likely to report not just the simulated attacks, but actual attacks, as opposed to engaging with them. As an example of the effectiveness, one of our financial services clients saw a 20% drop in “click rates” (users who open a link in a phishing e-mail instead of reporting it) over a three-year period after consistent training.

Read more: Why should you do information security awareness and training?

Criticisms of phishing simulation

Sounds great, right? Not to everyone. There’s been criticism about this practice, and it stems largely from teams who use unsavory content in their simulated campaigns. Think about an e-mail purporting to be from a company that promises to pay off all your student loan debt, or give you free lifesaving drugs if you’re a terminally ill patient. It’s pretty brutal to yell “surprise, we were just kidding, here’s some training!” after sending someone one of those e-mails. So it’s important to be sensitive about the pretext of a message we’re sending to train someone—we don’t want to be hurtful, even if the attackers don’t mind doing so.

Another study has shown that phishing training doesn’t help, that people continue to click on phishing links.

So there are contrary studies regarding the value of phishing training

Hurt feelings aside, we need to face facts: historically, the only way to determine if our security strategy is viable against real attacks is to use real attacks to test it. This is why we do penetration testing! But machines and humans react differently, so we have a thin line to walk: do what the attackers do without causing actual trauma. Some consider the risk of that trauma to be so great that it isn’t worth the potential benefits of training. What if the previous financial customer I mentioned only saw a 5% improvement over the three-year period? Or a 1%? Is that worth the monetary cost of the practice, as well as the frustration of the users who are targeted? These are important questions!

Why phishing simulation puts the odds in your favor

Let’s think about this like we thought about the pandemic. Why wear masks? Not because it completely prevents the spread of a disease, but because it lowers the occurrence of spread. If I have a hundred opportunities to be infected in a day, and wearing a mask means even one of those hundred opportunities is eliminated, that’s an improvement.

We are in the business of reducing risk, and that means any positive change is valuable. The idea that “this security control didn’t eliminate all risk, so it isn’t useful” is nonsense, in my opinion. This same attitude says, because this endpoint protection solution stopped 19 of20 pieces of malware but it allowed one, it is a failure. We know that’s illogical! That’s 19 pieces of malware we didn’t have to worry about—and, a situation where 19 attacks were unsuccessful is obviously better than 20 that were successful.

We cannot eliminate all risk, and those that set such a goal for themselves will always be disappointed and behind. They subscribe to an unrealistic, unattainable view of protecting an organization, and will be unsuccessful every time. Incremental gains in a security program’s effectiveness are not only meaningful, they’re usually the only type of growth we see. Rarely do organizations achieve wholesale, life-altering improvements in a short period of time. That’s the approach of a lazy security practitioner. But if we have 1,000 employees and we turn even one of them from a “clicker” to a “reporter”, that’s growth, and that means potentially dozens or even hundreds of chances to be compromised that are eliminated. In coordination with a larger strategy that includes other training, e-mail security systems, endpoint and network protection, least privilege, and strong authentication, we can start to have a real effect on minimizing the impact of these attacks.

How do we effectively use a phishing simulation?

Now, if you’re simply performing simulations to generate metrics and make your security team look successful, yeah, you’re going to have a bad time.

Simulations are useful as a way to identify weaknesses to which you will apply training. Here’s an example of what our security services team sees as a beneficial training cycle:

  • Acme Co receives a targeted phishing campaign that uses a Microsoft account credential theft attack and a scenario claiming to be a password reset request. A quarter of their employees (100 users) click the link, and 10% (40 users) submit credentials, resulting in a security incident.
  • Acme Co recovers and delivers training to their users, explaining what the attackers did, what they were after, and the recognizable content in the attack that was notable for future detection (an urgent request claiming to be from an authority figure, delivered in an unusual manner: an e-mail message). Users are asked to watch for these telltale signs, and report them in the future, even if they’re unsure if they’re dangerous.
  • Acme Co waits a month and delivers a series of phishing simulations.
    • To those that clicked the link, the same type of message as the actual attack is used.
    • To those that did not click the link, a similar, but slightly more sophisticated message is used, with slicker, more convincing graphics in the e-mail and on the website.
    • To those that reported the message, a simulation with the same attack vector (Microsoft account credential theft) but a different pretext (the employee’s manager is sending the e-mail) and scenario (the employee needs to verify their W-4 is up to date) is delivered.
  • The results of these exercises are collected and analyzed, with the following happening:
    • Employees that still fell for the simulated attack are coached in a 10-minute in-person/virtual training session by a member of the security team along with the employee’s manager.
    • Employees that ignored the message but did not report it are notified and reminded about the reporting process.
    • Employees that reported the simulated message are rewarded with a $5 Starbucks gift card.
    • Broad training content for all employees is updated to mention the telltale signs used in this type of attack and what to watch for.
    • A regular monthly communication to all employees mentions this phishing attack and re-emphasizes the warning signs and reporting process.
  • Acme Co repeats the simulation a few months later, with a slightly modified pretext and scenario and this time asks the user to provide their MFA one-time password along with their credentials. Results are analyzed and used to drive future training as before.

Remember that this is simply one piece of a larger strategy. Yes, it takes people and intentional planning and follow-up. That’s what good security looks like! Humans are harder to secure than machines.

Read up on all the security practices that are essential for protecting your business.


Like it or not, your users will be receiving phishing e-mails. You can’t stop every one of them from entering your inboxes. Either you teach them safely to recognize this content and respond well, or you leave them to their own capabilities and hope for the best. The attackers typically don’t share our qualms about using unsavory tactics. While we don’t want to stoop to their level, we do need to recognize that we’re facing actors that often go to any lengths to trick our users and we need to effectively prepare them for what they’ll face—and if reading about it in a slide deck or e-mail newsletter isn’t helping, we need to consider what will actually move the needle.

Contact us today to learn more about how we can help you build stronger security for your organization.

Subscribe to our blog