I’m writing this blog post because it needs to be written. It seems pretty obvious that you should test patches to your OS or applications in a test environment that closely matches your production environment before you deploy that patch in production. However, just because something seems obvious doesn’t mean that it is.
So that folks reading this can avoid the hours of recovery that I’ve had to endure back in the day—before I knew better—let’s go over some things that can go wrong with a patch, either to the OS or an application.
Just because the vendor issues a patch, do you have to apply it?
No, you don’t have to apply it, but you need to consider the risk of not applying the patch before you simply ignore it.
When a vendor issues a patch to an operating system (think Windows Patch Tuesday or Linux updates or Apple Mac updates), they do so for a reason. Almost always, the reason the vendor issues the patch is because a vulnerability has been detected. The patch ostensibly removes the vulnerability from the OS. Yay, risk averted!
Read more: our team looks at a recent incident in I just met a vuln named Follina
You’re a busy IT person, you don’t have time to reboot production servers after applying patches to the OS. What is the worst that could happen?
Well, let’s assume that the patch from the vendor fixes a vulnerability that allows an attacker to gain full control of your production server.
That’s bad and you want to mitigate or remove that risk as quickly as possible.
Of course, there are often mitigation techniques or compensating controls that can be used to lessen the risk of an identified vulnerability. Often the compensating controls involve removing a vulnerable service, or implementing firewall rules to block specially crafted packets, or adding Access Control Lists (ACLs) around a vulnerable device. The compensating control(s) can limit or mitigate the vulnerability, but they often come with side effects, or additional restrictions that limit the functionality of the server or device.
Consequently, establishing the habit of regularly applying patches to your production environment is good InfoSec hygiene.
Make sure you have a test environment where you can apply the patches. This environment should match your current production environment. The same operating system on your production servers should be on your test servers. You don’t have to have ten of the same kind of server in test, but if you have a production file server that is Windows Server 2016, that’s the kind of server you should have in test. If you have a mixed environment in production, like a 2016 server and 2019 server, duplicate that in test. If you have a Mac running some applications and a Windows server running another application, duplicate that configuration in test. If you have 100 virtual servers in production, all running the same OS, that’s fine; have one test server that mirrors the 100 virtual servers in production.
OK, you’ve got your test environment setup, you apply the first patch, and…. nothing!
The server comes back up cleanly, meaning that it’s up and running and you didn’t detect any errors. Great! You’re done, you can quickly apply the patches to production.
Have you done your testing now that the server is back up? Before you call it a day, ask and answer these questions:
Ok, let’s assume the answers to the questions above are satisfactory. Guess what? You should run the server for 24 or 48 hours in your test environment and check questions number 1, 2, 3, and 4 again just to be sure everything is working as expected. (Can you tell I used to run a production environment?)
If, after a quick but decent enough burn-in time—which is usually short for critical patches—you can now schedule the patches that have passed these questions for deployment to production.
When you apply the patch and confirm that the vulnerability has been removed, the risk regarding that vulnerability has been addressed.
As you can see, this is why we recommend testing patches in a test environment before rolling out to production. Hopefully this short blog post will help you as you build up your patch management process and plans.
If you have any questions please feel free to reach out to the CBTS Security Team!
Read more from John Bruggeman:
Cloud security controls that help mitigate risk
Cyber Insurance, part 1: What is Cyber Insurance and do I need it?
Cyber Insurance, part 2: Getting ready for the insurance company questionnaire
Cyber Insurance, part 3: Filling out the questionnaire
Cyber Insurance, part 4: What do you do if your cybersecurity insurance policy is denied?