In my previous blog post I talked about the value of centralized logging, a high level, non-complex overview of how centralizing your logs can help you determine if your controls/defense tools are working.
Now I will go a bit deeper with some best practices regarding centralized logging and what other logs you can put in your centralized log server. Before I do, imagine this scenario:
What is going on?
You haven’t implemented centralized logging yet so you call the Operations Team (Ops) and notify them that something is going wrong in New York. You wait for Ops to get back to you. Thirty minutes pass, then you get a text back:
Maybe some threat actor is working on a ransomware attack. Maybe someone has broken into the office in New York and is doing a denial of service attack. Maybe that new customer that asked for a demo on the Friday before Labor Day put a Raspberry Pi on the network in the conference room and is scanning the company network.
So, what is going on?
For you or your team to be able to answer this question quickly you need to know what is happening on your network. As you read this post you might start to think, “I can’t afford this John!” and you’re probably right. Information Security likely will not have the budget for centralized logging just for the sake of information security. But, once you have the logs in a central location they can be used for other business purposes. This is not a simple project, but then going to the moon wasn’t simple; nevertheless, it was accomplished. You need a good team and you need to stay focused to reap the big rewards. How, though, do you reap the big rewards?
First, you want to follow best practices, namely, plan ahead and think it through. Planning and thinking through this kind of project will pay off on several fronts, and not just for information security.
Here are some of the things to consider when you say to yourself, “I want centralized logging to improve my information security program.”
Read more: Software bill of materials (SBOMs): what is it good for?
Step 1. Create a plan and have a strategy for this project. Do NOT just buy the first centralized log tool you find. Plan for what you want to collect. As part of this planning process you’ll ask questions of your network team and others like:
Step 2. Make sure the structure of the logs you are collecting is consistent.
You won’t be able to ingest logs from multiple data sources unless there is a consistent log format. Your network infrastructure devices will have a format—most likely syslog format—and your firewall(s) will likely have a similar format, and then things can get proprietary (ugly, in other words). Remember, you are not just dumping data into an SQL server and then magically extracting useful information and meaningful insight into your network.
Step 3. A brief word about time and relativity and NTP.
This might be obvious, but to be clear, you need to make sure the logs all have the same time. All network devices and computer systems have a clock, so you will get the date and time for events that you are logging. You want to use Network Time Protocol (NTP) to sync all the systems to the same time source or you’ll have problems. Einstein proved that time is relative; for purposes of logging events in a central location for troubleshooting, you need the clocks on your devices set to the same time and time zone. If you have a switch (or two) that think it’s 1990, but you know it’s 2021, you are going to have a real tough time figuring out what happened that Saturday night of Labor Day weekend (note this is itself relative to Labor Day weekend in the U.S. and Canada because Labor Day is different in Australia, Japan, New Zealand, etc.). Threat actors have calendars and know when people are likely to be away from their computers and monitoring systems, so plan accordingly.
Step 4. Make sure each data source has unique identifiers.
If you are searching through log data looking to see what happened Saturday night at 11:00 p.m. Eastern Time, make sure you know that the switch in the server room is uniquely identified compared to the switch in the conference room. Here is an example of a switch log record; note the various fields and values that you want to be able search and index.
You can see lots of good information in that record, but what switch did it come from? You need to be able to answer that question or all your time and effort has been for naught.
Step 5. Keep your production logs and centralized logs separate.
This is probably obvious but I need to state it plainly: The centralized log server does not replace your SQL logs (or Oracle logs or other production logs). When you need to roll back transactions in SQL or Oracle, etc., you are going to use those production logs. The value of the centralized log tool is gathering other insights. I’m thinking security insights (telemetry, correlation, etc.), but it could be troubleshooting a cranky application, dropped VoIP calls, or providing customer support.
Yeah! I’m done! Wait? I’m not?
Well, you’re more than halfway done. You’ve done the heavy lifting of getting your log data organized and centralized so that you can identify problems on your network when they happen. That is great. Now you get to use this new tool to get insight into what is happening on your network.
Flash back to the start of this post and you can see how this tool can help you figure out what is happening.
What is going on?
You tell the helpdesk to put in a ticket to Network Operations, and the Ops team opens up the centralized log server and does a query. Sure enough, there is a switch in the conference room that is blasting out a ton of bad packets. Looking a bit deeper, they see it’s an IoT device that has gone bad and is flooding the network with bad packets.
No other alerts have been triggered.
The Ops team shuts off the port on the switch, traffic returns to normal, the event is logged in the ticketing system, and the New York network person has to replace the bad IoT device Tuesday morning.
Mystery solved, crisis averted, and you can chalk up that win to using the centralized log server to identify the offending switch. And as you continuously improve your cybersecurity posture throughout this year and into the next, it’s all the more reason add the centralized log server to your toolbox.
Need more help with your cyber defense? Contact the CBTS cybersecurity team today.
More blogs from CBTS Consulting CISO John Bruggeman:
What is Cyber Insurance and do I need it?
What do new TSA requirements mean for the security of your critical infrastructure?
How do you ensure the security of your supply chain?
Can you be ransomware-proof? Is that even possible?
Getting ransomware-proof, continued: CIS controls for medium-size organizations