Reducing the Breach Detection Time: Availability of the Log Data

Hello again! In previous article, we already established that many things might get out of hand when you are into building a virtual or a full-scale SOC, especially when it comes to operationalizing the SIEM as the core technology of any SOC. We have also established that automation is the way to go if one wants to keep up with modern threats and overhead that is produced by SIEM & SOC technologies. Today we begin analyzing each SIEM deployment and operations component piece by piece, the first that comes to mind is data availability. There are many factors that impact breach detection time and as of 2015 it is still above 200 days average, but this is not because SIEM is a technology of choice for breach detection. A SIEM can only provide you the output based on the inputs it gets, and if we miss something at the input, we should not expect the output to be whole, actionable and sometimes we will not have any output at all.

Is it really SIEM’s fault that it misses logs (and incidents)? Well, this again depends on many factors and to clear this mess up and to maintain SIEM continuously operational a person responsible for its administration/maintenance (your company has one, right?) needs to have enough information at any given point of time to answer these two questions: “Is all in scope log data being collected?” and if the answer is a NO “Is the trouble on SIEM side or outside of it?”. If the answer to first question is a NO, be assured that you are missing important things that have direct impact on your incident detection time and accuracy. And the truth is, no SIEM solution simply answers this out of the box – it is always a manual effort to respond to this question.

Have you ever seen a SIEM that says “Your network log data availability is 85%”? I didn’t. However, we still have a second question that obviously is not answered by system either since it cannot answer the first one. But, all is not lost and answers are there if one does enough research or has devoted enough time and effort to seek them manually.

Let us consider some examples based on how one collects log data and start with most common log collection mechanism – syslog. It is unlikely one can name one SIEM implementation that does not use syslog at all. There are many ways data can be lost with syslog: a daemon on pipe collection method loses data when the collecting component is stopped; a UDP (default) syslog protocol has no guarantee for packet delivery, high volume of syslog traffic (both UDP and TCP) can be impaired by buffer size limits and bandwidth limitations and even packet rate processing specs of a concrete NIC. Even in a case of reading log data from a file one must consider file rotation and integrity. Diagnostics of these issues is always a manual routine task that involves reading diagnostic logs of SIEM components itself.. If it has diagnostic data to begin with! Even the ugliest diagnostic data is better than no diagnostic data at all and a cheap excuse of “oh, our SIEM is so magical that it has no errors!”. Next we would be busy with building(or re-using?) correlation content that baselines log flow amounts and deviations (Is anyone really satisfied with results of such a content? What about the resources it eats up?), running TCPdump to check packet drop rates, monitoring components availability through external sources… Just wait, I think we came up with at 3 additional tools that need to be added to SIEM to monitor itself, just to answer a simple question “What is % of my network log data availability?”. If we talk concrete attempts to make this automatic, say, Splunk on Splunk, are they really efficient? How much extra $$ of license cost one has to add to be able to self-diagnose the SIEM/Log Management platform and how much performance overhead does this most popular app produce?..

Okay, let us put syslog and Splunk aside for few minutes and think about second most popular log source – Microsoft Windows Event Log. To keep issues short: log rotation, network bandwidth, authentication errors / password lockouts, WMI and JCIFS instability, high load of busy Windows Servers (file audit, Active Directory etc.). Monitoring this would again require a whole set of tools and those tools will be different when compared to syslog monitoring tools! We can go on for a long list including databases, firewalls/ngfw/ips/ngips/utm (hello OPSEC & SDEE) and will discover even more things that happen outside of SIEM, that SIEM has no effect on but must know about(!) and it has provide this information fast and accurate to the administrator. Yet, for ones who read this far, there are good news, since SIEM (or most SIEMs) itself has readable (almost) diagnostic logs. Those hard to find diagnostic files, hidden API calls or multiline java stack traces have bits and bytes of information that could provide answers to many of the issues raised above. And by combining those together and applying meaningful profiling metrics we can answer those two simple questions that mean a whole difference between knowing your mission critical security detection platform is working as intended or ignoring the issue / half-fixing it with throwing more FTE at the problem. Automation is here for all who want to assure their log data is available as planned in the project, required by organization/customer and they get a proper return on their SIEM investment. I hope this leaves you with some food for thought. Stay tuned!

p.s. In case you happen to be one of the HPE ArcSight experts, there are more good news! As part of global initiative to make SIEM platforms deliver the value and saving time of SIEM experts worldwide, SOC Prime provides a free online instant SIEM Health Check service, that can eat agent.log files and provide you top 5 critical issues and solutions to them < 5 minutes. Got an agent.log? Throw it here and see for yourself!