The Theory and Reality of SIEM ROI

Many things are written about SIEM, yet my personal experience with these wonderful tools began back in 2007. Today the technology itself is more than 18 years old and SIEM is by all means a mature market. Together with clients, team and partners I was privileged to actively participate in more than a hundred of SIEM projects all over the world. Together we’ve built SOC from scratch, implemented critical SOX log sources to comply with tight audit deadlines, resurrected an abandoned SIEM installation within a half a day to locate forensics clues and conducted fine-tuning of anti-fraud correlation rules… and simply had a proper amount of beer together with companies from all industries and parts of the world while discussing history of victories and defeats of SIEM projects. It is known that more than 40 thousand organizations around the world have SIEM systems so this is a rather big community and of course market. What is certain is that the story is far from over as the very concept of SIEM continues its evolution. Most organizations that are not indifferent to cybersecurity already have a SIEM in some form or another. The boundaries between SIEM technologies are blurred: there are first, second and third generations of the platform, though stating that IBM QRadar or Micro Focus ArcSight are technologies of the first generation and LogRhythm is of the second one is not entirely correct, as the technologies are constantly evolving. Also, claims that Splunk and Elastic are not a SIEM are incorrect since they have many SIEM properties and functionality.

So how do SIEM and ROI come together? If we look at ROI through the eyes of any business it would mean either an additional profit or a cost reduction. One can give many examples on the return of investment in information security enough to write a separate article, so for now let’s focus on the SIEM. It is difficult to believe that anyone invests in these systems to earn money so we’re left with the cost reduction driver. Let us try to figure it out.

How does SIEM appear in our lives?
5 basic ROI drivers

1. Compliance
2. Central log repository
3. Forensics
4. Security Operations Center (SOC)
5. Heritage

SIEM for PCI, SOX and HIPAA Compliance

Very often SIEMs are bought by financial institutions and banks since different compliance regulations require the collection and processing of log data. PCI DSS requires to centrally collect log data, store and retain it in an unmodified form, provide audit trail of its integrity (a very important requirement that not every SIEM can fulfill even to this day) and of course to monitor events on daily basis. If we need to comply with SOX it is necessary to track and handle incidents of all SOX critical systems and depending on this auditors trust or do not trust business data in the processed systems. If incidents are not handled, audit becomes much more expensive. Another example is HIPAA compliance with the requirement to maintain an audit trail on the access to patient data for up to 5 years. This primarily affects the architecture of SIEM data processing: the flexibility of 3rd party system integration capabilities, the efficiency and stability of aggregation, algorithms of data compression as well as integrity control.

So what kind of ROI can SIEM generate for Compliance? The most clear is the absence of fines and penalties due to not just following but exceeding requirements of the regulations. This makes an excellent return on investment from SIEM since investment into the project will likely be at least 10 times smaller than the penalty for compliance violation. It is advised to create a 5-year TCO model including all expenses such as support, hardware, software cost, team cost, training and retention program. Then match TCO with possible savings on Compliance violations based on risks you face. This would make ROI proven not just on paper and help to avoid miscalculations.

Central log repository

The best way to calculate the ROI from using SIEM as the central log repository is pretty straightforward: determine which log data we need to collect and store, how much space they will take, hardware specifications and workload for this task. It is necessary to take into account all parameters from HDD space to the cost of 1 CPU core (especially cloud-based and virtualization deployments). All log management and SIEM systems are very good at compressing the log data with 2X to > 10X efficiency which leads to direct savings on storage, even if data enrichment and normalization are performed. The ability of particular SIEM to granularly tune log data aggregation and filtering will directly improve storage efficiency and ROI. Aggregation is present in any mature SIEM platform, basically all systems you see in Gartner do have some kind of aggregation. Capabilities vary from vendor to vendor: very often it’s just an on/off button, while in other technologies we’ve got over 9000 parameters that can be configured to fit any infrastructure.

SIEMs are built to support quick search and timely provision of information. Additionally we use the technology to simplify backup and management of log data. Surprisingly enough, it is much easier to manage a centralized log storage system rather than look after each log source individually. The archives created by SIEM very often have built-in integrity control and are hard to tamper with: they are either encrypted or created as a single container where breaking integrity of the container will not go unnoticed. All mature SIEMs control the integrity of such containers and log retention.

We often see how SIEM projects arise as both IT and security initiatives mainly due to log data storage savings reason. This is especially true for Splunk and Elasticsearch platforms since they both are most suited for this task. A typical scenario here is when IT department gets management support and funding for the project, acquires the platform and then security team gets an additional license as an upgrade since the integration of technologies from the same vendor reduces TCO and simplifies support.

ROI on Proactive Forensics operations

Having a centralized secure storage of all log data opens up a way for ROI on Forensics: logs are stored in non-modified tamper-proof format, support fast searching, pivoting and all of this is doable in reasonable time if we did our SIEM implementation right. Thus our investigation of incidents can be performed faster and getting the evidence becomes more simple. When do we really need that? Well if our infosec experience taught us something it is that when an incident occurs most organizations do not have full-time experts to conduct an investigation and this responsibility falls to the information (or cyber) security department in one way or another. Only large enterprise and public sector organizations can afford to allocate an employee for these purposes. And in the eyes of the business forensics is a significant cost to the company that can’t be avoided. If SIEM has all the necessary data then a CHFI specialist will conduct an investigation much faster and focus on the main goal – finding adversary and evidence. It is much easier to run an investigation when log data is available in original format, predefined reports and exports actually exist and don’t take days to retrieve, traffic dumps (PCAP’s) are available and we may even have suspicious security events aka incident statistics to better build the full picture. In the opposite situation, forensic specialist will have to dig deep and likely travel onsite and personally collect the necessary data from servers bit-by-bit. In the latter situation time to perform proper investigation goes up which directly increases forensics costs or decreases the quality to a degree where final result may be disappointing. If incident never happened it is next to impossible to calculate ROI on proactive forensics. Though if we look at the growing trend of data breaches, APT attacks and cyber incidents statistics and embrace reality we realize that sooner or later any organization is getting hacked. And getting breached inevitably leads to dealing with forensics investigation directly or indirectly for any CISO.

SIEM and Forensics in real life

Some time ago We worked closely with our partner on investigating a possible insider attack. When we got onsite there was no SIEM or even central log management deployed and just as in classics someone had deleted logs wherever they had a chance, backups were not performed, ACLs were not there, no PCAP files were available. We analyzed target images, sent data to lab for recovery, performed some darknet surfing and digging.. As a result client’s management paid up for 5 days of forensics + time for report and presentation, which proved the insider attack traces and damages, but since company had no log management and SIEM it was impossible to pinpoint this attack to a specific person, so attribution was the best way to finalize the investigation. Company could have invested more resources into data recovery, but the estimate was yet another 30 days which is by all means a rather costly endeavor so we did not really advise that. Was the customer satisfied with the result considering that in fact we did not find anything? Yes, they got results of official investigation usable in court. Though IMHO this is one very expensive reason to check if whether a SIEM system is needed in the company.

There’s also a small example of the successful story of SIEM ROI in the investigation of APT attack in 2015-2016, when we acted as consultants in the investigation of incidents that appeared part of with BlackEnergy APT attack. APT target company was running a PoC implementation of SIEM (in this case it was arcsight) which essentially helped to find the Microsoft Event log codes that have shown the sequence of new service installation and we discovered how the malicious drivers were injected into the system. Since all audit logs were deleted on the attacked systems (BlackEnergy and KillDisk) SIEM was running on Unix and in the isolated segment so BlackEnergy did not reach it (KillDisk for unix did not/does not exist). So in this case SIEM helped to preserve the traces of the attack and was the first hint to begin full forensics investigation.

Measuring the SOC ROI

The most common driver for SIEM implementation is to build a center for proactive monitoring of security incidents in order to timely identify them and preventively reduce risks. So basically building a SOC is a common reason for buying SIEM. The main task of SIEM system in a SOC is to perform all kinds of events correlation and to find what human physically can not due to volumes of data that need to be analyzed. These days SIEMs are capable of processing billions of events per day, hundreds of thousands or even million+ of EPS and provide information in a form suitable for quick analysis and with all the necessary investigation tools. Meeting these requirements of unites Compliance, Forensics and Central Log Repository around SOC and raises them to a completely new level.

Determining SOC ROI is a big subject, however in its basics it is necessary to monitor and count all incidents detected and model prevented damage at an early stage. It is necessary to build and continuously maintain an asset and network model of the enterprise, cost and value of IT assets, applicable risks, ideally to build a CVSS model too. Based on cost of incidents on these assets it will be possible to determine potential damage prevented. SIEM itself also saves time for analysts and helps to find those extra hours needed for other security tasks such as responding to audit and risk management requests, performing root-cause analysis and investigations.

SIEM Use Cases make a valuable heritage

The last case we often see is a situation when someone inherits SIEM. No one knows why the system was acquired in first place because initial employees have left, new ones have arrived and this situation expands to all levels, from top managers to field specialists. Imagine this: you come to work in a new organization that has a SIEM. The management says: “We have integrated all the audit logs and it cost company several grands / millions of  dollars… Surely you can do something with it? I mean, the task at hand is pretty simple, we got documents, technology is mature and everything is working almost perfectly! So just a few tweaks are needed here and there and we will get total security awareness and network visibility. This would be very useful!” An ambitious SIEM project or even SOC should be started with something simple. This is actually ROI in its purest form. Here we need to focus on quick wins to show that the technology was bought for a good reason. To succeed with this task it is best to select Use Cases that were already built and tested for the technologies you have in your organization and select point-and-shoot solutions aimed at specific threats and risks that are most active in the wild, such as Ransomware, APT, Windows Monitoring, VPN, SSL security. And then we’d need to quickly tune them to match the ROI drivers.

Why SIEM ROI is so good in theory and yet so far from reality?

One lesson from overseeing over a hundred of implementations: just because a SIEM system is implemented in organization does not mean that it collects all the data that was planned and connected during initial integration phase. Integration is a process, call it continuous if you will. And a lack of data = absence of ROI. And it’s not just about acquisition: as any analytical system SIEM follows “garbage in = garbage out” principle. Simply put it will not produce your good results if data challenges are ignored and in a case of compliance audit the required data may not be available if Data Quality and Data Acquisition are not monitored. We also know that any company’s IT infrastructure is not static, architecture is changing constantly as new services are introduced and decommissioned, log data format and transport change, firmware and OS configurations change and this makes parsing events to ensure Data Quality a process that needs to be continually refined. If quality is not controlled with at least 0.25 FTE for the task then usefulness of data will be constantly decreasing and at the time of, say, PCI audit company may not pass it. Performance problems can also lead a situation where reports fail just when you need them and usually that’s when auditors or management request them “by yesterday”. This is especially critical for SOX.

SIEM is not self-sufficient: regardless of whether it is on premise software or a SaaS it needs trained personnel who are either part of in-house team or part of outsourcing service. Incorrect investment calculation happens quite often for SIEM projects. Investments in the SIEM technology during the first 5 years should only make up to 20-30% of total project budget. One needs to invest in equipment and in team, especially in training, staff retention and scaling of the operations. Another mistake is to assume that buying SIEM will significantly reduce the workload, on the contrary it will actually bring in a number of new tasks and required skills. Your experts simply receive an opportunity to work with huge amounts of security data analyze it and make meaning out if it to reduce risks, something that was previously impossible. Thus, improper resource planning is another reason why it is not possible to obtain a ROI from SIEM.

Finally, an approach to the implementation of Use Cases for SIEM requires understanding that use cases are not implemented as one time thing, and their quality will rapidly degrade if they are not improved and tuned continuously. The most effective and at the same time simple solution here is to use already available use cases and not reinvent the wheel (cases for Windows monitoring, Cisco, antivirus, etc.). It is necessary to focus the efforts of the team on the development of specific use cases for your organization. To verify that you are building required use cases: if use case can be copied to another company’s SIEM and it will work there with same value, this means it is universal. And there is a high probability that it is already available as freemium or commercial package that requires minimum to no tuning to produce security value.

Thus, we can define 5 processes which ensure positive ROI from SIEM

1. Monitoring and maintaining Data Acquisition and Data Quality
2. Monitoring, maintaining and continuously improving SIEM Use Cases
3. Proper technical planning of the project and architecture
4. Planning and adjusting FTE in advance to prevent team overload
5. A clear plan for investment in the team: training, retention and development

So here we have the main ways to get ROI from SIEM and some reasons on why they aren’t always working as intended. What is your experience of getting ROI from a SIEM?