ArcSight. Optimizing EPS (Aggregation and Filtration)

Almost all of the ArcSight beginners face a situation when there are a high incoming EPS from the log sources, especially when it is critical to License limits or causes performance issues.

To reduce incoming EPS, ArcSight has two native methods for event processing: Event Aggregation and Filtration. In this article, I will try to explain how to optimize incoming EPS using these two options.

Aggregation of Events

The first and the most effective option is Aggregation on connectors. Aggregation allows to aggregate a lot of similar events into one event; it is like smart compression. It can aggregate up to 10000 events into 1 event; this means you can reduce incoming EPS up to 10000 times. Let’s see how it works.
Firewall sent 3 similar events to ArcSight:
Outcome=Allow, Source IP=x.x.x.x, Source Port=xx, Destination IP=y.y.y.y, Destination Port=yy
Outcome=Deny, Source IP=x.x.x.x, Source Port=xx, Destination IP=z.z.z.z, Destination Port=zz
Outcome=Allow, Source IP=x.x.x.x, Source Port=xx, Destination IP=y.y.y.y, Destination Port=yy
Here we see that 1st and 3d events are the same, and in this case Connector with enabled aggregation will combine 1st and 3d events into one event with the field:
Outcome=Allow, Source IP=x.x.x.x, Source Port=xx, Destination IP=y.y.y.y, Destination Port=yy Aggregated Event Count=2.

To configure Aggregation, go to Connector’s settings chapter ‘Field Based Aggregation’.
Set parameters:
Time Interval. For how many seconds connector should group the same events. It is not recommended to set time more than 30 seconds, since for this time events will be held up by the connector and as a result, will be delivered to Destination with delay.
Event Threshold. How many events should be aggregated with the time window. Set event count to be aggregated. The connector will group this amount of similar events into one.
Field Names. Defines fields which should have the same value for aggregation. Choose all fields that you need to save in the ArcSight database. All other fields that are not defined in aggregation field list will be lost, so pay attention to this.
Fields to Sum. Choose numeric fields that you want to sum. Most often these are ‘Bytes In’ and ‘Bytes Out’ fields.
Preserve Common Fields. Set ‘yes’ if you want to save other fields in the aggregated event (if they are common).

Filtering Events

The second option for optimizing EPS is Filtering out unnecessary events on the connector. This option allows you discard events that are not important on the connector level, so you do not need to modify your log sources.
It is more convenient to configure it from the ArcSight Console in Connector Setting, tab Default, sub-tab Filters. Here you can set the filter for the events that you don’t want to come to ESM. Note that you filter OUT events here so if you don’t want the connector to send Windows Event ID 5156: ‘The Windows Filtering Platform has allowed a connection’ you need to add the filter as shown in the screenshot:

In case you want to send only specific events (create a whitelist of events), you need to add negotiation to the filter. For example, you want to send only Successful and Failed logons (Event IDs 4624 and 4625), so you need to configure the filter as shown in the screenshot below. If you do, only these events will be sent to ESM.

But what if you need to filter out events to Not ESM destination. You can configure this in Connector Setting menu in format ‘deviceEventClassId EQ “Microsoft-Windows-Security-Auditing:5156” ’ (without quotation marks). In the second case when you need to send only Events with Id 4624 and 4625 filter should be similar to this: ‘ Not (deviceEventClassId EQ “Microsoft-Windows-Security-Auditing:4624” Or deviceEventClassId EQ “Microsoft-Windows-Security-Auditing:4625″)’
In case you need to configure a complicated filter to not ESM destination, I recommend to set and test it first from ESM console and only after that copy it from the Connector’s destination configuration .xml file in /current/user/agent/ folder (name looks like 312jhSFgBABCV2Sp8uG1sLA==.xml). You need to find string:
zonebasedfiltering.zonedefinition=” Not (deviceEventClassId EQ “Microsoft-Windows-Security-Auditing:4624” Or deviceEventClassId EQ “Microsoft-Windows-Security-Auditing:4625″)”

Using aggregation together with filtering events, you can optimize and significantly reduce your incoming EPS rate. But be careful, an inconsiderate use of these parameters can lead to the loss of important data.

If you are interested in improving ArcSight operations, also read the article Deliver IT feeds into ArcSight without false positive triggers.