Active Lists in ArcSight, automatic clearing. Part 1

ArcSight beginners and experienced users very often face a situation when they need to automatically clear Active List in a use case. It could be the following scenario: count today’s logins for every user in real-time or reset some counters that are in Active List at the specified time.

I want to believe that ArcSight didn’t add such functionality to ESM yet for convincing reasons.

There are several possible ways to achieve automatic clearing of Active List:

• Via ssh, using custom script;
• By using rules;
• By using trends. Today I will describe this variant.

The main idea is about the usage of a scheduled trend to delete entries in Active List through temporary list and rule.

For lists with Key field:

1. Create Query(1) on Main Active List(1). Select only Key Fields to query.
2. Create new Trend with Query(1). Set short ‘Partition Retention Period (in days)’ parameter (few days). And schedule it to run every day for example at 23:59:00.
3. Create new Temporary Active List(2) with Fields that are similar to Key Fields from Main Active List(1). Set TTL parameter to 1 minute. This list will be used as a buffer for entries that you need to clear from Main Active List(1).
4. Edit Trend that was created in point 2. Add Action ‘Add to Active List’ and choose ‘Temporary Active List(2)’.
5. Create new Rule and add the condition:

& AND

Device Event Class ID = activelist:104

File Name = Temporary Active List(2)

Type = Base

Add Action ‘Remove From Active List’ and choose Main Active List(1) select ‘Device Custom String4’ field in line with the key field.

If you have more than one key field, you need to use local variables ‘EvaluateVelocityTemplate’ to divide the key value in Device Custom String4’ field.

Deploy rule into Realtime.

So this Trend will be run every day at 23:59:00, it will get all entries that are in Main Active list and add them to Temporary Active List. All entries in Temporary Active List will expire in 1 minute, and Rule will catch up all entries and remove them from Main Active List. Thus, you will get empty Active List at the beginning of a new day.

You need to create a new trend for each active list that you want to be cleared automatically. This method is not very convenient but solves the task of Active List automatic clearing. In the next posts, I will describe other two ways to achieve such results.