Standard Logstash Template for Event Processing (Gold Template)

This standard template for configuring Logstash pipelines, commonly referred to as a “gold template,” ensures consistent metadata enrichment for events processed through Logstash, making it particularly useful in environments where data comes from diverse sources.

Configuration Template

filter {
    if [type] == "TYPE_NAME" {  # Replace 'TYPE_NAME' with your specific type
        ruby {
            init => "require 'socket'"  # Load the 'socket' library for hostname resolution
            code => "
                event.set('[receipt0][time]', LogStash::Timestamp.new(Time.now))  # Add the current timestamp
                event.set('[receipt0][hostname]', Socket.gethostname)  # Add the Logstash server's hostname
            "
        }
        if [host] =~ /\d+\.\d+\.\d+\.\d+/ {  # Check if 'host' matches an IP address format
            mutate { rename => { "host" => "[agent][ip]" }}  # Rename 'host' to '[agent][ip]' if it is an IP address
        } else {
            mutate { rename => { "host" => "[agent][hostname]" }}  # Rename 'host' to '[agent][hostname]' otherwise
        }
        mutate {
            add_field => { "[receipt0][type]" => "logstash" }  # Add a field to indicate processing by Logstash
        }
    }
}

Key Features

Ruby Block for Metadata Enrichment

Purpose: the Ruby block adds dynamic metadata to the event, including:

• Current processing time ([receipt0][time])

• Hostname of the Logstash server processing the event ([receipt0][hostname])

Advantages: this metadata provides a traceable record of when and where the event was processed, aiding in debugging and monitoring.

Host Field Renaming

Logic: Depending on whether the host field contains an IP address or a hostname, it is renamed to [agent][ip] or [agent][hostname], respectively.

Benefit: this standardizes the structure of events for downstream systems, simplifying data querying and analysis.

Adding Logstash Type Information: the [receipt0][type] field explicitly marks the event as processed by Logstash, helping identify its origin.

How to Use

1. Replace TYPE_NAME. Replace "TYPE_NAME" with the appropriate event type you want this template to process (e.g., "auditd", "syslog", etc.).
2. Insert into Pipeline Configuration. Add the template to your Logstash pipeline configuration under the filter section.
3. Test the Configuration: validate your configuration using:
   • bin/logstash –path.config /path/to/config –config.test_and_exit
4. Deploy to Production. Once tested, deploy the updated configuration to your production environment.

Example Use Case

filter {
    if [type] == "syslog" {
        ruby {
            init => "require 'socket'"
            code => "
                event.set('[receipt0][time]', LogStash::Timestamp.new(Time.now))
                event.set('[receipt0][hostname]', Socket.gethostname)
            "
        }
        if [host] =~ /\d+\.\d+\.\d+\.\d+/ {
            mutate { rename => { "host" => "[agent][ip]" }}
        } else {
            mutate { rename => { "host" => "[agent][hostname]" }}
        }
        mutate {
            add_field => { "[receipt0][type]" => "logstash" }
        }
    }
}

• The current timestamp and Logstash hostname are added.
• The host field is standardized.
• A marker field is added to indicate Logstash processing.

Advantages of the Gold Template

• Consistent Metadata: Enriches events with uniform metadata for easier traceability.
• Standardized Field Names: Reduces ambiguity by renaming fields based on content type.
• Adaptable to Multiple Use Cases: Easily configurable for different event types.