How to Convert Arrays of Hashes Into a Structured Key-Value Format During Log Processing

In some log formats, fields can be arrays of hashes, requiring conversion into a structured key-value format. Fluentd supports this through inline Ruby scripts, enabling transformations during log processing.

For example, I need to convert the event_data field:

{
  "event_data": [
    {"name": "feature_enabled", "boolValue": true},
    {"name": "user_roles", "multiValue": ["admin", "editor"]},
    {"name": "session_id", "value": "abc123"}
  ]
}

Step 1. Write Ruby-Based Transformation Logic

The transformation requires iterating over the event_data array, extracting meaningful information, and producing a hash. The logic can be implemented as:

record['event_data'].map { |item| [item['name'], item['boolValue'] || item['multiValue'] || item['value']] }.to_h

• Access the event_data Field:

• Iterate Over the Array:

• Extract Key-Value Pairs:

• Convert to a Hash:

Step 2. Setup Implementing in Fluentd

Use the record_transformer filter to apply the transformation dynamically. Enable Ruby scripting with the parameter enable_ruby true.

In Fluentd Configuration:

<filter app.logs>
  @type record_transformer
  enable_ruby true
  <record>
    event_data ${!record.dig('event_data').nil? ? record['event_data'].map { |item| [item['name'], item['boolValue'] || item['multiValue'] || item['value']] }.to_h : record['event_data']}
  </record>
</filter>

• Conditional Logic:

• Transformation Logic:

• Fallback:

{
  "event_data": {
    "feature_enabled": true,
    "user_roles": ["admin", "editor"],
    "session_id": "abc123"
  }
}