Fluentd: How to Make Nested Hash from Dot-Separated Keys

When a log record has keys like test.test, you can use two methods record_transformer and explode plugin, to process these keys and create a nested structure.

For example, Input Log Record:

{
  "message": "A test message",
  "test.test": "value1",
  "test.another": "value2"
}

Method: record_transformer

<filter *>
    @type record_transformer
    enable_ruby true
    auto_typecast true
    remove_keys test.test,test.another
    <record>
    test                               ${{"test": record['test.test']}}
    test                               ${{"another": record['test.another']}}
    </record>
</filter>

Method: explode

<filter *>
    @type explode
</filter>

Using those methods, you will get one result.

Output Log:

{
  "message": "A test message",
  "test": {
    "test": "value1",
    "another": "value2"
  }
}

But when using record_transformer, you have to follow many steps to create nested fields, and the result is identical to the result where you use explode and turn it on in one step.

The Explode plugin takes the top-level keys with dots and breaks them into nested structures.
This way you can do with fields that go through from the explode plugin any action without an issue

For example, Input Log Record:

{
  "user.name.first": "John",
  "user.name.last": "Doe",
  "user.contact.email": "john.doe@example.com"
}

<filter **>
  @type explode
</filter>

<filter **>
  @type record_transformer
  <record>
    full_name ${record["user"]["name"]["first"]} ${record["user"]["name"]["last"]}
  </record>
</filter>

Output Log Record:

{
  "full_name": "John Doe"
  "user": {
    "name": {
      "first": "John",
      "last": "Doe"
    },
    "contact": {
      "email": "john.doe@example.com"
    }
  }
}

This example creates a new full_name field by extracting data from the nested structure created by the plugin explode.

Explode is a custom plugin so you will need to install it using the command below:

gem install fluent-plugin-explode_filter