Tag: Fluentd

This guide explains configuring Fluentd to extract structured data from unstructured log messages using the parser plugin with a regular expression (regexp). If you need to extract specific fields, such as log_source and index, from a log message, you can do this as follows. Input Log: Configuration: Explanation: key_name message: Specifies that the message field should be […]

@label is a feature that defines multiple processing pipelines within a single instance. Labels allow you to route log data through different pipelines, applying different configurations or processing steps to different kinds of logs. How @label Works @label is used to route logs to specific pipelines. For instance, you might want to route logs from […]

When a log record has keys like test.test, you can use two methods record_transformer and explode plugin, to process these keys and create a nested structure. For example, Input Log Record: Method: record_transformer Method: explode Using those methods, you will get one result. Output Log: But when using record_transformer, you have to follow many steps […]

I have a case where I need to drop unnecessary logs. I found a plugin that helps do that.The rewrite_tag_filter plugin is used to dynamically modify the tags of incoming log records based on their content. You can rewrite tags,  route logs more effectively, organize them based on certain conditions, and ensure logs are processed […]

In some log formats, fields can be arrays of hashes, requiring conversion into a structured key-value format. Fluentd supports this through inline Ruby scripts, enabling transformations during log processing. For example, I need to convert the event_data field: Step 1. Write Ruby-Based Transformation Logic The transformation requires iterating over the event_data array, extracting meaningful information, and producing […]