Fluentd: Work With Multiple Log Sources Within a Single Instance by Using @label

@label is a feature that defines multiple processing pipelines within a single instance. Labels allow you to route log data through different pipelines, applying different configurations or processing steps to different kinds of logs.

How @label Works

@label is used to route logs to specific pipelines. For instance, you might want to route logs from different applications or services to different destinations or process them differently depending on the type of log.

@label Usage Example

Here’s an example of how @label is used in a Fluentd configuration file:

<source>
    @type tcp
    tag tcp.events # required
    @label                              @test-label-001
    <parse>
        @type none
    </parse>
    port 20001   # optional. 5170 by default
    bind 0.0.0.0 # optional. 0.0.0.0 by default
    delimiter "\n" # optional. "\n" (newline) by default
</source>

<source>
    @type                               http
    @label                              @test-label-002
    port 9881
    bind 0.0.0.0
    body_size_limit 32m
    keepalive_timeout 10s
</source>

<label @test-label-001>
    <filter tcp.events>
        @type                                                   record_transformer
        enable_ruby                                             true
        <record>
            @timestamp                                         ${Time.now.strftime('%Y-%m-%dT%H:%M:%S.%9N%z')}
            message                                            test message
        </record>
    </filter>
    <match tcp.events>
      @type stdout
    </match>
</label>

<label @test-label-002>
    <match **>
      @type stdout
    </match>
</label>

Explanation

• @type tcp: Specifies that this input source listens for incoming data over TCP.
• @type http: Specifies that this input source listens for incoming HTTP requests.

• Logs from @test-label-001 are routed to the pipeline labeled @test-label-001
• Logs from @test-label-002 are routed to the pipeline labeled @test-label-002.

• Logs from tcp source are printed to stdout in the @test-label-001 pipeline with a filter to add new fields.
• Logs from http source are printed to stdout in the @test-label-002 pipeline without any filter.

When to Use

• Separation of Concerns: Use labels when you want to handle different log sources separately, either for routing to different destinations or applying different processing.
• Performance Optimization: Labels can help optimize performance by allowing you to isolate pipelines, reducing the complexity of a single pipeline.

Key Points

• @label defines a separate pipeline.
• Each pipeline can have its own configuration.
• @label allows you to isolate the processing of different logs in the same Fluentd instance.