How to Process Multiline Log Entry With Logstash Filter?

To process multiline log entries in Logstash, you can use the codec option within the file input plugin to treat multiline messages as a single event. Here’s how:

Example Configuration

1. Set up the multiline codec: Define the start of a multiline event with a regular expression.
2. Combine lines based on patterns: Use patterns to identify which lines should be grouped together.

input {
  file {
    path => "/path/to/your/logs/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"

    codec => multiline {
      pattern => "^\\["                  # Example: Starts with a timestamp or specific pattern
      negate => true                    # Indicates lines not matching the pattern should be appended
      what => "previous"                # Appends lines to the previous line that matches the pattern
    }
  }
}

filter {
  # Additional filters go here
}

output {
  stdout {
    codec => rubydebug
  }
}

Explanation

• pattern: Regular expression defining the start of a new event. For example, ^\\[ captures lines starting with [ (a typical timestamp format).
• negate => true: This option tells Logstash to treat lines not matching the pattern as part of the previous line.
• what => "previous": Specifies that each new line matching the pattern should create a new event, while non-matching lines are added to the previous event.

Tips

• Adjust the pattern to match the beginning of each log entry (like a timestamp or specific keyword).
• Use additional filters (such as grok, mutate, etc.) to further process the combined multiline log event as needed.

Got an article suggestion? Let us know

Explore more

Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.