How to Parse Nested JSON Fields in Fluentd
Here is how you can parse log entries with nested JSON fields. Consider the following log entry:
{"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message": "{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}
If you have a Fluentd configuration like this:
<source>
@type dummy
dummy '{"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message": "{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}'
format json
tag mylogs
</source>
<match mylogs>
@type stdout
</match>
You'll notice that Fluentd does not parse the nested contents under the message field, as seen in these log entries:
2024-01-27 09:03:18.656280395 +0000 mylogs: {"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message":"{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}
To resolve this, you need to use a filter with a parser. Here's how it's done:
<source>
@type dummy
dummy '{"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message": "{\"user\":{\"userId\":\"456\"},\"action\":\"Payment\\ failed\"}"}'
format json
tag mylogs
</source>
<filter mylogs>
@type parser
key_name message
reserve_data true
hash_value_field message
<parse>
@type json
</parse>
</filter>
<match mylogs>
@type stdout
</match>
In this configuration, you introduce the filter block with @type parser. The key_name specifies the field (message) to be parsed. Next, the reserve_data field maintains the original data structure. After that, the hash_value_field determines where the parsed values are stored. In this case, it overwrites the original message field. Alternatively, you can use a different field name like parsed_message.
After making these changes, Fluentd will successfully parse the nested JSON content:
2024-01-27 10:03:05.016725559 +0000 mylogs: {"timestamp":"2024-01-27T08:33:17","severity":"ERROR","message":{"user":{"userId":"456"},"action":"Payment failed"}}
If you are still learning to use Fluentd, check out our guide on how to collect, process, and ship log data with Fluentd, which is a great starting point.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
-
How to Add Tags to My Fluentd Events
Here is how you can add tags to Fluentd events. Let's assume you configured Fluentd to process Nginx access logs from the access.log file with a source configuration like this: @type tail pa...
Questions -
How to Use Conditional Statements In Fluentd
Conditional statements can be seamlessly integrated into Fluentd's record_transformer plugin. Here’s how you can do it: @type recordtransformer enableruby true is_successful ${record["s...
Questions -
Fluentd
Learn how to use Fluentd to collect, process, and ship log data at scale, and improve your observability and troubleshooting capabilities.
Guides -
How to Parse Nested JSON Fields in Fluentd
Learn how to configure Fluentd for nested JSON parsing in log messages for enhanced structured logging
Questions