Configure Fluentd for Log Collection

Fluentd sidecar

For components with multiple log streams or that cannot log to stdout, Verrazzano deploys a Fluentd sidecar which parses and translates the log stream. The resulting log is sent to stdout of the sidecar container and then written to /var/log/containers by the kubelet service.

For example, in a WebLogic deployment, AdminServer.log is consumed, translated, and written to stdout by the Fluentd sidecar. You can view these logs using kubectl on the container named fluentd-stdout-sidecar.

$ kubectl logs tododomain-adminserver \
   -n todo-list \
   -c fluentd-stdout-sidecar

The Verrazzano Fluentd Docker image comes with these plug-ins:

• fluent-plugin-concat
• fluent-plugin-dedot_filter
• fluent-plugin-detect-exceptions
• fluent-plugin-opensearch
• fluent-plugin-grok-parser
• fluent-plugin-json-in-json-2
• fluent-plugin-kubernetes_metadata_filter
• fluent-plugin-multi-format-parser
• fluent-plugin-parser-cri
• fluent-plugin-prometheus
• fluent-plugin-record-modifier
• fluent-plugin-rewrite-tag-filter
• fluent-plugin-systemd
• fluent-plugin-oci-logging

The Verrazzano Fluentd Docker image also has two local default plug-ins, kubernetes_parser and kubernetes_multiline_parser. These plug-ins help to parse Kubernetes management log files.

Here are example use cases for these plug-ins:

# ---- fluentd.conf ----
# kubernetes parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>

# kubernetes multi-line parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>
# ----   EOF      ----

Fluentd DaemonSet

Verrazzano deploys a Fluentd DaemonSet which runs one Fluentd replica per node in the verrazzano-system namespace. Each instance pulls logs from the node’s /var/log/containers directory and writes them to the target OpenSearch data stream. Verrazzano system applications receive special handling, and write their logs to the verrazzano-system data stream. Verrazzano application logs are exported to a data stream based on the application’s namespace, following this format: verrazzano-application-<application namespace>.

For example, opensearch-dashboards logs written to /var/log/containers will be pulled by Fluentd and written to OpenSearch. The logs are exported to the verrazzano-system data stream, because opensearch-dashboards is a Verrazzano system application. For a non-system application, if it is in the myapp namespace, then its logs will be exported to the verrazzano-application-myapp data stream.