Fluentd: Variables in config are useful

Oct 12, 2020 · 1 min read · fluentd haproxy  ·
Share on:

Fluentd is necessary software for operating Kubernetes clusters. Variables in the Fluentd’s configs are useful because they reduce redundant writing and also it will come in handy to switch environments between development and production.

For example, I configure the fluentd which is running on our Kubernetes cluster as a DaemonSet as follows.

 1<match **>
 2  @type elasticsearch
 3  type_name _doc
 4  include_tag_key true
 5  host "#{ENV['ELASTICSEARCH_HOST']}"
 6  port "#{ENV['ELASTICSEARCH_PORT']}"
 7  time_key time
 8  reconnect_on_error true
 9  reload_on_failure true
10
11  ## Logstash format settings (index will be "${tag}-%Y.%m")
12  logstash_format true
13  logstash_prefix ${tag}
14  logstash_prefix_separator -
15  logstash_dateformat %Y.%m 
16  
17  <buffer tag, time> 
18    @type file
19    timekey 1h
20    path /var/log/fluentd-buffers/${tag}.buffer
21    flush_mode interval
22    retry_type exponential_backoff
23    flush_thread_count 2
24    flush_interval 5s
25    retry_forever
26    retry_max_interval 30
27    chunk_limit_size 32M
28    queue_limit_length 100
29    overflow_action drop_oldest_chunk
30  </buffer>
31</match>

As you see on line 5 and 6, I defined the ElasticSearch host and port by variables. And then, I configured the Kubernetes YAML file like this.

1        env:
2        - name: ELASTICSEARCH_HOST
3          value: "es.mydomain"
4        - name: ELASTICSEARCH_PORT
5          value: "9200"

On the other hand, the YAML file for the development environment is like this.

1　    env:
2        - name: ELASTICSEARCH_HOST
3          value: "host.docker.internal"
4        - name: ELASTICSEARCH_PORT
5          value: "9200"

So, this will be able to switch these environments easily.