Ruan Bekker's Blog

From a Curious mind to Posts on Github

Ship Your Docker Logs to Loki Using Fluentbit

In this tutorial, I will show you how to ship your docker containers logs to Grafana Loki via Fluent Bit.

Grafana and Loki

First we need to get Grafana and Loki up and running and we will be using docker and docker-compose to do that.

Our docker-compose-loki.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

version: "3.7"

services:
  grafana:
    image: grafana/grafana:7.4.2
    container_name: 'grafana'
    restart: unless-stopped
    volumes:
      - ./data/grafana/data:/var/lib/grafana
      - ./configs/grafana/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yml
    networks:
      - public
    ports:
      - 3000:3000
    depends_on:
      - loki
    logging:
      driver: "json-file"
      options:
        max-size: "1m"  
  
  loki:
    image: grafana/loki:2.1.0
    container_name: loki
    command: -config.file=/mnt/loki-local-config.yaml
    user: root
    restart: unless-stopped
    volumes:
      - ./data/loki/data:/tmp/loki
      - ./configs/loki/loki.yml:/mnt/loki-local-config.yaml
    ports:
      - 3100:3100
    networks:
      - public
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

networks:
  public:
    name: public

We are referencing 2 config files, first our loki datasource defined by ./configs/grafana/datasource.yml:

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: 1

datasources:
- name: loki
  type: loki
  access: proxy
  orgId: 1
  url: http://loki:3100
  basicAuth: false
  isDefault: true
  version: 1
  editable: true

And our second config is our loki config ./configs/loki/loki.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Once you have everything in place, boot the grafana and loki containers:

1

$ docker-compose -f docker-compose-loki.yml up -d

Fluent Bit

Next we need to boot our log processor and forwarder, fluent bit. In our docker-compose-fluentbit.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

version: "3.7"

services:
  fluent-bit:
    image: grafana/fluent-bit-plugin-loki:latest
    container_name: fluent-bit
    environment:
      - LOKI_URL=http://loki:3100/loki/api/v1/push
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    networks:
      - public

networks:
  public:
    name: public

And as you can see we are referencing a config ./configs/fluentbit/fluent-bit.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224
[Output]
    Name grafana-loki
    Match *
    Url ${LOKI_URL}
    RemoveKeys source,container_id
    Labels {job="fluent-bit"}
    LabelKeys container_name
    BatchWait 1s
    BatchSize 1001024
    LineFormat json
    LogLevel info

Once you have your configs in place, boot fluent-bit:

1

$ docker-compose -f docker-compose-fluentbit.yml up -d

Nginx App

Now to configure our docker container to ship its logs to fluent-bit, which will forward the logs to Loki.

In our docker-compose-app.yml:

1
2
3
4
5
6
7
8
9
10
11
12

version: "3"

services:
  nginx-json:
    image: ruanbekker/nginx-demo:json
    container_name: nginx-app
    ports:
      - 8080:80
    logging:
      driver: fluentd
      options:
        fluentd-address: 127.0.0.1:24224

The fluent-bit container listens on port 24224 locally on our docker host and is not reachable via its container network, so let’s boot our application:

1

$ docker-compose -f docker-compose-app.yml up -d

Once our application is up, let’s make a request to our nginx-app:

1
2

$ curl http://localhost:8080/
ok

Now head over to Grafana at http://localhost:3000/explore and query: {job="fluent-bit", container_name="/nginx-app"} and you should see something like this:

image

Beautiful right? I know.

Github Repo

The source code for this can be found on: