Ruan Bekker's Blog

From a Curious mind to Posts on Github

Deploy a Monitoring Stack on Docker Swarm With Grafana and Prometheus

Say Thanks! Slack Status Chat on Slack GitHub followers

In this tutorial we will deploy a monitoring stack to docker swarm, that includes Grafana, Prometheus, Node-Exporter, cAdvisor and Alertmanager.

If you are looking for more information on Prometheus, have a look at my other Prometheus and Monitoring blog posts.

What you will get out of this

Once you deployed the stacks, you will have the following:

  • Access Grafana through Traefik reverse proxy
  • Node-Exporter to expose node level metrics
  • cAdvisor to expose container level metrics
  • Prometheus to scrape the exposed entpoints and ingest it into Prometheus
  • Prometheus for your Timeseries Database
  • Alertmanager for firing alerts on configured rules

The compose file that I will provide will have pre-populated dashboards

Deploy Traefik

Get the traefik stack sources:

1
2
$ git clone https://github.com/bekkerstacks/traefik
$ pushd traefik

Have a look at HTTPS Mode if you want to deploy traefik on HTTPS, as I will use HTTP in this demonstration.

Set your domain and deploy the stack:

1
2
3
4
5
6
7
8
9
10
$ DOMAIN=localhost PROTOCOL=http bash deploy.sh

Username for Traefik UI: demo
Password for Traefik UI: 
deploying traefik stack in http mode
Creating network public
Creating config proxy_traefik_htpasswd
Creating service proxy_traefik
Traefik UI is available at:
- http://traefik.localhost

Your traefik service should be running:

1
2
3
$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
0wga71zbx1pe        proxy_traefik       replicated          1/1                 traefik:1.7.14      *:80->80/tcp

Switch back to the previous directory:

1
$ popd

Deploy the Monitoring Stack

Get the sources:

1
2
$ git clone https://github.com/bekkerstacks/monitoring-cpang
$ pushd monitoring-cpang

If you want to deploy the stack with no pre-configured dashboards, you would need to use ./docker-compose.html, but in this case we will deploy the stack with pre-configured dashboards.

Set the domain and deploy the stack:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ docker stack deploy -c alt_versions/docker-compose_http_with_dashboards.yml mon

Creating network private
Creating config mon_grafana_config_datasource
Creating config mon_grafana_dashboard_prometheus
Creating config mon_grafana_dashboard_docker
Creating config mon_grafana_dashboard_nodes
Creating config mon_grafana_dashboard_blackbox
Creating config mon_alertmanager_config
Creating config mon_prometheus_config
Creating config mon_prometheus_rules
Creating service mon_blackbox-exporter
Creating service mon_alertmanager
Creating service mon_prometheus
Creating service mon_grafana
Creating service mon_cadvisor
Creating service mon_node-exporter

The endpoints is configured as ${service_name}.${DOMAIN} so you will be able to access grafana on http://grafana.localhost as showed in my use-case.

Use docker stack services mon to see if all the tasks has checked into its desired count then access grafana on http://grafana.${DOMAIN}

Accessing Grafana

Access Grafana on http://grafana.${DOMAIN} and logon with the user admin and the password admin:

image

You will be asked to reset the password:

image

You will then be directed to the ui:

image

From the top, when you list dashboards, you will see the 3 dashboards that was pre-configured:

image

When looking at the Swarm Nodes Dashboard:

image

The Swarm Services Dashboard:

image

Exploring Metrics in Prometheus

Access prometheus on http://prometheus.${DOMAIN} and from the search input, you can start exploring though all the metrics that is available in prometheus:

image

If we search for node_load15 and select graph, we can have a quick look on how the 15 minute load average looks like for the node where the stack is running on:

image

Having a look at the alerts section:

image

Resources

For more information and configuration on the stack that we use, have a look at the wiki: - https://github.com/bekkerstacks/monitoring-cpang/wiki

The github repository: - https://github.com/bekkerstacks/monitoring-cpang

Thank You

Let me know what you think. If you liked my content, feel free to checkout my content on ruan.dev or follow me on twitter at @ruanbekker

Comments