Prometheus is one of those awesome open source monitoring services that I simply cannot live without. Prometheus is a Time Series Database that collects metrics from services using it’s exporters functionality. Prometheus has its own query language called PromQL and makes graphing epic visualiztions with services such as Grafana a breeze.
What are we doing today
We will install the prometheus service and set up node_exporter to consume node related metrics such as cpu, memory, io etc that will be scraped by the exporter configuration on prometheus, which then gets pushed into prometheus’s time series database. Which can then be used by services such as Grafana to visualize the data.
Other exporters is also available, such as: haproxy_exporter, blackbox_exporter etc, then you also get pushgateway which is used to push data to, and then your exporter configuration scrapes the data from the pushgateway endpoint. In a later tutorial, we will set up push gateway as well.
First, let’s provision our dedicated system users for prometheus and node exporter:
Let’s look at the status to see if everything works as expected:
$ systemctl status prometheus
prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2019-03-26 11:59:10 UTC; 6s ago
Main PID: 16374 (prometheus)
Tasks: 9 (limit: 4704)
└─16374 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=
Mar 26 11:59:10 ip-172-31-41-126 prometheus: level=info ts=2019-03-26T11:59:10.893770598Z caller=main.go:655 msg="TSDB started"
Seems legit! Enable the service on startup:
$ systemctl enable prometheus
Install Node Exporter
Now since we have prometheus up and running, we can start adding exporters to publish data into our prometheus time series database. As mentioned before, with node exporter, we will allow prometheus to scrape the node exporter endpoint to consume metrics about the node:
You will find the latest version from their website, which I have added at the top of this post.
Once the config is saved, restart prometheus and have a look at the status if everything is going as expected:
$ systemctl restart prometheus
$ systemctl status prometheus
Nginx Reverse Proxy
Let’s add a layer of security and front our setup with a nginx reverse proxy, so that we don’t have to access prometheus on high ports and we have the option to enable basic http authentication. Install nginx:
$ apt install nginx apache2-utils -y
Create the authentication file:
$ htpasswd -c /etc/nginx/.htpasswd admin
Create the nginx site configuration, this will tel nginx to route connections on port 80, to reverse proxy to localhost, port 9090, if authenticated:
$ rm /etc/nginx/sites-enabled/default
$ vim /etc/nginx/sites-enabled/prometheus.conf
I will demonstrate how to use the Vault CLI to interact with Vault. Let’s start by installing the vault cli tools, I am using mac, so I will be using brew:
$ brew install vault
Set environment variables:
$ export VAULT_ADDR='http://127.0.0.1:8200'
Initialize the Vault Cluster:
Initialize new vault cluster with 6 key shares:
$ vault operator init -key-shares=6 -key-threshold=3
Unseal Key 1: RntjR...DQv
Unseal Key 2: 7E1bG...0LL+
Unseal Key 3: AEuhl...A1NO
Unseal Key 4: bZU76...FMGl
Unseal Key 5: DmEjY...n7Hk
Unseal Key 6: pC4pK...XbKb
Initial Root Token: s.F0JGq..98s2U
Vault initialized with 10 key shares and a key threshold of 3. Please
securely distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
In order to unseal the vault cluster, we need to supply it with 3 key shares:
$ vault login s.tdlEqsfzGbePVlke5hTpr9Um
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Using the cli your auth token will be saved locally at ~/.vault-token.
Enable the secret kv engine:
$ vault secrets enable -version=1 -path=secret kv
Create and Read Secrets
Write a secret to the path enabled above:
$ vault kv put secret/my-app/password password=123
List your secrets:
$ vault kv list secret/
Read the secret (defaults in table format):
$ vault kv get secret/my-app/password
The first step of our deploy will invoke a shell script that will establish a ssh tunnel to the docker host, mounting the docker socket to a tcp local port, then exporting the docker host port to the tunneled port, ci/scripts/deploy.sh:
Another nice thing about Drone CI is the “Services” configuration within your pipeline. At times your unit or integration testing steps might be dependent of a database such as MongoDB, MySQL etc.
Drone allows you to spin up a ephemeral database service such as MongoDB using a Docker container as the fist step within your pipeline, defined in the services section. This step will always run first.
The service container will be reachable via the configured container name as its hostname. Keep note that if you run multiple paralel jobs that the service container will only be reachable from the container where the mongodb container is running.
What are we doing today
We will setup a really basic (and a bit useless) pipeline that will spin up a mongodb service container, use a step to write random data to mongodb and a step that reads data from mongodb.
For demonstration purposes, the data is really random but more focused on the service section.
First we define our service, mongodb. Once the mongodb service is running, we will have our build step, our step that runs the mongodb version against our database, write data into our mongodb database, then read the data from mongodb, then the last step running a shell command with the date.
---kind:pipelinename:mongotestsservices:-name:mongoimage:mongo:4command:[--smallfiles]ports:-27017steps:-name:build-stepimage:alpinecommands:-echo "this should be a step that does something"-name:mongodb-return-versionimage:mongo:4commands:-date-mongo --host mongo --eval "db.version()"-name:mongodb-test-writesimage:mongo:4commands:-date-sh scripts/write_mongo.sh-name:mongodb-test-readsimage:mongo:4commands:-date-sh scripts/read_mongo.sh-name:last-stepimage:alpinecommands:-echo "completed at $(date)"
Our scripts referenced in our steps:
The first will be our script that write random data into mongodb, scripts/write_mongo.sh:
Some time ago, I assisted a customer who was trying to do a select count(*) via beeline and failed with:
[hadoop@ip-10-10-9-226 ~]$ beeline -u jdbc:hive2://nn-emr.sysint.dxone.local:10000/default --silent=true --outputformat=csv2 -e "select count(*) from basetables_rms.rms_site"
19/04/26 06:41:15 [main]: WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not support autoCommit=false.
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)
When reproducing this I found a jira: https://issues.apache.org/jira/browse/HIVE-14631 which related to the same issue and the workaround was to switch your execution engine to mapreduce. By doing that, it worked, but wanted a better resolution for the customer.
When setting enabling debugging, I found that the error is related to permissions:
$ beeline -u jdbc:hive2://172.31.31.247:10000/default --silent=false --outputformat=csv2 -e "select count(*) from testdb.users"
Connecting to jdbc:hive2://172.31.31.247:10000/default
Connected to: Apache Hive (version 2.1.1-amzn-0)
Driver: Hive JDBC (version 2.1.1-amzn-0)
19/04/26 10:24:01 [main]: WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not support autoCommit=false.
ERROR : Failed to execute tez graph.
org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=WRITE, inode="/user/anonymous":hdfs:hadoop:drwxr-xr-x
So it seems that when the client (anonymous) is trying to copy the hive execution jar to is home path in HDFS, in this case (/home/anonymous/.hiveJars/) it fails due to permissions.
By passing the hadoop user, I was able to get the expected results:
$ beeline -n hadoop -u jdbc:hive2://172.31.31.247:10000/default --silent=false --outputformat=csv2 -e "select count(*) from testdb.users"
INFO : Completed executing command(queryId=hive_20190426103246_33253d86-3ebc-462f-a5a1-f01877dd00a8); Time taken: 17.08 seconds
INFO : OK
1 row selected (17.282 seconds)
* Whenever a push to master is receive on github, the pipeline will be triggered
* The content from our github repository will be cloned to the agent on a container
* Jekyll will build and the output will be transferred to docker swarm using rsync
* The docker-compose.yml will be transferred to the docker swarm host using scp
* A docker stack deploy is ran via ssh
Install Jekyll Locally
Install Jekyll locally, as we will use it to create the initial site. I am using a mac, so I will be using brew. For other operating systems, have a look at this post.
I will be demonstrating with a weightloss blog as an example.
$ brew install jekyll
Go ahead and create a new site which will host the data for your jekyll site:
$ jekyll new blog-weightloss
Create a Github Repository
First we need to create an empty github repository, in my example it was github.com/ruanbekker/blog-weightloss.git. Once you create the repo change into the directory created by the jekyll new command:
$ cd blog-weightloss
Now initialize git, set the remote, add the jekyll data and push to github:
You should see your data on your github repository.
Create Secrets on Drone
Logon to the Drone UI, sync repositories, activate the new repository and head over to settings where you will find the secrets section.
Add the following secrets:
Secret Name: swarm_host
Secret Value: ip address of your swarm
Secret Name: swarm_key
Secret Value: contents of your private ssh key
Secret Name: swarm_user
Secret Value: the user that is allowed to ssh
You should see the following:
Add the Drone Config
Drone looks from a .drone.yml file in the root directory for instructions on how to do its tasks. Lets go ahead and declare our pipeline:
Now we need to push our .drone.yml and docker-compose.yml to github. Since the repository is activated on drone, any push to master will trigger the pipeline, so after this push we should go to drone to look at our pipeline running.