Ruan Bekker's Blog

From a Curious mind to Posts on Github

How to Setup a Redis Exporter for Prometheus

In this tutorial we will visualize our Redis Cluster’s Metrics with Grafana. In order to do that we will setup a redis exporter which will authenticate with redis and then configure prometheus to scrape the endpoint of the redis exporter’s http endpoint to write the time series data to prometheus.

Install Golang

We need to build a binary from the redis exporter project, and we need a Golang environment. If you don’t have golang installed already:

1
2
3
4
5
6
$ cd /tmp/
$ wget https://dl.google.com/go/go1.14.2.linux-amd64.tar.gz
$ tar -xf go1.14.2.linux-amd64.tar.gz -C /usr/local
$ mkdir -p $HOME/go/{bin,src,pkg}
$ export GOPATH=/go
$ export PATH=${PATH}:${GOPATH}/bin:/usr/local/go/bin

You should now be able to get a response:

1
2
$ go version
go version go1.14.2 linux/amd64

Redis Exporter

Get the source code and build the binary:

1
2
3
$ git clone https://github.com/oliver006/redis_exporter.git
$ cd redis_exporter
$ go build .

Now the binary should be built, and you should be able to get a response when running the following:

1
$ ./redis_exporter --help

Copy the binary the the following path:

1
$ cp redis_exporter /usr/bin/

Then create the systemd unit file, in /etc/systemd/system/redis_exporter.service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[Unit]
Description=Redis Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/bin/redis_exporter \
    -web.listen-address ":9121" \
    -redis.addr "redis://ip.of.redis.server:6379" \
    -redis.password "your-strong-redis-password"

[Install]
WantedBy=multi-user.target

Reload systemd:

1
$ systemctl daemon-relaod

Then start the redis exporter:

1
$ systemctl restart redis_exporter

Now you should be able to get redis metrics when you hit the redis exporter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ curl http://127.0.0.1:9121/metrics
...
# TYPE redis_commands_duration_seconds_total counter
redis_commands_duration_seconds_total{cmd="auth"} 0.000308
redis_commands_duration_seconds_total{cmd="client"} 0.000251
redis_commands_duration_seconds_total{cmd="config"} 0.010594
redis_commands_duration_seconds_total{cmd="evalsha"} 229.214873
redis_commands_duration_seconds_total{cmd="get"} 0.002343
redis_commands_duration_seconds_total{cmd="info"} 0.013722
redis_commands_duration_seconds_total{cmd="latency"} 0.000557
redis_commands_duration_seconds_total{cmd="lrange"} 11.102069
redis_commands_duration_seconds_total{cmd="ltrim"} 3.731263
redis_commands_duration_seconds_total{cmd="ping"} 2e-05
redis_commands_duration_seconds_total{cmd="rpush"} 3.460981
redis_commands_duration_seconds_total{cmd="script"} 0.008393
redis_commands_duration_seconds_total{cmd="set"} 0.001329
redis_commands_duration_seconds_total{cmd="slowlog"} 0.001308
...

Configure Prometheus

If you don’t have prometheus setup, you can view this blogpost to get it setup.

Then configure your prometheus.yml and add the target to scrape the redis exporter endpoint to write the time series data into prometheus:

1
2
3
4
scrape_configs:
  - job_name: redis_exporter
    static_configs:
    - targets: ['ip.of.redis.exporter:9121']

Then restart prometheus, if you have docker redeploy your stack or prometheus container. For prometheus as a service you can use systemctl restart prometheus, depending on your operating system distribution.

Grafana

Head over to Grafana, if you don’t have Grafana, you can view this post to install Grafana.

Then import the dashboard 763 and after some time, you should see a dashboard more or less like this:

image

Nginx Analysis Dashboard Using Grafana and Elasticsearch

In this post we will be setting up a analytical dashboard using grafana to visualize our nginx access logs.

grafana-nginx-elasticsearch-prometheus

In this tutorial I will be using my other blog sysadmins.co.za which is being served on nginx. We will also be setting up the other components such as filebeat, logstash, elasticsearch and redis, which require if you would like to follow along.

The End Result

We will be able to analyze our Nginx Access logs to answer questions such as:

  • Whats the Top 10 Countries accessing your website in the last 24 hours
  • Who’s the Top 10 Referers?
  • Whats the most popular page for the past 24 hours?
  • How does the percentage of 200’s vs 404’s look like?
  • Ability to view results based on status code
  • Everyone loves a World Map to view hotspots

At the end of the tutorial, your dashboard will look similar to this:

grafana-elasticsearch-nginx-dashboard

High Level Overview

Our infrastructure will require Nginx with Filebeat, Redis, Logstash, Elasticsearch and Grafana and will look like this:

grafana-elasticsearch-logs-setup

I will drill down how everything is connected:

  1. Nginx has a custom log_format that we define, that will write to /var/log/nginx/access_json.log, which will be picked up by Filebeat as a input.
  2. and Filebeat has an output that pushes the data to Redis
  3. Logstash is configured with Redis as an input with configured filter section to transform the data and outputs to Elasticsearch
  4. From Grafana we have a configured Elasticsearch datasource
  5. Use the grafana template to build this awesome dashboard on Grafana

But first, a massive thank you to akiraka for templatizing this dashboard and made it available on grafana

Let’s build all the things

I will be using LXD to run my system/server containers (running ubuntu 18), but you can use a vps, cloud instance, multipass, virtualbox, or anything to host your servers that we will be deploying redis, logstash, etc.

Servers provisioned for this setup:

  • Nginx
  • Redis
  • Logstash
  • Elasticsearch
  • Grafana
  • Prometheus

Elasticsearch

If you don’t have a cluster running already, you can follow this tutorial which will help you deploy a HA Elasticsearch Cluster, or if you prefer docker, you can follow this tutorial

Redis

For our in-memory data store, I will be securing my redis installation with a password as well.

Install redis:

1
$ apt update && apt install redis-server -y

Generate a password:

1
2
$ openssl rand -base64 36
9V5YlWvm8WuC4n1KZLYUEbLruLJLNJEnDzhu4WnAIfgxMmlv

In your redis config /etc/redis/redis.conf, you need to change the following:

1
2
3
4
5
6
7
...
bind 0.0.0.0
port 6379
daemonize yes
supervised systemd
requirepass 9V5YlWvm8WuC4n1KZLYUEbLruLJLNJEnDzhu4WnAIfgxMmlv
...

Restart redis to activate your changes:

1
$ systemctl restart redis.service

and then set and get a key using your password:

1
2
3
$ redis-cli -a "9V5YlWvm8WuC4n1KZLYUEbLruLJLNJEnDzhu4WnAIfgxMmlv" set test ok
$ redis-cli -a "9V5YlWvm8WuC4n1KZLYUEbLruLJLNJEnDzhu4WnAIfgxMmlv" get test
ok

Logstash

On the logstash server, install the requirements:

1
2
3
$ apt update && apt install wget apt-transport-https default-jre -y
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list

Now the repository for elastic is setup now we need to update and install logstash:

1
$ apt update && apt install logstash -y

Once logstash is installed, we need to provide logstash with a configuration, in our scenario we will have a input for redis, a filter section to transform and output as elasticsearch.

Just make sure of the following:

  • Populate the connection details of redis (we will define the key in filebeat later)
  • Ensure that GeoLite2-City.mmdb is in the path that I have under filter
  • Populate the connectiond details of Elasticsearch and choose a suitable index name, we will need to provide that index name in Grafana later

Create the config: /etc/logstash/conf.d/logs.conf and my config will look like the following. (config source)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
input {
  redis {
    data_type =>"list"
    key =>"nginx_logs"
    host =>"10.47.127.37"
    port => 6379
    password => "9V5YlWvm8WuC4n1KZLYUEbLruLJLNJEnDzhu4WnAIfgxMmlv"
    db => 0
  }
}

filter {
  geoip {
    target => "geoip"
    source => "client_ip"
    database => "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"
    add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
    add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
    remove_field => ["[geoip][latitude]", "[geoip][longitude]", "[geoip][country_code]", "[geoip][country_code2]", "[geoip][country_code3]", "[geoip][timezone]", "[geoip][continent_code]", "[geoip][region_code]"]
  }
  mutate {
    convert => [ "size", "integer" ]
    convert => [ "status", "integer" ]
    convert => [ "responsetime", "float" ]
    convert => [ "upstreamtime", "float" ]
    convert => [ "[geoip][coordinates]", "float" ]
    remove_field => [ "ecs","agent","host","cloud","@version","input","logs_type" ]
  }
  useragent {
    source => "http_user_agent"
    target => "ua"
    remove_field => [ "[ua][minor]","[ua][major]","[ua][build]","[ua][patch]","[ua][os_minor]","[ua][os_major]" ]
  }
}
output {
  elasticsearch {
    hosts => ["10.47.127.132", "10.47.127.199", "10.47.127.130"]
    #user => "myusername"
    #password => "mypassword"
    index => "logstash-nginx-sysadmins-%{+YYYY.MM.dd}"
  }
}

Nginx

On our nginx server we will install nginx and filebeat, then configure nginx to log to a custom log format, and configure filebeat to read the logs and push it to redis.

Installing nginx:

1
$ apt update && apt install nginx -y

Installing filebeat:

1
2
$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-amd64.deb
$ dpkg -i filebeat-7.6.2-amd64.deb

Next we will configure nginx to log to a seperate file with a custom log format to include data such as the, request method, upstream response time, hostname, remote address, etc.

Under the http directive in your /etc/nginx/nginx.conf, configure the log_format and access_log:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
http {
...
        log_format json_logs '{"@timestamp":"$time_iso8601","host":"$hostname",'
                            '"server_ip":"$server_addr","client_ip":"$remote_addr",'
                            '"xff":"$http_x_forwarded_for","domain":"$host",'
                            '"url":"$uri","referer":"$http_referer",'
                            '"args":"$args","upstreamtime":"$upstream_response_time",'
                            '"responsetime":"$request_time","request_method":"$request_method",'
                            '"status":"$status","size":"$body_bytes_sent",'
                            '"request_body":"$request_body","request_length":"$request_length",'
                            '"protocol":"$server_protocol","upstreamhost":"$upstream_addr",'
                            '"file_dir":"$request_filename","http_user_agent":"$http_user_agent"'
                            '}';

        access_log  /var/log/nginx/access_json.log  json_logs;
...
}

Restart nginx to activate the changes:

1
$ systemctl restart nginx

Next we need to configure filebeat to read from our nginx access logs and configure the output to redis. Edit the filebeat config:

1
$ vim /etc/filebeat/filebeat.yml

And configure filebeat with the following and make sure to change the values where you need to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# config source: akiraka.net
# filebeat input 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access_json.log
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true

# filebeat modules 
filebeat.config.modules:
  # remove the escape character before the wildcard below
  path: ${path.config}/modules.d/\*.yml
  reload.enabled: false

# elasticsearch template settings
setup.template.settings:
  index.number_of_shards: 3

# redis output
output.redis:
  hosts: ["10.47.127.140:6379"]
  password: "9V5YlWvm8WuC4n1KZLYUEbLruLJLNJEnDzhu4WnAIfgxMmlv"
  key: "nginx_logs"
  # ^ this key needs to be the same as the configured key on logstash 
  db: 0
  timeout: 5

Restart filebeat:

1
$ systemctl restart filebeat

When you make a request to your nginx server, you should see a similar logline like below:

1
2
$ tail -n1 /var/log/nginx/access_elg.log
{"@timestamp":"2020-04-28T20:05:03+00:00","host":"sysadmins-blog","server_ip":"10.68.100.89","client_ip":"x.x.x.x","xff":"x.x.x.x","domain":"sysadmins.co.za","url":"/","referer":"-","args":"-","upstreamtime":"0.310","responsetime":"0.312","request_method":"GET","status":"200","size":"4453","request_body":"-","request_length":"519","protocol":"HTTP/1.1","upstreamhost":"127.0.0.1:2369","file_dir":"/var/www/web/root/","http_user_agent":"Mozilla/5.0"}

Grafana

On the grafana server, install grafana:

1
2
3
4
$ apt update && apt install apt-transport-https software-properties-common wget -y
$ wget -q -O - https://packages.grafana.com/gpg.key | apt-key add -
$ add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
$ apt update && apt install grafana -y

Now we need to install a couple of grafana plugins that we require for our dashboards:

1
2
3
$ grafana-cli plugins install grafana-worldmap-panel
$ grafana-cli plugins install grafana-clock-panel
$ grafana-cli plugins install grafana-piechart-panel

Now reload systemd and restart grafana:

1
2
$ systemctl daemon-reload
$ systemctl restart grafana-server

If you would like to setup nginx as a reverse proxy to grafana, you can have a look at this blogpost on how to do that.

Prometheus

If you don’t have Prometheus installed already, you can view my blogpost on setting up Prometheus.

Verifying

To verify if everything works as expected, make a request to your nginx server, then have a look if your index count on elasticsearch increases:

1
2
3
$ curl http://elasticsearch-endpoint-address:9200/_cat/indices/logstash-*?v
health status index                               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logstash-nginx-x-2020.04.28 SWbHCer-TeOcw6bi_695Xw   5   1      58279            0     32.6mb         16.3mb

If you dont, make sure that all the processes are running on the servers, and that each server is able to reach each other on the targeted ports.

The Fun Part: Dashboarding

Now that we have everything in place, the fun part is to build the dashboards, first we need to configure elasticsearch as our datasource and specify the index we want to read from. Open grafana on http://ip.of.grafana.server:3000, default user and password is admin.

Select config on the left and select datasources, add a datasource, select elasticsearch and specify your datasource name, mine is es-nginx in this example, the url of your elasticsearch endpoint, if you have secured your elasticsearch cluster with authentication, provide the auth, then provide your index name as as provided in logstash.

My configured index will look like logstash-nginx-sysadmins-YYYY-MM-dd, therefore I specified index name as logstash-nginx-sysadmins-* and my timefield as @timestamp, the version, and select save and test, which would look like this:

AC025E20-38D0-4676-B576-9F5932913BA1

Now we will import our dashboard template (Once again a massive thank you to Shenxiang, Qingkong and Ruixi which made this template available!), head over to dashboards and select import, then provide the ID: 11190, after that it will prompt what your dashboard needs to be named and you need to select your Elasticsearch and Prometheus datasource.

The description of the panels is in Chinese, if you would like it in english, I have translated mine to english and made the dashboard json available in this gist

Tour of our Dashboard Panels

Looking at our hotspot map:

grafana

The summary and top 10 pages:

76E8CBE1-4B03-4226-8041-B98879BAD66A

Page views, historical trends:

grafana-page-views

Top 10 referers and table data of our logs:

B17C4F55-DF91-4EA0-9669-C237FF560459

Thank You

I hope this was useful, if you have any issues with this feel free to reach out to me. If you like my work, please feel free to share this post, follow me on Twitter at @ruanbekker or visit me on my website

ko-fi

How to Do Port Forwarding With Iptables

This is a quick post on how to do port forwarding with iptables on linux.

What would we like to achieve

We have a lxc container running a redis server and we would like to do port forwarding so that we can reach the server over the internet

LXC Host

On our host that hosts our lxc containers, we want to forward the host port 5379 to 6379 of the container (10.37.117.37), so we can connect on a non-standard redis port:

1
$ iptables -t nat -I PREROUTING -p tcp --dport 5379 -j DNAT --to-destination 10.37.117.37:6379

Test over the Internet

Test the connection by connecting to the LXC Host’s IP:

1
2
$ redis-cli -h lxc.host.ip.address -p 5379 -a "${REDIS_PW}"  get test
"It's working!"

Thank You

Thanks for reading my short post on how to use iptables to do port forwarding.

How to Set a Static IP in Ubuntu 18

This is a short post on how to set a static ip address on ubuntu 18.04 using netplan

Netplan

At the moment my network interfaces uses dhcp, and we can see that below:

1
2
3
4
5
6
$ cat /etc/netplan/50-cloud-init.yaml
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true

Changing the configuration to static:

1
2
3
4
5
6
7
8
9
10
$ cat /etc/netplan/50-cloud-init.yaml
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: false
            addresses: [10.37.117.37/24]
            gateway4: 10.37.117.1
            nameservers:
                addresses: [127.0.0.53,8.8.8.8]

After changing the configuration, you need to apply your changes:

1
$ netplan apply

Thank You

Thank you for reading my short post on how to change a static ip address on ubuntu 18.04 using netplan

Graphing Covid-19 Stats With Grafana and Elasticsearch Using Python

coronavirus-covid19-grafana-metrics

I stumbled upon a github repository that stores time-series data in json format of corona virus / covid19 statistics, which get updated daily.

I was curious to see data about my country and want to see how metrics will look like after our lockdown started, so I decided to consume that data with Python and the requests library, then ingest data about covid19 into Elasticsearch and the visualize the data with Grafana.

Sample of the Data

Let’s have a peek at the data to determine how we will use it to write to Elasticsearch. Let’s consume the data with python:

1
2
3
>>> import requests
>>> import json
>>> response = requests.get('https://pomber.github.io/covid19/timeseries.json').json()

Now let’s determine the data type:

1
2
>>> type(response)
<type 'dict'>

Now as it’s a dictionary, let’s look at they keys:

1
2
>>> response.keys()
[u'Canada', u'Sao Tome and Principe', u'Lithuania', u'Cambodia', u'Ethiopia',....

So let’s take a look how the data looks like if we do a lookup for Canada:

1
2
>>> type(response['Canada'])
<type 'list'>

As we can see it’s a list, we can count how many items is in our list:

1
2
>>> len(response['Canada'])
94

Now let’s peek at the data by accessing our first index of our list:

1
2
>>> response['Canada'][0]
{u'date': u'2020-1-22', u'confirmed': 0, u'recovered': 0, u'deaths': 0}

So our data will look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
  [
    'Country Name': [
      {
        'date': '<string>', 
        'confirmed': '<int>', 
        'recovered': '<int>', 
        'deaths': '<int>'
      },
      {
        'date': '<string>',
        'confirmed': '<int>',
        'recovered': '<int>',
        'deaths': '<int>'
      },
    ],
    'Country Name': [
      ...
    ]
  ]
}

Some issues we need to fix

As you can see the date is displayed as 2020-1-22 instead of 2020-01-22, I want to make it consistent as I will be ingesting the data with a @timestamp key which we will use the date from the returned data. So first we will need to convert that before we ingest the data.

The other thing I was thinking of is that, if for some reason we need to ingest this data again, we dont want to sit with duplicates (same document with different _id’s), so for that I decided to generate a hash value that consist of the date and the country, so if the script run to ingest the data, it will use the same id for the specific document, which would just overwrite it, therefore we won’t sit with duplicates.

So the idea is to ingest a document to elasticsearch like this:

1
2
3
4
5
6
7
8
9
doc = {
    "_id": "sha_hash_value",
    "day": "2020-01-22",
    "timestamp": "@2020-01-22 00:00:00",
    "country": "CountryName",
    "confirmed": 0,
    "recovered": 0,
    "deaths": 0
}

How we will ingest the data

The first run will load all the data and ingest all the data up to the current day to elasticsearch. Once that is done, we will add code to our script to only ingest the most recent day’s data into elasticsearch, which we will control with a cronjob.

Create a index with a mapping to let Elasticsearch know timestamp will be a date field:

1
2
3
$ curl -XPUT -H 'Content-Type: application/json' \
  -u username:pass 'https://es.domain.com/coronastats' -d \
  '{"mappings": {"foo1": {"properties": {"timestamp" : {"type" : "date","format" : "yyyy-MM-dd HH:mm:ss"}}}}}'

Once our index is created, create the python script that will load the data, loop through each country’s daily data and ingest it into elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
#!/usr/bin/python
import requests
import datetime as dt
import json
import hashlib

url = 'https://pomber.github.io/covid19/timeseries.json'
elasticsearch_url = "https://es.domain.com"
elasticsearch_username = ""
elasticsearch_password = ""

api_response = requests.get(url).json()

def convert_datestamp(day):
    return str(dt.datetime.strptime(day, '%Y-%m-%d'))

def hash_function(country, date):
    string_to_hash = country + date
    hash_obj  = hashlib.sha1(string_to_hash.encode('utf-8'))
    hash_value = hash_obj.hexdigest()
    return hash_value

def map_es_doc(payload, country):
    doc = {
        "day": payload['date'],
        "timestamp": convert_datestamp(payload['date']),
        "country": country,
        "confirmed": payload['confirmed'],
        "recovered": payload['recovered'],
        "deaths": payload['deaths']
    }
    return doc

def ingest(doc_id, payload):
    response = requests.put(
        elasticsearch_url + '/coronastats/coronastats/' + doc_id,
        auth=(elasticsearch_username, elasticsearch_password),
        headers={'content-type': 'application/json'},
        json=payload
    )
    return response.status_code

for country in api_response.keys():
    try:
        for each_payload in api_response[country]:
            doc_id = hash_function(country, each_payload['date'])
            doc = map_es_doc(each_payload, country)
            response = ingest(doc_id, doc)
            print(response)
    except Exception as e:
        print(e)

Run the script to ingest all the data into elasticsearch. Now we will create the script that will run daily to only ingest the previous day’s data, so that we only ingest the latest data and not all the data from scratch again.

I will create this file in /opt/scripts/corona_covid19_ingest.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#!/usr/bin/python
import requests
import datetime as dt
import json
import hashlib

url = 'https://pomber.github.io/covid19/timeseries.json'
elasticsearch_url = "https://es.domain.com"
elasticsearch_username = ""
elasticsearch_password = ""

api_response = requests.get(url).json()

yesterdays_date = dt.date.today() - dt.timedelta(days=1)

def convert_datestamp(day):
    return str(dt.datetime.strptime(day, '%Y-%m-%d'))

def hash_function(country, date):
    string_to_hash = country + date
    hash_obj  = hashlib.sha1(string_to_hash.encode('utf-8'))
    hash_value = hash_obj.hexdigest()
    return hash_value

def map_es_doc(payload, country):
    doc = {
        "day": payload['date'],
        "timestamp": convert_datestamp(payload['date']),
        "country": country,
        "confirmed": payload['confirmed'],
        "recovered": payload['recovered'],
        "deaths": payload['deaths']
    }
    return doc

def ingest(doc_id, payload):
    response = requests.put(
        elasticsearch_url + '/coronastats/coronastats/' + doc_id,
        auth=(elasticsearch_username, elasticsearch_password),
        headers={'content-type': 'application/json'},
        json=payload
    )
    return response.status_code

for country in api_response.keys():
    try:
        for each_payload in api_response[country]:
            if convert_datestamp(each_payload['date']).split()[0] == str(yesterdays_date):
                print("ingesting latest data for {country}".format(country=country))
                doc_id = hash_function(country, each_payload['date'])
                doc = map_es_doc(each_payload, country)
                response = ingest(doc_id, doc)
                print(response)
    except Exception as e:
        print(e)

The only difference with this script is that it checks if the date is equals to yesterday’s date, and if so the document will be prepared and ingested into elasticsearch. We will create a cronjob that runs this script every morning at 08:45.

First make the file executable:

1
$ chmod +x /opt/scripts/corona_covid19_ingest.py

Run crontab -e and add the following

1
45 8 * * * /opt/scripts/corona_covid19_ingest.py

Visualize the Data with Grafana

We will create this dashboard:

corona-covid-19-dashboard

We need a elasticsearch datasource that points to the index that we ingest our data into. Head over to datasources, add a elasticsearch datasource and set the index to coronastats and add the timefield as timestamp.

We want to make the dashboard dynamic to have a “country” dropdown selector, for that go to the dashboard settings, select variable and add a country variable:

covid19-dashboard-variables

First panel: “Reported Cases per Day”:

covid19-reported-cases

Second panel: “Confirmed Cases”:

covid19-confirmed-cases

Third panel: “Recovered Cases”:

covid19-recovered-cases

Now, if we select Italy, Spain and France as an example, we will see something like this:

covid19-country-stats

Thank You

Although its pretty cool visualizing data, the issue that we are in at the moment with coronavirus / covid19 is really scary and we should all do our part to try and stay home, sanitize and try not to spread the virus. Together we can all do great things by reducing the spread of this virus.

Stay safe everyone.

Nginx Metrics on Prometheus With the Nginx Log Exporter

In this post we will setup a nginx log exporter for prometeus to get metrics of our nginx web server, such as number of requests per method, status code, processed bytes etc. Then we will configure prometheus to scrape our nginx metric endpoint and also create a basic dashbaord to visualize our data.

If you follow along on this tutorial, it assumes that you have Prometheus and Grafana up and running. But if not the embedded links will take you to the blog posts to set it up.

Nginx Webserver

Install nginx:

1
2
$ apt update
$ apt install nginx -y

Configure your nginx server’s log format to match the nginx log exporter’s expected format, we will name it custom:

1
2
3
  log_format custom   '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" "$http_x_forwarded_for"';

Edit your main nginx config:

1
$ vim /etc/nginx/nginx.conf

This is how my complete config looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
user www-data;
worker_processes auto;
pid /run/nginx.pid;
# remote the escape char if you are going to use this config
include /etc/nginx/modules-enabled/\*.conf;

events {
  worker_connections 768;
}

http {

  # basic config
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;
  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  # ssl config
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2; 
  ssl_prefer_server_ciphers on;

  # logging config
  log_format custom   '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" "$http_x_forwarded_for"';

  access_log /var/log/nginx/access.log custom;
  error_log /var/log/nginx/error.log;

  # gzip
  gzip on;

  # virtual host config
  include /etc/nginx/conf.d/myapp.conf;

}

I will delete the default host config:

1
$ rm -rf /etc/nginx/sites-enabled/default

And then create my /etc/nginx/conf.d/myapp.conf as referenced in my main config, with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
server {

  listen 80 default_server;
  # remove the escape char if you are going to use this config
  server_name \_;

  root /var/www/html;
  index index.html index.htm index.nginx-debian.html;

  location / {
    try_files $uri $uri/ =404;
  }

}

When you make a GET request to your server, you should see something like this in your access log:

1
10x.1x.2x.1x - - [25/Apr/2020:00:31:11 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15" "-"

Nginx Log Exporter

Head over to the prometheus-nginxlog-exporter releases page and get the latest version, in the time of writing it is v1.4.0:

1
$ wget https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/download/v1.4.0/prometheus-nginxlog-exporter

Make it executable and move it to your path:

1
2
$ chmod +x prometheus-nginxlog-exporter
$ mv prometheus-nginxlog-exporter /usr/bin/prometheus-nginxlog-exporter

Create the directory where we will place our config for our exporter:

1
$ mkdir /etc/prometheus

Create the config file:

1
$ vim /etc/prometheus/nginxlog_exporter.yml

You can follow the instructions from github.com/prometheus-nginxlog-exporter for more information on configuration, but I will be using the following config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
listen:
  port: 4040
  address: "0.0.0.0"

consul:
  enable: false

namespaces:
  - name: myapp
    format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""
    source:
      files:
        - /var/log/nginx/access.log
    labels:
      service: "myapp"
      environment: "production"
      hostname: "myapp.example.com"
    histogram_buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]

Create the systemd unit file:

1
$ vim /etc/systemd/system/nginxlog_exporter.service

And my configuration that I will be using:

1
2
3
4
5
6
7
8
9
10
11
12
13
[Unit]
Description=Prometheus Log Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/bin/prometheus-nginxlog-exporter -config-file /etc/prometheus/nginxlog_exporter.yml

[Install]
WantedBy=multi-user.target

Reload systemd and enable the service on boot:

1
2
$ systemctl daemon-reload
$ systemctl enable nginxlog_exporter

Restart the service:

1
$ systemctl restart nginxlog_exporter

Ensure that the service is running:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ systemctl status nginxlog_exporter

● nginxlog_exporter.service - Prometheus Log Exporter
   Loaded: loaded (/etc/systemd/system/nginxlog_exporter.service; disabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-04-25 00:50:06 UTC; 5s ago
 Main PID: 4561 (prometheus-ngin)
    Tasks: 7 (limit: 2317)
   CGroup: /system.slice/nginxlog_exporter.service
           └─4561 /usr/bin/prometheus-nginxlog-exporter -config-file /etc/prometheus/nginxlog_exporter.yml

Apr 25 00:50:06 nginx-log-exporter systemd[1]: Started Prometheus Log Exporter.
Apr 25 00:50:06 nginx-log-exporter prometheus-nginxlog-exporter[4561]: loading configuration file /etc/prometheus/nginxlog_exporter.yml
Apr 25 00:50:06 nginx-log-exporter prometheus-nginxlog-exporter[4561]: using configuration {Listen:{Port:4040 Address:0.0.0.0} Consul:{Enable:false Address: Datacenter: Scheme: Toke
Apr 25 00:50:06 nginx-log-exporter prometheus-nginxlog-exporter[4561]: starting listener for namespace myapp
Apr 25 00:50:06 nginx-log-exporter prometheus-nginxlog-exporter[4561]: running HTTP server on address 0.0.0.0:4040
Apr 25 00:50:06 nginx-log-exporter prometheus-nginxlog-exporter[4561]: 2020/04/25 00:50:06 Seeked /var/log/nginx/access.log - &{Offset:0 Whence:2}

Test the exporter

Make a couple of requests against your webserver:

1
$ for each in {1..10}; do curl http://78.141.211.49 ; done

So prometheus will now scrape the exporter http endpoint (:4040/metrics) and push the returned values into prometheus. But to get a feel on how the metrics look like, make a request to the metrics endpoint:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ curl http://localhost:4040/metrics
...
# HELP myapp_http_response_count_total Amount of processed HTTP requests
# TYPE myapp_http_response_count_total counter
myapp_http_response_count_total{environment="production",hostname="myapp.example.com",method="GET",service="myapp",status="200"} 10
myapp_http_response_count_total{environment="production",hostname="myapp.example.com",method="POST",service="myapp",status="404"} 1
# HELP myapp_http_response_size_bytes Total amount of transferred bytes
# TYPE myapp_http_response_size_bytes counter
myapp_http_response_size_bytes{environment="production",hostname="myapp.example.com",method="GET",service="myapp",status="200"} 6120
myapp_http_response_size_bytes{environment="production",hostname="myapp.example.com",method="POST",service="myapp",status="404"} 152
# HELP myapp_parse_errors_total Total number of log file lines that could not be parsed
# TYPE myapp_parse_errors_total counter
myapp_parse_errors_total 0
...

As you can see we are getting metrics such as response count total, response size, errors, etc.

Configure Prometheus

Let’s configure prometheus to scrape this endpoint. Head over to your prometheus instance, and edit your prometheus config:

1
$ vim /etc/prometheus/prometheus.yml

Note that in my config I have 2 endpoints that I am scraping, the prometheus endpoint which exists and I will be adding the nginx endpoint, so in full, this is how my config will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'nginx'
    scrape_interval: 15s
    static_configs:
      - targets: ['ip.of.nginx.exporter:4040']

Restart prometheus:

1
$ systemctl restart prometheus

To verify that the exporter is working as expected, head over to your prometheus ui on port 9090, and query up{} to see if your exporters are returning 1:

image

We can then query prometheus with myapp_http_response_count_total{service="myapp"} to see the response counts:

image

Dashboarding in Grafana

If you don’t have Grafana installed, you can look at my Grafana Installation post to get that up and running.

If you have not created the Prometheus datasource, on Grafana, head over to the configuration section on your left, select Datasources, add a Prometheus datasource and add the following (this is assuming grafana runs on the prometheus node - which is fine for testing):

image

Create a new dashboard and add a new panel:

image

Let’s query our data to show us HTTP Method and Status code per 30s: rate(myapp_http_response_count_total{service="myapp"}[$__interval])

image

Thank You

Hope you found this helpful, if you haven’t seen my other posts on Prometheus, have a look at the following:

IPSec Site to Site VPN With Dynamic IPs With Openswan

In this tutorial we will setup a site to site ipsec vpn with strongswan and we will enable each server to discover the other vpn server via dynamic dns. We will also append to our config the ability of roadwarriors so that you will be able to connect to your homelab from any mobile or laptop device from any remote source.

Some background

Me and one of my friends decided to build a site to site vpn with strongswan so that our homelabs could be reachable to each other over private networks.

One challenge that I thought of is that both of our internet providers don’t support static ip addressing, so each vpn server needs to know where to connect to whenever the ip address changes.

What we will be doing

We will setup strongswan vpn on both servers and allow the private LAN ranges to be reachable for both sides. As I have a domain hosted on cloudflare, I will be using cloudflare’s api to update the A record of each sides dns whenever the IP changes.

Environment

On my side, which I will be referring to as Side-A:

1
2
3
Public DNS Name: side-a.example.com
Private Range: 192.168.0.0/24
VPN Server IP: 192.168.0.2

On my friend’s side, which I will be referring to as Side-B:

1
2
3
Public DNS Name: side-b.example.com
Private Range: 192.168.1.0/24
VPN Server IP: 192.168.1.2

Cloudflare Dynamic DNS

You don’t need to use Cloudflare, theres services such as dyndns.com, no-ip.com. But for this tutorial I will be using cloudflare to utilize my own domain.

I will be using the cloudflare-ddns-client

First we need to create a API Token, head over to your dashboard: dash.cloudflare.com, head over to “my profile”, select “API Tokens”, then allow “Read Zones” and “Edit DNS”, then select “Create Token”. Keep the returned token value in a safe place.

Install the pre-requirements:

1
$ apt install python python-dev python-pip make curl build-essential -y

Get the source and install:

1
2
3
$ git clone https://github.com/LINKIWI/cloudflare-ddns-client.git
$ cd cloudflare-ddns-client
$ make install

We will now configure the cloudflare dynamic dns client, this will be done on both sides, but will only demonstrate for side-a:

1
2
3
4
5
6
7
8
$ cloudflare-ddns --configure
Use API token or API key to authenticate?
Choose [T]oken or [K]ey: T
Enter the API token you created at https://dash.cloudflare.com/profile/api-tokens.
Required permissions are READ Account.Access: Organizations, Identity Providers, and Groups; READ Zone.Zone; EDIT Zone.DNS
CloudFlare API token: [redacted]
Enter the domains for which you would like to automatically update the DNS records, delimited by a single comma.
Comma-delimited domains: side-a.example.com

Testing it out to ensure the A record can be updated:

1
2
3
4
5
6
$ cloudflare-ddns --update-now
Found external IPv4: "1.x.x.x"
Listing all zones.
Finding all DNS records.
Updating the A record (ID x) of (sub)domain side-a.example.com (ID x) to 1.x.x.x.
DNS record updated successfully!

We can run this command from above in a cron, but I will use a bash script to only run when the public ip changed: /opt/scripts/detect_ip_change.sh:

1
2
3
4
5
6
7
8
#!/bin/bash
set -ex
MY_DDNS_HOST="side-a.example.com"

if [ $(dig ${MY_DDNS_HOST} +short) == $(curl -s icanhazip.com) ];
  then exit 0;
  else /usr/local/bin/cloudflare-ddns --update-now;
fi

Make the file executable: chmod +x /opt/scripts/detect_ip_change.sh then edit your cronjobs: crontab -e and add the script:

1
* * * * * /opt/scripts/detect_ip_change.sh

This will keep your DNS updated, this needs to be done on both sides, if you want to use dynamic dns.

Port Forwarding

We will need to forward UDP traffic from the router to the VPN server, on both sides:

1
2
3
4
5
Port: UDP/500 
Target: VPN-Server-IP:500

Port: UDP/4500
Target: VPN-Server-IP:4500

Create a Pre-Shared Key

Create a preshared key that will be used on both sides to authenticate:

1
2
$ openssl rand -base64 36
pgDU4eKZaQNL7GNRWJPvZbaSYFn2PAFjK9vDOvxAQ85p7qc4

This value will be used on both sides, which we will need later.

Install Strongswan on Side-A

Install strongswan and enable the service on boot:

1
2
$ apt install strongswan -y
$ systemctl enable strongswan

The left side will be the side we are configuring and the right side will be the remote side.

Create the config: /etc/ipsec.conf and provide the following config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
config setup
    charondebug="all"
    uniqueids=yes
    virtual_private=
    cachecrls=no

conn vpn-to-side-b
    type=tunnel
    authby=secret
    left=%defaultroute
    leftid=side-a.example.com
    leftsubnet=192.168.0.0/24
    right=%side-b.example.com
    rightid=side-b.example.com
    rightsubnet=192.168.1.0/24
    ike=aes256-sha2_256-modp1024!
    esp=aes256-sha2_256!
    keyingtries=0
    ikelifetime=1h
    lifetime=8h
    dpddelay=30
    dpdtimeout=120
    dpdaction=restart
    auto=start

Create the secrets file: /etc/ipsec.secrets:

1
side-b.example.com : PSK "pgDU4eKZaQNL7GNRWJPvZbaSYFn2PAFjK9vDOvxAQ85p7qc4"

Append the following kernel parameters to /etc/sysctl.conf:

1
2
3
net.ipv4.ip_forward = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0

Save:

1
$ sysctl -p

We now want to add a POSTROUTING and FORWARD rule using iptables:

1
2
$ iptables -t nat -A POSTROUTING -s 192.168.1.0/24  -d 192.168.0.0/24 -j MASQUERADE
$ iptables -A FORWARD -s 192.168.1.0/24 -d 192.168.0.0/24 -j ACCEPT

Now we need to route back:

1
$ ip route add 192.168.1.0/24 via 192.168.0.2 dev eth0

We want to persist the iptables and static route across reboots, so edit the /etc/rc.local file, if it’s not there create it with the following values:

1
2
3
4
5
#!/bin/bash
iptables -t nat -A POSTROUTING -s 192.168.1.0/24  -d 192.168.0.0/24 -j MASQUERADE
iptables -A FORWARD -s 192.168.1.0/24 -d 192.168.0.0/24 -j ACCEPT
ip route add 192.168.1.0/24 via 192.168.0.2 dev eth0
exit 0

If you created the file, make sure to apply executable permissions:

1
$ chmod +x /etc/rc.local

Read the secrets and restart strongswan:

1
2
$ ipsec rereadsecrets
$ systemctl restart strongswan

Install Strongswan on Side-B

Install strongswan and enable the service on boot:

1
2
$ apt install strongswan -y
$ systemctl enable strongswan

The left side will be the side we are configuring and the right side will be the remote side.

Create the config: /etc/ipsec.conf and provide the following config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
config setup
    charondebug="all"
    uniqueids=yes
    virtual_private=
    cachecrls=no

conn vpn-to-side-a
    type=tunnel
    authby=secret
    left=%defaultroute
    leftid=side-b.example.com
    leftsubnet=192.168.1.0/24
    right=%side-a.example.com
    rightid=side-a.example.com
    rightsubnet=192.168.0.0/24
    ike=aes256-sha2_256-modp1024!
    esp=aes256-sha2_256!
    keyingtries=0
    ikelifetime=1h
    lifetime=8h
    dpddelay=30
    dpdtimeout=120
    dpdaction=restart
    auto=start

Create the secrets file: /etc/ipsec.secrets:

1
side-a.example.com : PSK "pgDU4eKZaQNL7GNRWJPvZbaSYFn2PAFjK9vDOvxAQ85p7qc4"

Append the following kernel parameters to /etc/sysctl.conf:

1
2
3
net.ipv4.ip_forward = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0

Save:

1
$ sysctl -p

We now want to add a POSTROUTING and FORWARD rule using iptables:

1
2
$ iptables -t nat -A POSTROUTING -s 192.168.0.0/24  -d 192.168.1.0/24 -j MASQUERADE
$ iptables -A FORWARD -s 192.168.0.0/24 -d 192.168.1.0/24 -j ACCEPT

Now we need to route back:

1
$ ip route add 192.168.0.0/24 via 192.168.1.2 dev eth0

We want to persist the iptables and static route across reboots, so edit the /etc/rc.local file, if it’s not there create it with the following values:

1
2
3
4
5
#!/bin/bash
iptables -t nat -A POSTROUTING -s 192.168.0.0/24  -d 192.168.1.0/24 -j MASQUERADE
iptables -A FORWARD -s 192.168.0.0/24 -d 192.168.1.0/24 -j ACCEPT
ip route add 192.168.0.0/24 via 192.168.1.2 dev eth0
exit 0

If you created the file, make sure to apply executable permissions:

1
$ chmod +x /etc/rc.local

Read the secrets and restart strongswan:

1
2
$ ipsec rereadsecrets
$ systemctl restart strongswan

Verify Status

Verify that the ipsec tunnel is up on side-a:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ ipsec statusall

Connections:
  vpn-to-side-b:  %any...side-b.example.com,0.0.0.0/0,::/0  IKEv1/2
  vpn-to-side-b:   local:  [side-a.example.com] uses pre-shared key authentication
  vpn-to-side-b:   remote: [side-b.example.com] uses pre-shared key authentication
  vpn-to-side-b:   child:  192.168.0.0/24 === 192.168.1.0/24 TUNNEL
Security Associations (1 up, 0 connecting):
  vpn-to-side-b[1]: ESTABLISHED 28 minutes ago, 192.168.0.2[side-a.example.com]...4x.x.x.214[side-b.example.com]
  vpn-to-side-b[1]: IKEv2 SPIs: 81996170df1c927d_i e8294946491ddf08_r, pre-shared key reauthentication in 2 hours
  vpn-to-side-b[1]: IKE proposal: AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
  vpn-to-side-b{2}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: cc4504be_i c294cb26_o
  vpn-to-side-b{2}:  AES_CBC_128/HMAC_SHA2_256_128, 0 bytes_i, 240 bytes_o (4 pkts, 7s ago), rekeying in 18 minutes
  vpn-to-side-b{2}:   192.168.0.0/24 === 192.168.1.0/24

Verify that the ipsec tunnel is up on side-b:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ ipsec statusall

Connections:
 vpn-to-side-a:  %any...side-a.example.com,0.0.0.0/0,::/0  IKEv1/2
 vpn-to-side-a:   local:  [side-b.example.com] uses pre-shared key authentication
 vpn-to-side-a:   remote: [side-a.example.com] uses pre-shared key authentication
 vpn-to-side-a:   child:  192.168.1.0/24 === 192.168.0.0/24 TUNNEL
Security Associations (1 up, 0 connecting):
 vpn-to-side-a[2]: ESTABLISHED 20 minutes ago, 192.168.1.2[side-b.example.com]...14x.x.x.x[side-a.example.com]
 vpn-to-side-a[2]: IKEv2 SPIs: 81996170df1c927d_i e8294946491ddf08_r, pre-shared key reauthentication in 2 hours
 vpn-to-side-a[2]: IKE proposal: AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256
 vpn-to-side-a{2}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c294cb26_i cc4504be_o
 vpn-to-side-a{2}:  AES_CBC_128/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 26 minutes
 vpn-to-side-a{2}:   192.168.1.0/24 === 192.168.0.0/24

From side-a (192.168.0.2) ping the gateway on side-b (192.168.1.1):

1
2
3
$ $ ping -c2 192.168.1.1
PING 10.3.96.2 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=62 time=11.9 ms

If you want to be able to reach the private range of the other side of the vpn from any device on your network, you should add a static route on your router to inform your default gateway where to route traffic to.

In this case on side-a (192.168.0.0/24) we want to inform our default gateway to route (192.168.1.0/24) to the VPN as it knows to route that destination over the VPN.

On side-a, on your router, add a static route:

1
2
3
Route: 192.168.1.0
Subnet: 255.255.255.0
Gateway: 192.168.0.2

On side-b, on your router, add a static route:

1
2
3
Route: 192.168.0.0
Subnet: 255.255.255.0
Gateway: 192.168.1.2

Optional: Roadwarrior VPN Clients

This step is optional, but since we can access each others homelabs, we thought it would be nice to be able to access the resources from mobile devices or laptops when we are on remote locations.

We made it that each VPN owner will connect to its own endpoint (for roadwarriors), so side-a (which will be me) will connect to its own dns endpoint to connect when away from home..

I will only demonstrate how to append your config to add the ability for a roadwarrion vpn connection, append to the /etc/ipsec.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
# ...
conn ikev2-vpn
    auto=add
    type=tunnel
    authby=secret
    left=%any
    leftid=side-a.roadwarrior
    leftsubnet=0.0.0.0/0
    right=%any
    rightid=%any
    rightsourceip=10.10.0.0/24
    rightdns=192.168.0.1,8.8.8.8
    auto=start

Append the secret in /etc/ipsec.secrets:

1
2
# ...
side-a.roadwarrior my-laptop : PSK "MySuperSecureSecret123"

Add the vpn ip’s that we will assign to the roardwarrior clients to the routing table:

1
$ ip route add 10.10.0.0/24 via 192.168.0.2 dev eth0

If you only want the roadwarriors to be able to reach your network, you will only forward to the local network such as:

1
$ iptables -A FORWARD -s 10.10.0.0/24 -d 192.168.0.0/24 -j ACCEPT

But we will be forwarding traffic to all destinations:

1
2
$ iptables -A FORWARD -s 10.10.0.0/24 -d 0.0.0.0/0 -j ACCEPT
$ iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -d 0.0.0.0/0 -j MASQUERADE

Remember to append the routes to /etc/rc.local to persist across reboots.

Reread the secrets and restart strongswan:

1
2
$ ipsec rereadsecrets
$ systemctl restart strongswan

Connecting your VPN Client, I will be using my Laptop, with the following details:

1
2
3
4
5
6
7
VPN Type: IKEv2
Description: Home VPN
Server: side-a.example.com
Remote ID: side-a.roadwarrior
Local ID: my-laptop
User Authentication: None
Secret: MySuperSecureSecret123

Thank You

In this tutorial I demonstrated how to setup a site to site ipsec vpn between 2 sides that consists of internet connections that has dynamic ip’s and also appending roadwarrior config so that you can connect to your homelab from anywhere in the world.

Persistent Volumes With K3d Kubernetes

With k3d we can mount the host to container path, and with persistent volumes we can set a hostPath for our persistent volumes. With k3d, all the nodes will be using the same volume mapping which maps back to the host.

We will test the data persistence by writing a file inside a container, kill the pod, then exec into the pod again and test if the data persisted

The k3d Cluster

Create the directory on the host where we will persist the data:

1
> mkdir -p /tmp/k3dvol

Create the cluster:

1
2
> k3d create --name "k3d-cluster" --volume /tmp/k3dvol:/tmp/k3dvol --publish "80:80" --workers 2
> export KUBECONFIG="$(k3d get-kubeconfig --name='k3d-cluster')"

Our application will be a busybox container which will keep running with a ping command, map the persistent volume to /data inside the pod.

Our app.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/k3dvol"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  selector:
    matchLabels:
      app: echo
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: echo
    spec:
      volumes:
        - name: task-pv-storage
          persistentVolumeClaim:
            claimName: task-pv-claim
      containers:
      - image: busybox
        name: echo
        volumeMounts:
          - mountPath: "/data"
            name: task-pv-storage
        command: ["ping", "127.0.0.1"]

Deploy the workload:

1
2
3
4
> kubectl apply -f app.yml
persistentvolume/task-pv-volume created
persistentvolumeclaim/task-pv-claim created
deployment.apps/echo created

View the persistent volumes:

1
2
3
> kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
task-pv-volume                             1Gi        RWO            Retain           Bound    default/task-pv-claim    manual                  6s

View the Persistent Volume Claims:

1
2
3
> kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
task-pv-claim    Bound    task-pv-volume                             1Gi        RWO            manual         11s

View the pods:

1
2
3
> kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
echo-58fd7d9b6-x4rxj   1/1     Running   0          16s

Exec into the pod:

1
2
3
4
5
6
7
8
9
> kubectl exec -it echo-58fd7d9b6-x4rxj sh
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  58.4G     36.1G     19.3G  65% /
osxfs                   233.6G    139.7G     86.3G  62% /data
/dev/sda1                58.4G     36.1G     19.3G  65% /etc/hosts
/dev/sda1                58.4G     36.1G     19.3G  65% /dev/termination-log
/dev/sda1                58.4G     36.1G     19.3G  65% /etc/hostname
/dev/sda1                58.4G     36.1G     19.3G  65% /etc/resolv.conf

Write the hostname of the current pod to the persistent volume path:

1
2
3
4
/ # echo $(hostname)
echo-58fd7d9b6-x4rxj
/ # echo $(hostname) > /data/hostname.txt
/ # exit

Exit the pod and read the content from the host (workstation/laptop):

1
2
> cat /tmp/k3dvol/hostname.txt
echo-58fd7d9b6-x4rxj

Look at the host where the pod is running on:

1
2
3
4
5
> kubectl get nodes -o wide
NAME                       STATUS   ROLES    AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE   KERNEL-VERSION     CONTAINER-RUNTIME
k3d-k3d-cluster-server     Ready    master   13m   v1.17.2+k3s1   192.168.32.2   <none>        Unknown    4.9.184-linuxkit   containerd://1.3.3-k3s1
k3d-k3d-cluster-worker-1   Ready    <none>   13m   v1.17.2+k3s1   192.168.32.4   <none>        Unknown    4.9.184-linuxkit   containerd://1.3.3-k3s1
k3d-k3d-cluster-worker-0   Ready    <none>   13m   v1.17.2+k3s1   192.168.32.3   <none>        Unknown    4.9.184-linuxkit   containerd://1.3.3-k3s1

Delete the pod:

1
2
> kubectl delete pod/echo-58fd7d9b6-x4rxj
pod "echo-58fd7d9b6-x4rxj" deleted

Wait until the pod is rescheduled again and verify if the pod is running on a different node:

1
2
3
> kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP          NODE                       NOMINATED NODE   READINESS GATES
echo-58fd7d9b6-fkvbs   1/1     Running   0          35s   10.42.2.9   k3d-k3d-cluster-worker-1   <none>           <none>

Exec into the new pod:

1
> kubectl exec -it echo-58fd7d9b6-fkvbs sh

View if the data is persisted:

1
2
3
4
5
/ # hostname
echo-58fd7d9b6-fkvbs

/ # cat /data/hostname.txt
echo-58fd7d9b6-x4rxj

Asynchronous Function With OpenFaas

In this post we will explore how to use asynchronous functions in OpenFaas.

What are we doing

A synchronous request blocks the client until operation completes, where a asynchronous request doesn’t block the client, which is nice to use for long-running tasks or function invocations to run in the background through the use of NATS Streaming.

We will be building a Python Flask API Server which will act as our webhook service. When we invoke our function by making a http request, we also include a callback url as a header which will be the address where the queue worker will post it’s results.

Then we will make a http request to the synchronous function where we will get the response from the function and a http request to the asynchronous function, where we will get the response from the webhook service’s logs

Deploy OpenFaas

Deploy OpenFaas on a k3d Kubernetes Cluster if you want to follow along on your laptop. You can follow this post to deploy a kubernetes cluster and deploying openfaas:

Webhook Service

Lets build the Python Flask Webhook Service, our application code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from flask import Flask, request
from logging.config import dictConfig

dictConfig({
    'version': 1,
    'formatters': {'default': {
        'format': '[%(asctime)s] %(levelname)s in %(module)s: %(message)s',
    }},
    'handlers': {'wsgi': {
        'class': 'logging.StreamHandler',
        'stream': 'ext://flask.logging.wsgi_errors_stream',
        'formatter': 'default'
    }},
    'root': {
        'level': 'INFO',
        'handlers': ['wsgi']
    }
})

app = Flask(__name__)

@app.route("/", methods=["POST", "GET"])
def main():
    response = {}

    if request.method == "GET":
        response["event"] = "GET"
        app.logger.info("Received Event: GET")

    if request.method == "POST":
        response["event"] = request.get_data()
        app.logger.info("Receveid Event: {}".format(response))

    else:
        response["event"] == "OTHER"

    print("Received Event:")
    print(response)
    return "event: {} \n".format(response)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Our Dockerfile:

1
2
3
4
5
FROM python:3.7-alpine
RUN pip install flask
ADD app.py /app.py
EXPOSE 5000
CMD ["python", "/app.py"]

Building and Pushing to Docker Hub (or you can use my docker image):

1
2
$ docker build -t yourusername/python-flask-webhook:openfaas .
$ docker push yourusername/python-flask-webhook:openfaas

Create the deployment manifest webhook.yml for our webhook service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
$ cat > webhook.yml << EOF
apiVersion: v1
kind: Service
metadata:
  name: webhook-service
spec:
  selector:
    app: webhook
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000
      name: web
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webhook-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: webhook.localdns.xyz
    http:
      paths:
      - backend:
          serviceName: webhook-service
          servicePort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: webhook
  name: webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webhook
  template:
    metadata:
      labels:
        app: webhook
    spec:
      containers:
      - name: webhook
        image: ruanbekker/python-flask-webhook:openfaas
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000
          name: http
          protocol: TCP
EOF

Now deploy to kubernetes:

1
$ kubectl apply -f webhook.yml

After a minute or so, verify that you get a response when making a http request:

1
2
$ curl http://webhook.localdns.xyz
event: {'event': 'GET'}

Deploy the OpenFaas Function

We will deploy a dockerfile type function which will return the data that we feed it:

1
2
3
4
5
6
7
$ faas-cli new --lang dockerfile function-async-task
$ faas-cli up -f function-async-task.yml

Deploying: function-async-task.

Deployed. 202 Accepted.
URL: http://openfaas.localdns.xyz/function/function-async-task

List the functions:

1
2
3
$ faas-cli list
Function                       Invocations      Replicas
function-async-task            0               1

Describe the function:

1
2
3
4
5
6
7
8
9
10
11
12
$ faas-cli describe function-async-task
Name:                function-async-task
Status:              Ready
Replicas:            1
Available replicas:  1
Invocations:         0
Image:               ruanbekker/function-async-task:latest
Function process:
URL:                 http://openfaas.localdns.xyz/function/function-async-task
Async URL:           http://openfaas.localdns.xyz/async-function/function-async-task
Labels:              faas_function : function-async-task
Annotations:         prometheus.io.scrape : false

Testing

Test synchronous function:

1
2
$ curl http://openfaas.localdns.xyz/function/function-async-task -d "test"
test

Test asynchronous function, remember, here we need to provide the callback url which the queue worker will inform, which will be our webhook service:

1
2
3
4
5
6
7
$ curl -i -H "X-Callback-Url: http://webhook-service.default.svc.cluster.local:5000" http://openfaas.localdns.xyz/async-async-function/function-async-task -d "asyyyyync"
HTTP/1.1 202 Accepted
Content-Length: 0
Date: Mon, 17 Feb 2020 13:57:26 GMT
Vary: Accept-Encoding
X-Call-Id: d757c10f-4293-4daa-bf52-bbdc17b7dea3
X-Start-Time: 1581947846737501600

Check the logs of the webhook pod:

1
2
3
$ kubectl logs -f pod/$(kubectl get pods --selector=app=webhook --output=jsonpath="{.items..metadata.name}")
[2020-02-17 13:57:26,774] INFO in app: Receveid Event: {'event': b'asyyyyync'}
[2020-02-17 13:57:26,775] INFO in internal: 10.42.0.6 - - [17/Feb/2020 13:57:26] "POST / HTTP/1.1" 200 -

Check the logs of the queue worker:

1
2
3
4
5
6
7
$ kubectl logs -f deployment/queue-worker -n openfaas
[45] Received on [faas-request]: 'sequence:45 subject:"faas-request" data:"{\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip\"],\"Content-Length\":[\"9\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"User-Agent\":[\"curl/7.54.0\"],\"X-Call-Id\":[\"d757c10f-4293-4daa-bf52-bbdc17b7dea3\"],\"X-Callback-Url\":[\"http://webhook-service.default.svc.cluster.local:5000\"],\"X-Forwarded-For\":[\"10.42.0.0\"],\"X-Forwarded-Host\":[\"openfaas.localdns.xyz\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-6787cddb4b-87zss\"],\"X-Real-Ip\":[\"10.42.0.0\"],\"X-Start-Time\":[\"1581947846737501600\"]},\"Host\":\"openfaas.localdns.xyz\",\"Body\":\"YXN5eXl5eW5j\",\"Method\":\"POST\",\"Path\":\"\",\"QueryString\":\"\",\"Function\":\"openfaas-function-cat\",\"CallbackUrl\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"webhook-service.default.svc.cluster.local:5000\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"}}" timestamp:1581947846738308800 '
Invoking: openfaas-function-cat with 9 bytes, via: http://gateway.openfaas.svc.cluster.local:8080/function/openfaas-function-cat/
Invoked: openfaas-function-cat [200] in 0.029029s
Callback to: http://webhook-service.default.svc.cluster.local:5000
openfaas-function-cat returned 9 bytes
Posted result for openfaas-function-cat to callback-url: http://webhook-service.default.svc.cluster.local:5000, status: 200

Make 1000 Requests:

1
2
3
4
5
6
$ date > time.date
  for x in {1..1000}
    do
      curl -i -H "X-Callback-Url: http://webhook-service.default.svc.cluster.local:5000" http://openfaas.localdns.xyz/async-function/openfaas-function-cat -d "asyyyyync"
    done
  date >> time.date

View the log file that we wrote before we started and finished our requests:

1
2
3
$ cat time.date
Mon Feb 17 16:03:16 SAST 2020
Mon Feb 17 16:03:48 SAST 2020

The last request was actioned at:

1
[2020-02-17 14:03:52,421] INFO in internal: 10.42.0.6 - - [17/Feb/2020 14:03:52] "POST / HTTP/1.1" 200 -

Thank You

This was a basic example to demonstrate async functions using OpenFaas

OpenFaas Documentation:

Traefik Ingress for OpenFaas on Kubernetes (K3d)

In this post we will deploy OpenFaas on Kubernetes locally using k3sup and k3d, then deploy a Traefik Ingress so that we can access the OpenFaas Gateway on HTTP over the standard port 80.

K3d is a amazing wrapper that deploys a k3s cluster on docker, and k3sup makes it very easy to provision OpenFaas to your Kubernetes cluster.

Deploy a Kubernetes Cluster

If you have not installed k3d, you can install k3d on mac with brew:

1
$ brew install k3d

We will deploy our cluster with 2 worker nodes and publish port 80 to the containers port 80:

1
$ k3d create --name="demo" --workers="2" --publish="80:80"

Point the kubeconfig to the location that k3d generated:

1
$ export KUBECONFIG="$(k3d get-kubeconfig --name='demo')"

Deploy OpenFaas

First we need to get k3sup:

1
$ curl -sLfS https://get.k3sup.dev | sudo sh

Once k3sup is installed, deploy OpenFaas to your cluster:

1
$ k3sup app install openfaas

Give it a minute or so and check if everything is running:

1
2
3
4
5
6
7
8
9
$ kubectl get pods -n openfaas
NAMESPACE     NAME                                 READY   STATUS      RESTARTS   AGE
openfaas      alertmanager-546f66b6c6-qtb69        1/1     Running     0          5m
openfaas      basic-auth-plugin-79b9878b7b-7vlln   1/1     Running     0          4m59s
openfaas      faas-idler-db8cd9c7d-8xfpp           1/1     Running     2          4m57s
openfaas      gateway-7dcc6d694d-dmvqn             2/2     Running     0          4m56s
openfaas      nats-d6d574749-rt9vw                 1/1     Running     0          4m56s
openfaas      prometheus-d99669d9b-mfxc8           1/1     Running     0          4m53s
openfaas      queue-worker-75f44b56b9-mhhbv        1/1     Running     0          4m52s

Traefik Ingress

In my scenario, I am using openfaas.localdns.xyz which resolves to 127.0.0.1. Next we need to know to which service to route the traffic to, we can find that by:

1
2
3
$ kubectl get svc/gateway -n openfaas
NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
gateway   ClusterIP   10.43.174.57   <none>        8080/TCP   23m

Below is our ingress.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: openfaas-gateway-ingress
  namespace: openfaas
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: openfaas.localdns.xyz
    http:
      paths:
      - backend:
          serviceName: gateway
          servicePort: 8080

Apply the ingress:

1
2
$ kubectl apply -f ingress.yml
ingress.extensions/openfaas-gateway-ingress created

We can the verify that our ingress is visible:

1
2
3
$ kubectl get ingress -n openfaas
NAMESPACE   NAME                       HOSTS               ADDRESS      PORTS   AGE
openfaas    openfaas-gateway-ingress   openfaas.co.local   172.25.0.4   80      28s

OpenFaas CLI

Install the OpenFaas CLI:

1
$ curl -SLsf https://cli.openfaas.com | sudo sh

Export the OPENFAAS_URL to our ingress endpoint and OPENFAAS_PREFIX for your dockerhub username:

1
2
$ export OPENFAAS_URL=http://openfaas.localdns.xyz
$ export OPENFAAS_PREFIX=ruanbekker # change to your username

Get your credentials for the OpenFaas Gateway and login with the OpenFaas CLI:

1
2
$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
$ echo -n $PASSWORD | faas-cli login --username admin --password-stdin

Deploy a Function

Deploy the figlet function as an example:

1
2
3
4
$ faas-cli store deploy figlet

Deployed. 202 Accepted.
URL: http://openfaas.localdns.xyz/function/figlet

Invoke the function:

1
2
3
4
5
6
7
$ curl http://openfaas.localdns.xyz/function/figlet -d 'hello, world'
 _          _ _                             _     _
| |__   ___| | | ___    __      _____  _ __| | __| |
| '_ \ / _ \ | |/ _ \   \ \ /\ / / _ \| '__| |/ _` |
| | | |  __/ | | (_) |   \ V  V / (_) | |  | | (_| |
|_| |_|\___|_|_|\___( )   \_/\_/ \___/|_|  |_|\__,_|
                    |/

Delete the Cluster

Delete your k3d Kubernetes Cluster:

1
$ k3d delete --name demo

Thank You