Ruan Bekker's Blog

From a Curious mind to Posts on Github

How to Bootstrap Nodes With Python Using Ansible

As Ansible depends on Python, therefore we can bootstrap our nodes with Python using a Ansible Playbook

Inventory

The nodes we want to bootstrap:

inventory.ini
1
2
3
4
5
6
7
[new]
node-1
node-2
node-3

[new:vars]
ansible_python_interpreter=/usr/bin/python3

Playbook

Our playbook with what we want to do:

bootstrap-python.yml
1
2
3
4
5
6
7
---
- hosts: all
  gather_facts: False

  tasks:
  - name: install python
    raw: test -e /usr/bin/python || ( apt update && apt install python -y )

Deploy

Deploy with Ansible:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ansible-playbook -i inventory.ini bootstrap-python.yml

PLAY [all] ***********************************************************************************************************************************************************************************************

TASK [install python] ************************************************************************************************************************************************************************************
changed: [node-1]
changed: [node-2]
changed: [node-3]

PLAY RECAP ***********************************************************************************************************************************************************************************************
node-1                     : ok=2    changed=2    unreachable=0    failed=0
node-2                     : ok=2    changed=2    unreachable=0    failed=0
node-3                     : ok=2    changed=2    unreachable=0    failed=0

This is it for this post, all posts for this tutorial will be posted under #ansible-tutorial

How to Install Packages on Remote Systems With Ansible

We will use Ansible to deploy packages to remote systems and in this case all the remote systems are running Debian, therefore we will be using the APT package manager.

Pre-Requisites:

Ensure that you have installed Ansible and setup the SSH Config for your remote systems, how to do that can be found under the post: setting up ansible

Our Inventory

The inventory file that describes our hosts:

inventory.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[scaleway]
cluster-node-1
cluster-node-2

[hetzner]
docker-node-1
docker-node-2
docker-node-3
glusterfs-node-1
glusterfs-node-2
elasticsearch-node-1
elasticsearch-node-2

[scaleway:vars]
ansible_python_interpreter=/usr/bin/python3
location=france

[hetzner:vars]
ansible_python_interpreter=/usr/bin/python3
location=germany

Playbook

Our playbook that we will define that we want to deploy packages using apt to all hosts:

packages.yml
1
2
3
4
5
6
7
8
9
10
11
12
---
- hosts: all
  tasks:
  - name: Install Packages
    apt: name= state=latest update_cache=yes
    with_items:
      - ntp
      - python
      - tcpdump
      - wget
      - openssl
      - curl

Deploy

Running the playbook to deploy the packages to the remote servers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ ansible-playbook -i inventory.ini packages.yml

PLAY [all] ***********************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
ok: [glusterfs-node-2]
ok: [glusterfs-node-1]
ok: [docker-node-1]
ok: [docker-node-2]
ok: [docker-node-3]
ok: [elasticsearch-node-1]
ok: [elasticsearch-node-2]
ok: [cluster-node-1]
ok: [cluster-node-2]

TASK [Install Packages] **********************************************************************************************************************************************************************************
changed: [docker-node-1] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
changed: [docker-node-2] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
changed: [docker-node-3] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
changed: [elasticsearch-node-1] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
changed: [glusterfs-node-1] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
changed: [glusterfs-node-2] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
changed: [elasticsearch-node-2] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
ok: [cluster-node-1] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])
ok: [cluster-node-2] => (item=[u'ntp', u'python', u'tcpdump', u'wget', u'openssl', u'curl'])

PLAY RECAP ***********************************************************************************************************************************************************************************************
docker-node-1              : ok=2    changed=1    unreachable=0    failed=0
docker-node-2              : ok=2    changed=1    unreachable=0    failed=0
docker-node-3              : ok=2    changed=1    unreachable=0    failed=0
elasticsearch-node-1       : ok=2    changed=1    unreachable=0    failed=0
elasticsearch-node-2       : ok=2    changed=1    unreachable=0    failed=0
glusterfs-node-1           : ok=2    changed=1    unreachable=0    failed=0
glusterfs-node-2           : ok=2    changed=1    unreachable=0    failed=0
cluster-node-1             : ok=2    changed=0    unreachable=0    failed=0
cluster-node-2             : ok=2    changed=0    unreachable=0    failed=0

This is it for this post, all posts for this tutorial will be posted under #ansible-tutorials

Query 24 Hours Worth of Data Using BatchGet on Amazon DynamoDB Using Scan and Filter Without a GSI

I’m testing how to query data in DynamoDB which will always be the retrieval of yesterdays data, without using a Global Secondary Index.

This is done just to see what other ways you can use to query data based on a specific timeframe.

Use-Case:

Data from DynamoDB needs to be batch processed (daily for the last 24-hours), into a external datasource. Data will be written into DynamoDB, the HK (uuid) and RK (timestamp) will be duplicated to the daily table. But only uuid and timestamp will be duplicated to the daily table, and only data for that day will be written into that datestamp formatted table name.

Let’s say data for 2018-10-30 needs to be written into our external data source, we will do a scan on table tbl-test_20181030, then from our response we will have a list of HashKeys (uuid) which we will use to do a BatchGet Item on our base table: tbl-test_base, which essentially grabs all the data for that day.

If deeper filtering needs to be done on that day, the FilterExpression can be used to do a deeper filtering which leads to grabbing only the filtered down data from the base table.

Note: The base table might have millions of items, so a Scan operation on the Base table would be really expensive, as it reads all the items in the table.

Once the data has been processed, the daily or metadata table can be removed.

DynamoDB Table Design

The base table: tbl-test_base will have:

  • HashKey: uuid (string)
  • RangeKey: timestamp (number)
  • Attributes: city, stream, transaction_date, name, metric_uri
  • Item will look like:
1
2
3
4
5
6
7
8
9
{
  u'uuid': u'fb4ddeb9-3b5e-47b3-bbab-1aa1d8e8f47b',
  u'timestamp': 1540891276,
  u'city': u'sydney',
  u'stream': u'NONE',
  u'transaction_date': u'2018-10-30 11:21:16',
  u'metric_uri': u'some-dummy-metric-uri',
  u'name': u'frank'
}

he Daily Table: tbl-test_20181030 will look like:

  • HashKey: uuid
  • Attributes: timestamp
  • Item will look like:
1
2
3
4
{
  u'uuid': u'fb4ddeb9-3b5e-47b3-bbab-1aa1d8e8f47b',
  u'timestamp': 1540891276
}

Demonstration using Python

Creating the Metadata table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import boto3, time, uuid, random

session = boto3.Session(region_name='eu-west-1', profile_name='dev')
resource = session.resource('dynamodb')
client = session.client('dynamodb')

def create_table():
    table_name = "tbl-test_{0}".format(time.strftime("%Y%m%d"))
    response = resource.create_table(
        TableName=table_name,
        KeySchema=[{
            'AttributeName': 'uuid',
            'KeyType': 'HASH'
        }],
        AttributeDefinitions=[{
            'AttributeName': 'uuid',
            'AttributeType': 'S'
        }],
        ProvisionedThroughput={
            'ReadCapacityUnits': 1,
            'WriteCapacityUnits': 1
        }
    )

    resource.Table(table_name).wait_until_exists()

    arn = client.describe_table(TableName=table_name)['Table']['TableArn']
    client.tag_resource(
        ResourceArn=arn,
        Tags=[
            {'Key': 'Name','Value': 'dynamo_table'},
            {'Key': 'Environment','Value': 'Dev'},
            {'Key': 'CreatedBy','Value': 'Ruan'}
        ]
    )

    return resource.Table(table_name).table_status

print(create_table())

Write 400 Items to DynamoDB:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import boto3, time, uuid, random

session = boto3.Session(region_name='eu-west-1', profile_name='dev')
resource = session.resource('dynamodb')
client = session.client('dynamodb')

base_table = 'tbl-test_base'
meta_table = 'tbl-test_{0}'.format(time.strftime("%Y%m%d"))

people = ['james', 'john', 'frank', 'paul', 'nathan', 'kevin']
cities = ['ireland', 'cape town', 'pretoria', 'paris', 'amsterdam', 'auckland', 'sydney']

def write_dynamo(uuid, timestamp):
    resource.Table(base_table).put_item(
        Item={
            'uuid': uuid,
            'timestamp': timestamp,
            'metric_uri': 'some-dummy-metric-uri',
            'transaction_date': time.strftime("%Y-%m-%d %H:%M:%S"),
            'name': random.choice(people),
            'stream': 'NONE',
            'city': random.choice(cities)
        }
    )

    resource.Table(meta_table).put_item(
        Item={
            'uuid': uuid,
            'timestamp': timestamp
        }
    )

    return 'Written'

for x in xrange(400):
    time.sleep(1)
    write_dynamo(str(uuid.uuid4()), int(time.time()))
    print(x)

Getting Data for 20181030 but also filter data greater than the timestamp attribute, greater than 1540841144 in epoch time (which will give us about 254 items).

The BatchGet Item supports up to 100 items per call, we will limit the scans on 100 items per call, then paginate using the ExlusiveStartKey with the value of our LastEvaluatedKey that we will get from our response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import boto3,time
from boto3.dynamodb.conditions import Key

base_table = 'tbl-test_base'
meta_table = 'tbl-test_20181030'

session = boto3.Session(region_name='eu-west-1', profile_name='dev')
resource = session.resource('dynamodb')
table = resource.Table(meta_table)
filtering_expression = Key('timestamp').gt(1540841144)

response = table.scan(FilterExpression=filtering_expression, Limit=100)

finished=False
while finished != True:
    if 'LastEvaluatedKey' in response.keys():
        print("Getting {} Items".format(response['Count']))
        items = resource.batch_get_item(RequestItems={base_table: {'Keys': response['Items']}})
        print(items['Responses'][base_table])
        time.sleep(2)
        response = table.scan(FilterExpression=filtering_expression, Limit=100, ExclusiveStartKey=response['LastEvaluatedKey'])
    else:
        print("Getting {} Items".format(response['Count']))
        items = resource.batch_get_item(RequestItems={base_table: {'Keys': response['Items']}})
        print(items['Responses'][base_table])
        finished=True

Running it:

1
2
3
4
5
6
7
8
9
$ python dynamodb-batch-get.py
Getting 100 Items
[{u'city': u'pretoria', u'uuid': u'e8bc0d1c-2b57-4de2-b0e1-35ef1fe0edf1', u'stream': u'NONE', u'timestamp': Decimal('1540846990'), u'transaction_date': u'2018-10-29 23:03:10', u'metric_uri': u'some-dummy-metric-uri', u'name': u'frank'}, {u'city': u'amsterdam', u'uuid':
...
Getting 100 Items
[{u'city': u'sydney', u'uuid': u'5bc51ce9-2809-46c9-a3f2-ff8180086d92', u'stream': u'NONE', u'timestamp': Decimal('1540848599'), u'transaction_date': u'2018-10-29 23:29:59', u'metric_uri': u'some-dummy-metric-uri', u'name': u'frank'}
...
Getting 54 Items
[{u'city': u'cape town', u'uuid': u'5e069f34-0e97-4a49-9ca9-da2213edb689'...

Verifying that each call only scans 100 at a time:

1
2
3
4
5
6
7
8
9
>>> response = table.scan(FilterExpression=filtering_expression, Limit=100)
>>> response.keys()
[u'Count', u'Items', u'LastEvaluatedKey', u'ScannedCount', 'ResponseMetadata']
>>> response.get('LastEvaluatedKey')
{u'uuid': u'e8c52a55-ca9e-4718-83d2-1b44a90f43e6'}
>>> response.get('Count')
100
>>> response.get('ScannedCount')
100

Other Thoughts:

Querying data is a lot easier using a Global Secondary Index where you could similarly have the metric_uri as the HashKey and transaction_date as the RangeKey:

1
2
3
4
5
6
>>> response = table.query(
    IndexName='metric_uri-transaction_date-index',
    KeyConditionExpression=Key('metric_uri').eq('some-dummy-metric-uri') & Key('transaction_date').begins_with('2018-10-30')
)
>>> response['Count']
400

Also note that depending on how you setup your GSI, in most cases its a exact duplicate in storage from your base table, so could potentially be double the costs.

Using Python Flask and JavaScript for Client Side Filtering Through Returned Data

This post will cover 2 sections, using Python Flask and Javascript to filter returned data, where you could have a table that represents 100 items, and you want to have a search box to filter down your results as you type.

The other section will be used as a demo, with solving a problem with Amazon CloudWatch Logs. I’m a Massive AWS Fanatic, but when it comes to CloudWatch Logs, I’m not so big of a fan of that specific service. Especially when you use Docker Swarm for AWS and have your logdriver set to CloudWatch Logs.

The Problem I have with CloudWatch Logs

When you point to your CloudWatch LogGroups, you can search for your streams, and in my case searching for a specific swarm service, but you can’t sort by date, like this:

This makes it really tedious when trying to search find your logs in a quick way.

Python Flask to the Resque

We will create a Python Flask application that retrieves your data about all your Docker Swarm Services and Container Id’s running on each node. For this demonstration, I have hard coded the services and container id’s, but using it in a real environment, you can utilise the Docker API or some logic that retrieves it from a datastore where a process populates it to.

The Application Code will do the following:

  • returns a list of your swarm services (mock data in the code)
  • when you select a service, it will get a list of the container ids and run through a for loop unsing jinja templates and display them in table format
  • when you select the containerId, it will populate the containerId to the cloudwatch logs filter, giving you the exact logstream which you are looking for
  • this will do a redirect to the AWS Console, and you will see the data in the sorted time of interest

  • app.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
from flask import Flask, render_template

app = Flask(__name__)

# faking datasets that can be returned from a api or database
swarm_services = ['my-web-service', 'my-api-service']
swarm_tasks = {
    "my-web-service": {
        "container_names": [
            "my-web-service.1.alfjshoehfosfn",
            "my-web-service.2.fuebchduehakjdu"
        ]
    },
    "my-api-service": {
        "container_names": [
            "my-api-service.1.oprudhyuythvbzx",
            "my-api-service.2.sjduebansifotuf"
        ]
    }
}

def get_container_name(app_name):
    data = []
    response = swarm_tasks[app_name]
    for container in response['container_names']:
        data.append(container)
    return render_template('index.html', app_name=app_name, number=len(data), data=data)

@app.route('/')
def list():
    return render_template('list.html', number=len(swarm_services), apps=swarm_services, aws_region='eu-west-1', cloudwatch_log_stream='docker-swarm-lg')

@app.route('/describe/<string:app_name>')
def get_app(app_name):
    app = get_container_name(app_name)
    return app

if __name__ == '__main__':
    app.run()

The index.html:

The list.html :

Filtering the Data

So at this moment all your data will be returned when a list is done, if you are in a case where you have lots of information, it can be overwelming and you will need to search for the service of interest. Using HTML and JavaScript, you can filter through the results:

The JavaScript Function: assets/js/filter.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
function SearchAndFilterThingy() {
  var input, filter, table, tr, td, x;
  input = document.getElementById("UserInput");
  filter = input.value.toUpperCase();
  table = document.getElementById("ServicesTable");
  tr = table.getElementsByTagName("tr");

  for (x = 0; x < tr.length; x++) {
    td = tr[x].getElementsByTagName("td")[0];
    if (td) {
      if (td.innerHTML.toUpperCase().indexOf(filter) > -1) {
        tr[x].style.display = "";
      }
      else {
        tr[x].style.display = "none";
      }
    }
  }
}

Screenshot

Once you search for a specific keyword on the service you are looking for the output should more or less look like the following:

Building Ghost Version 2 Blog for the RaspberryPi

In this post we will setup Ghost 2.0.3 for the Raspberry Pi on Docker Swarm

Dockerfile

Our dockerfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
FROM rbekker87/armhf-node:8.11

RUN apk add --no-cache 'su-exec>=0.2' && apk --update add bash gcc g++ make python && npm install sqlite3 --build-from-source

ENV NODE_ENV production
ENV GHOST_CLI_VERSION 1.9.1
ENV GHOST_VERSION 2.0.3
ENV GHOST_INSTALL /var/lib/ghost
ENV GHOST_CONTENT /var/lib/ghost/content

RUN npm install -g "ghost-cli@$GHOST_CLI_VERSION"

RUN set -ex; \
        mkdir -p "$GHOST_INSTALL" \
        && adduser -s /bin/sh -D node \
        && chown node:node "$GHOST_INSTALL" \
        && su-exec node ghost install "$GHOST_VERSION" --db sqlite3 --no-prompt --no-stack --no-setup --dir "$GHOST_INSTALL" \
        && cd "$GHOST_INSTALL" \
        && su-exec node ghost config --ip 0.0.0.0 --port 2368 --no-prompt --db sqlite3 --url http://localhost:2368 --dbpath "$GHOST_CONTENT/data/ghost.db" \
        && su-exec node ghost config paths.contentPath "$GHOST_CONTENT" \
        && su-exec node ln -s config.production.json "$GHOST_INSTALL/config.development.json" \
        && readlink -f "$GHOST_INSTALL/config.development.json" \
        && mv "$GHOST_CONTENT" "$GHOST_INSTALL/content.orig" \
        && mkdir -p "$GHOST_CONTENT" && chown node:node "$GHOST_CONTENT" \
        && "$GHOST_INSTALL/current/node_modules/knex-migrator/bin/knex-migrator" --version

ENV PATH $PATH:$GHOST_INSTALL/current/node_modules/knex-migrator/bin

WORKDIR $GHOST_INSTALL

COPY docker-entrypoint.sh /usr/local/bin
RUN chmod +x /usr/local/bin/docker-entrypoint.sh

ENTRYPOINT ["docker-entrypoint.sh"]

CMD ["node", "current/index.js"]

Our Boot Script

Our entrypoint script docker-entrypoint.sh:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
#!/bin/bash
set -e

if [[ "$*" == node*current/index.js* ]] && [ "$(id -u)" = '0' ];
  then
    chown -R node "$GHOST_CONTENT"
    exec su-exec node "$BASH_SOURCE" "$@"
fi

if [[ "$*" == node*current/index.js* ]];
  then
    baseDir="$GHOST_INSTALL/content.orig"
    for src in "$baseDir"/*/ "$baseDir"/themes/*;
      do
        src="${src%/}"
        target="$GHOST_CONTENT/${src#$baseDir/}"
        mkdir -p "$(dirname "$target")"
        if [ ! -e "$target" ];
          then
            tar -cC "$(dirname "$src")" "$(basename "$src")" | tar -xC "$(dirname "$target")"
        fi
      done

    knex-migrator-migrate --init --mgpath "$GHOST_INSTALL/current"
fi

prod() {
cat > /var/lib/ghost/config.development.json << EOF
{
  "url": "http://${SERVER_URL:-localhost}:${SERVER_PORT:-2368}",
  "server": {
    "port": ${SERVER_PORT:-2368},
    "host": "0.0.0.0"
  },
  "database": {
    "client": "sqlite3",
    "connection": {
      "filename": "/var/lib/ghost/content/data/ghost.db"
    }
  },
  "mail": {
    "transport": "SMTP",
    "from": "${FROM_NAME:-MyBlog} <${FROM_EMAIL:-ghost-blog@localhost}>",
    "options": {
      "service": "Mailgun",
      "host": "${SMTP_HOST:-localhost}",
      "port": ${SMTP_PORT:-25},
      "auth": {
        "user": "${SMTP_AUTH_USERNAME:-root}",
        "pass": "${SMTP_AUTH_PASSWORD:-password}"
      }
    }
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/lib/ghost/content"
  }
}
EOF
}

dev() {
cat > /var/lib/ghost/config.development.json << EOF
{
  "url": "http://${SERVER_URL:-localhost}:${SERVER_PORT:-2368}",
  "server": {
    "port": ${SERVER_PORT:-2368},
    "host": "0.0.0.0"
  },
  "database": {
    "client": "sqlite3",
    "connection": {
      "filename": "/var/lib/ghost/content/data/ghost.db"
    }
  },
  "mail": {
    "transport": "Direct"
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/lib/ghost/content"
  }
}
EOF
}

test(){
cat > /var/lib/ghost/config.development.json << EOF
{
  "url": "http://localhost:2368",
  "server": {
    "port": 2368,
    "host": "0.0.0.0"
  },
  "database": {
    "client": "sqlite3",
    "connection": {
      "filename": "/var/lib/ghost/content/data/ghost.db"
    }
  },
  "mail": {
    "transport": "Direct"
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/lib/ghost/content"
  }
}
EOF
}

if  [ "${ENV_TYPE}" = "PROD" ]
  then prod

elif [ "${ENV_TYPE}" = "DEV" ]
  then dev
  else test

fi

exec "$@"

The entrypoint script takes a couple of environment variables, as you can see if they are not defined, defaults will be inherited.

Configurable Environment Variables:

1
2
3
4
5
6
7
8
  - ENV_TYPE=PROD
  - SERVER_PORT=2368
  - SERVER_URL=myblog.pistack.co.za
  - FROM_NAME=MyName
  - SMTP_HOST=mail.mydomain.co.za
  - SMTP_PORT=587
  - SMTP_AUTH_USERNAME=me@mydomain.co.za
  - SMTP_AUTH_PASSWORD=secret

Building our Ghost Image

I have a public image available if you dont want to build/push, but for building:

1
$ docker build -t your-name/repo:tag

Deploy Ghost with Traefik

Our ghost-compose.yml with traefik will look like the following, note that I mounted the source path to the container’s path, the source path is running on a replicated glusterfs volume, which can be setup following this post

Also for this demonstration I was using the domain pistack.co.za, where you need to utilize the domain of your choice.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
version: "3.4"

services:
  ghost:
    image: rbekker87/armhf-ghost:2.0.3
    networks:
      - appnet
    volumes:
      - type: bind
        source: /mnt/volumes/myblog/content/data
        target: /var/www/ghost/content/data
    environment:
      - ENV_TYPE=PROD
      - SERVER_PORT=2368
      - SERVER_URL=myblog.pistack.co.za
      - FROM_NAME=MyName
      - SMTP_HOST=mail.mydomain.co.za
      - SMTP_PORT=587
      - SMTP_AUTH_USERNAME=me@mydomain.co.za
      - SMTP_AUTH_PASSWORD=secret
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.backend=ghost"
        - "traefik.backend.loadbalancer.swarm=true"
        - "traefik.docker.network=appnet"
        - "traefik.port=2368"
        - "traefik.frontend.passHostHeader=true"
        - "traefik.frontend.rule=Host:myblog.pistack.co.za"
      replicas: 3
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == worker]

networks:
  appnet:
    external: true

Deploy the stack:

1
$ docker stack deploy -c ghost-compose.yml web

Once the service is up, you will be able to reach your blog on the provided traefik.frontend.rule. If you don’t have traefik running, you can follow this post to get traefik up and running.

Resources:

Build a Traefik Proxy Image for Your Raspberry Pi on Docker Swarm

In this post we will setup a Docker Image for Traefik Proxy on the ARM Architecture, specifically on the Raspberry Pi, which we will deploy to our Raspberry Pi Docker Swarm.

Then we will build and push our image to a registry, then setup traefik and also setup a web application that sits behind our Traefik Proxy.

What is Traefik

Traefik is a modern load balancer and reverse proxy built for micro services.

Dockerfile

We will be running Traefik on Alpine 3.8:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
FROM rbekker87/armhf-alpine:3.8

ENV TRAEFIK_VERSION 1.7.0-rc3
ENV ARCH arm

ADD https://github.com/containous/traefik/releases/download/v${TRAEFIK_VERSION}/traefik_linux-${ARCH} /traefik

RUN apk add --no-cache ca-certificates \
    && chmod +x /traefik \
    && rm -rf /var/cache/apk/*

EXPOSE 80 8080 443

ENTRYPOINT ["/traefik"]

Build and Push

Build and Push your image to your registry of choice:

1
2
$ docker build -t your-user/repo:tag .
$ docker push your-user/repo:tag

If you do not want to push to a registry, I have a public image available at https://hub.docker.com/r/rbekker87/armhf-traefik/, the image itself is rbekker87/armhf-traefik:1.7.0-rc3

Deploy Traefik to the Swarm

From our traefik-compose.yml, you will notice that I have set that our network is external, so the network should exist prior to deploying the stack.

Let’s create the overlay network:

1
$ docker network create --driver overlay appnet

Below, the traefik-compose.yml, note that I’m using pistack.co.za as my domain:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
version: "3.4"

services:
  traefik:
    image: rbekker87/armhf-traefik:1.7.0-rc3
    command:
      - "--api"
      - "--docker"
      - "--docker.swarmmode"
      - "--docker.domain=pistack.co.za"
      - "--docker.watch"
      - "--logLevel=DEBUG"
      - "--web"
    networks:
      - appnet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 80:80
      - 8080:8080
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == manager]

networks:
  appnet:
    external: true

Deploy the stack:

1
$ docker stack deploy -c traefik-compose.yml proxy

List the stacks:

1
2
3
$ docker stack ls
NAME                SERVICES
proxy               1

Check if the services in your stack is running. Since our deploy mode was global, there will be a replica running on each node, and in my swarm I’ve got 3 nodes:

1
2
3
$ docker stack services proxy
ID                  NAME                MODE                REPLICAS            IMAGE                    PORTS
16x31j7o0f0r        proxy_traefik       global              3/3                 rbekker87/armhf-traefik:1.7.0-rc3   *:80->80/tcp,*:8080->8080/tcp

Deploy a Web Service hooked up to Traefik

Pre-Requirement:

To register subdomains on the fly, set you DNS for your domain to the following (im using pistack.co.za in this example):

  • pistack.co.za A x.x.x.x
  • *.pistack.co.za A x.x.x.x

Next, we will deploy we app that will be associated to our Traefik service domain, so we will inform Traefik that our web app fqdn and port that will be registered with the proxy.

Our app-compose.yml file for our webapp:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
version: "3.4"

services:
  whoami:
    image: rbekker87/golang-whoami:alpine-amrhf
    networks:
      - appnet
    deploy:
      replicas: 3
      labels:
        - "traefik.backend=whoami"
        - "traefik.port=80"
        - "traefik.frontend.rule=Host:whoami.pistack.co.za"
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == worker]
    healthcheck:
      test: nc -vz 127.0.0.1 80 || exit 1
      interval: 60s
      timeout: 3s
      retries: 3

networks:
  appnet:
    external: true

In the above compose, you will notice that our traefik backend is set to our service name, our port is the port that the proxy will forward requests to the containers port, since the proxy and the whoami container is in the same network, they will be able to communicate with each other. Then we also have our frontend rule which will be the endpoint we will reach our application on.

Deploy the stack:

1
2
$ docker stack deploy -c whoami.yml web
Creating service web_whoami

List the tasks running in our web stack:

1
2
3
$ docker stack services web
ID                  NAME                MODE                REPLICAS            IMAGE                                  PORTS
31ylfcfb7uyw        web_whoami          replicated          3/3                 rbekker87/golang-whoami:alpine-amrhf

Once all the replicas is running, move along to test the application

Testing our Application:

I have 3 replicas each running on their own container, so each container will respond with its own hostname:

1
2
3
4
5
$ docker service ps web_whoami
ID                  NAME                IMAGE                                  NODE                DESIRED STATE       CURRENT STATE            ERROR                              PORTS
ivn8fgfosvgd        web_whoami.1        rbekker87/golang-whoami:alpine-amrhf   rpi-01              Running             Running 26 minutes ago
rze6u6z56aop        web_whoami.2        rbekker87/golang-whoami:alpine-amrhf   rpi-02              Running             Running 26 minutes ago
6fjua869r498        web_whoami.3        rbekker87/golang-whoami:alpine-amrhf   rpi-04              Running             Running 23 minutes ago

Making our 1st GET request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ $ curl http://whoami.pistack.co.za/
Hostname: 43f5f0a6682f
IP: 127.0.0.1
IP: 10.0.0.138
IP: 10.0.0.218
IP: 172.18.0.4
GET / HTTP/1.1
Host: whoami.pistack.co.za
User-Agent: curl/7.38.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 165.73.96.95, 10.255.0.2
X-Forwarded-Host: whoami.pistack.co.za
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 31b37f9714d3
X-Real-Ip: 10.255.0.2

Our 2nd GET Request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ curl http://whoami.pistack.co.za/
Hostname: d1c17a476414
IP: 127.0.0.1
IP: 10.0.0.138
IP: 10.0.0.71
IP: 172.19.0.5
GET / HTTP/1.1
Host: whoami.pistack.co.za
User-Agent: curl/7.38.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 165.73.96.95, 10.255.0.2
X-Forwarded-Host: whoami.pistack.co.za
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 02b0ff6eab73
X-Real-Ip: 10.255.0.2

And our 3rd GET Request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ curl http://whoami.pistack.co.za/
Hostname: 17c817a1813b
IP: 172.18.0.6
IP: 127.0.0.1
IP: 10.0.0.138
IP: 10.0.0.73
GET / HTTP/1.1
Host: whoami.pistack.co.za
User-Agent: curl/7.38.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 165.73.96.95, 10.255.0.2
X-Forwarded-Host: whoami.pistack.co.za
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 31b37f9714d3
X-Real-Ip: 10.255.0.2

Hope this was useful.

Resources:

Building a Raspberry Pi Nginx Image With Caching on Alpine for Docker Swarm

In this guide, we will be creating a nginx reverse proxy with the ability to cache static content using a alpine image.

We will then push the image to gitlab’s private registry, and then run the service on docker swarm.

Create the backend service:

We will upstream to our blog using ghost, which you can deploy using:

1
$ docker service create --name blog --network docknet rbekker87/armhf-ghost:2.0.3

Current File Structure:

Our file structure for the assets we need to build the reverse proxy:

1
2
3
4
5
$ find .
./conf.d
./conf.d/blog.conf
./Dockerfile
./nginx.conf
  • Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
FROM hypriot/rpi-alpine-scratch
MAINTAINER Ruan Bekker

RUN apk update && \
    apk add nginx && \
    rm -rf /etc/nginx/nginx.conf && \
    chown -R nginx:nginx /var/lib/nginx && \
    rm -rf /var/cache/apk/*

ADD nginx.conf /etc/nginx/
ADD conf.d/blog.conf /etc/nginx/conf.d/

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]
  • nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
user nginx;
worker_processes 1;

events {
    worker_connections 1024;
    }

error_log  /var/log/nginx/nginx_error.log warn;

http {

    sendfile        on;
    tcp_nodelay         on;

    gzip              on;
    gzip_http_version 1.0;
    gzip_proxied      any;
    gzip_min_length   500;
    gzip_disable      "MSIE [1-6]\.";
    gzip_types        text/plain text/xml text/css
                      text/comma-separated-values
                      text/javascript
                      application/x-javascript
                      application/atom+xml;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    error_log   /var/log/nginx/error.log;

    proxy_cache_path /var/cache/nginx/ levels=1:2 keys_zone=nginx_cache:5m max_size=128m inactive=60m;

    keepalive_timeout  60;
    server_tokens      off;

    include /etc/nginx/conf.d/*.conf;

}

Hostname resolution to our Ghost Blog Service: In our swarm we have a service called blog which is associated to the docknet network, so the dns resolution will resolve to the vip of the service. As seen in the figure below:

1
2
3
$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                    PORTS
nq42a6jfwx3d        blog                replicated          1/1                 rbekker87/armhf-ghost:2.0.3
  • conf.d/blog.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
upstream ghost_blog {
    server blog:2368;
    }

server {
    listen 80;
    server_name blog.yourdomain.com;

    access_log  /var/log/nginx/blog_access.log  main;
    error_log   /var/log/nginx/blog_error.log;

    location / {

        proxy_cache                 nginx_cache;
        add_header                  X-Proxy-Cache $upstream_cache_status;
        proxy_ignore_headers        Cache-Control;
        proxy_cache_valid any       10m;
        proxy_cache_use_stale       error timeout http_500 http_502 http_503 http_504;

        proxy_pass                  http://ghost_blog;
        proxy_redirect              off;

        proxy_set_header            Host $host;
        proxy_set_header            X-Real-IP $remote_addr;
        proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header            X-Forwarded-Host $server_name;
    }
}

Building the Image and Pushing to Gitlab

I’m using Gitlab in this demonstration, but you can use the registry of your choice:

1
2
3
4
$ docker login registry.gitlab.com
$ docker build -t registry.gitlab.com/user/docker/arm-nginx:caching .
$ docker tag registry.gitlab.com/user/docker/arm-nginx:caching registry.gitlab.com/user/docker/arm-nginx:caching
$ docker push registry.gitlab.com/user/docker/arm-nginx:caching

Deploy

Create the Nginx Reverse Proxy Service on Docker Swarm:

1
2
3
4
5
$ docker service create --name nginx_proxy \
--network docknet \
--publish 80:80 \
--replicas 1 \
--with-registry-auth registry.gitlab.com/user/docker/arm-nginx:caching

Listing our Services:

1
2
3
4
$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                    PORTS
je7x21l7egoh        nginx_proxy         replicated          1/1                 registry.gitlab.com/user/docker/arm-nginx:caching   *:80->80/tcp
nq42a6jfwx3d        blog                replicated          1/1                 rbekker87/armhf-ghost:2.0.3

Once you access your proxy on port 80, you should see your Ghost Blog Homepage like below:

Have a look at the benchmark performance when using Nginx with caching enabled

Resources:

Nginx Caching Performance for Static Content on Docker Swarm With RaspberryPi

The Environment:

I had my Ghost Blog listening on port 2368 and exposing port 80 on Docker so that the port translation directs port 80 traffic to port 2368 on Ghost directly.

Alex responded on my tweet and introduced Nginx Caching:

With this approach benchmarking results was not so great in terms of requests per second, and as this hostname will be only used for a blog, its a great idea to cache the content, this was achieved with the help from Alex’s blog: blog.alexellis.io/save-and-boost-with-nginx/

How Nginx was Configured:

I have a blogpost on how I setup Nginx on an Alpine Image, where I setup caching and proxy-pass the connections through to my ghost blog.

Benchmarking: Before Nginx with Caching was Implemented:

When doing an apache benchmark I got 9.31 requests per second performing the test on my LAN:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
$ ab -n 500 -c 10 http://rbkr.ddns.net/

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking rbkr.ddns.net (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:
Server Hostname:        blog.pistack.co.za
Server Port:            80

Document Path:          /
Document Length:        5470 bytes

Concurrency Level:      10
Time taken for tests:   53.725 seconds
Complete requests:      500
Failed requests:        0
Total transferred:      2863000 bytes
HTML transferred:       2735000 bytes
Requests per second:    9.31 [#/sec] (mean)
Time per request:       1074.501 [ms] (mean)
Time per request:       107.450 [ms] (mean, across all concurrent requests)
Transfer rate:          52.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    2   0.5      2       6
Processing:   685 1068  68.7   1057    1306
Waiting:      683 1067  68.6   1056    1306
Total:        689 1070  68.7   1058    1312

Percentage of the requests served within a certain time (ms)
  50%   1058
  66%   1088
  75%   1102
  80%   1110
  90%   1163
  95%   1218
  98%   1240
  99%   1247
 100%   1312 (longest request)

Benchmarking: After Nginx Caching was Implemented:

After Nginx Caching was Implemented, I got 1067.73 requests per second using apache benchmark over a LAN connection! Absolutely awesome!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
$ ab -n 500 -c 10 http://blog.pistack.co.za/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking blog.pistack.co.za (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:        nginx
Server Hostname:        blog.pistack.co.za
Server Port:            80

Document Path:          /
Document Length:        5470 bytes

Concurrency Level:      10
Time taken for tests:   0.468 seconds
Complete requests:      500
Failed requests:        0
Total transferred:      2880500 bytes
HTML transferred:       2735000 bytes
Requests per second:    1067.73 [#/sec] (mean)
Time per request:       9.366 [ms] (mean)
Time per request:       0.937 [ms] (mean, across all concurrent requests)
Transfer rate:          6007.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        3    4   1.4      4      10
Processing:     3    5   1.6      4      10
Waiting:        2    4   1.6      4      10
Total:          6    9   2.7      8      17

Percentage of the requests served within a certain time (ms)
  50%      8
  66%      8
  75%      9
  80%      9
  90%     15
  95%     15
  98%     15
  99%     16
 100%     17 (longest request)

Resources:

Thanks to Alex Ellis for the suggestion on this, and definitely have a look at blog.alexellis.io as he has some epic content on his blog!

Setting Up a Docker Swarm Cluster on 3 RaspberryPi Nodes

As the curious person that I am, I like to play around with new stuff that I stumble upon, and one of them was having a docker swarm cluster running on 3 Raspberry Pi’s on my LAN.

The idea is to have 3 Raspberry Pi’s (Model 3 B), a Manager Node, and 2 Worker Nodes, each with a 32 GB SanDisk SD Card, which I will also be part of a 3x Replicated GlusterFS Volume that will come in handy later for some data that needs persistent data.

More Inforamtion on: Docker Swarm

Provision Raspbian on each RaspberryPi

Grab the Latest Raspbian Lite ISO and the following source will help provisioning your RaspberryPi with Raspbian.

Installing Docker on Raspberry PI

On each node, run the following to install docker, and also add your user to the docker group, so that you can run docker commands with a normal user:

1
2
3
4
$ apt-get update && sudo apt-get upgrade -y
$ sudo apt-get remove docker.io
$ curl https://get.docker.com | sudo bash
$ sudo usermod -aG docker pi

If you have an internal DNS Server, set an A Record for each node, or for simplicity, set your hosts file on each node so that your hostname for each node responds to it’s provisioned IP Address:

1
2
3
4
$ cat /etc/hosts
192.168.0.2   rpi-01
192.168.0.3   rpi-02
192.168.0.4   rpi-03

Also, to have passwordless SSH, from each node:

1
2
3
4
$ ssh-keygen -t rsa
$ ssh-copy-id rpi-01
$ ssh-copy-id rpi-02
$ ssh-copy-id rpi-03

Initialize the Swarm

Time to set up our swarm. As we have more than one network interface, we will need to setup our swarm by specifying the IP Address of our network interface that is accessible from our LAN:

1
2
3
$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr a1:12:bc:d3:cd:4d
          inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0

Now that we have our IP Address, initialize the swarm on the manager node:

1
2
3
4
5
6
7
8
9
10
pi@rpi-01:~ $ docker swarm init --advertise-addr 192.168.0.2
Swarm initialized: current node (siqyf3yricsvjkzvej00a9b8h) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 \
    192.168.0.2:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Then from rpi-02 join the manager node of the swarm:

1
2
pi@rpi-02:~ $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 192.168.0.2:2377
This node joined a swarm as a worker.

Then from rpi-03 join the manager node of the swarm:

1
2
pi@rpi-03:~ $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 192.168.0.2:2377
This node joined a swarm as a worker.

Then from the manager node: rpi-01, ensure that the nodes are checked in:

1
2
3
4
5
pi@rpi-01:~ $ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
62s7gx1xdm2e3gp5qoca2ru0d     rpi-03              Ready               Active
6fhyfy9yt761ar9pl84dkxck3 *   rpi-01              Ready               Active              Leader
pg0nyy9l27mtfc13qnv9kywe7     rpi-02              Ready               Active

Setting Up a Replicated GlusterFS Volume

I have decided to setup a replicated glusterfs volume to have data replicated throughout the cluster if I would like to have some persistent data. From each node, install the GlusterFS Client and Server:

1
$ sudo apt install glusterfs-server glusterfs-client -y && sudo systemctl enable glusterfs-server

Probe the other nodes from the manager node:

1
2
3
4
5
pi@rpi-01:~ $ sudo gluster peer probe rpi-02
peer probe: success.

pi@rpi-01:~ $ sudo gluster peer probe rpi-03
peer probe: success.

Ensure that we can see all 3 nodes in our GlusterFS Pool:

1
2
3
4
5
pi@rpi-01:~ $ sudo gluster pool list
UUID                                    Hostname        State
778c7463-ba48-43de-9f97-83a960bba99e    rpi-02          Connected
00a20a3c-5902-477e-a8fe-da35aa955b5e    rpi-03          Connected
d82fb688-c50b-405d-a26f-9cb2922cce75    localhost       Connected

From each node, create the directory where GlusterFS will store the data for the bricks that we will specify when creating the volume:

1
2
3
pi@rpi-01:~ $ sudo mkdir -p /gluster/brick
pi@rpi-02:~ $ sudo mkdir -p /gluster/brick
pi@rpi-03:~ $ sudo mkdir -p /gluster/brick

Next, create a 3 Way Replicated GlusterFS Volume:

1
2
3
4
5
6
7
pi@rpi-01:~ $ sudo gluster volume create rpi-gfs replica 3 \
rpi-01:/gluster/brick \
rpi-02:/gluster/brick \
rpi-03:/gluster/brick \
force

volume create: rpi-gfs: success: please start the volume to access data

Start the GlusterFS Volume:

1
2
pi@rpi-01:~ $ sudo gluster volume start rpi-gfs
volume start: rpi-gfs: success

Verify the GlusterFS Volume Info, and from the below output you will see that the volume is replicated 3 ways from the 3 bricks that we specified

1
2
3
4
5
6
7
8
9
10
11
12
pi@rpi-01:~ $ sudo gluster volume info

Volume Name: rpi-gfs
Type: Replicate
Volume ID: b879db15-63e9-44ca-ad76-eeaa3e247623
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: rpi-01:/gluster/brick
Brick2: rpi-02:/gluster/brick
Brick3: rpi-03:/gluster/brick

Mount the GlusterFS Volume on each Node, first on rpi-01:

1
2
3
4
pi@rpi-01:~ $ sudo umount /mnt
pi@rpi-01:~ $ sudo echo 'localhost:/rpi-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
pi@rpi-01:~ $ sudo mount.glusterfs localhost:/rpi-gfs /mnt
pi@rpi-01:~ $ sudo chown -R pi:docker /mnt

Then on rpi-02:

1
2
3
4
pi@rpi-02:~ $ sudo umount /mnt
pi@rpi-02:~ $ sudo echo 'localhost:/rpi-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
pi@rpi-02:~ $ sudo mount.glusterfs localhost:/rpi-gfs /mnt
pi@rpi-02:~ $ sudo chown -R pi:docker /mnt

And lastly on rpi-03:

1
2
3
4
pi@rpi-03:~ $ sudo umount /mnt
pi@rpi-03:~ $ sudo echo 'localhost:/rpi-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
pi@rpi-03:~ $ sudo mount.glusterfs localhost:/rpi-gfs /mnt
pi@rpi-03:~ $ sudo chown -R pi:docker /mnt

Then your GlusterFS Volume will be mounted on all the nodes, and when a file is written to the /mnt/ partition, data will be replicated to all the nodes in the Cluster:

1
2
3
4
pi@rpi-01:~ $ df -h
Filesystem          Size  Used Avail Use% Mounted on
/dev/root            30G  4.5G   24G  16% /
localhost:/rpi-gfs   30G  4.5G   24G  16% /mnt

Create a Web Service on Docker Swarm:

Let’s create a Web Service in our Swarm, called web and by specifying 1 replica and publishing the exposed port 80 to our containers port 80:

1
2
pi@rpi-01:~ $ docker service create --name web --replicas 1 --publish 80:80 hypriot/rpi-busybox-httpd
vsvyanuw6q6yf4jr52m5z7vr1

Verifying that our Service is Started and equals to the desired replica count:

1
2
3
pi@rpi-01:~ $ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                    PORTS
vsvyanuw6q6y        web                 replicated          1/1                 hypriot/rpi-busybox-httpd:latest                         *:891->80/tcp

Inspecting the Service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
pi@rpi-01:~ $ docker service inspect web
[
    {
        "ID": "vsvyanuw6q6yf4jr52m5z7vr1",
        "Version": {
            "Index": 2493
        },
        "CreatedAt": "2017-07-16T21:20:00.017836646Z",
        "UpdatedAt": "2017-07-16T21:20:00.026359794Z",
        "Spec": {
            "Name": "web",
            "Labels": {},
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "hypriot/rpi-busybox-httpd:latest@sha256:c00342f952d97628bf5dda457d3b409c37df687c859df82b9424f61264f54cd1",
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {
                    "Limits": {},
                    "Reservations": {}
                },
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {},
                "ForceUpdate": 0
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 80,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "zjerz0xsw39icnh24enja4cgk",
                    "Addr": "10.255.0.13/16"
                }
            ]
        }
    }
]

Docker Swarm’s Routing mesh takes care of the internal routing, so requests will respond even if the container is not running on the node that you are making the request against.

With that said, verifying on which node our service is running:

1
2
3
pi@rpi-01:~ $ docker service ps web
ID                  NAME                IMAGE                              NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
sd67cd18s5m0        web.1               hypriot/rpi-busybox-httpd:latest   rpi-02              Running             Running 2 minutes ago

When we make a HTTP Request to one of these Nodes IP Addresses, our request will be responded with this awesome static page:

We can see we only have one container in our swarm, let’s scale that up to 3 containers:

1
2
pi@rpi-01:~ $ docker service scale web01=3
web01 scaled to 3

Now that the service is scaled to 3 containers, requests will be handled using the round-robin algorithm. To ensured that the service scaled, we will see that we will have 3 replicas:

1
2
3
pi@rpi-01:~ $ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                    PORTS
vsvyanuw6q6y        web                 replicated          3/3                 hypriot/rpi-busybox-httpd:latest                         *:891->80/tcp

Verifying where these containers are running on:

1
2
3
4
5
pi@rpi-01:~ $ docker service ps web01
ID                  NAME                IMAGE                              NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
sd67cd18s5m0        web.1               hypriot/rpi-busybox-httpd:latest   rpi-02              Running             Running 2 minutes ago
ope3ya7hh9j4        web.2               hypriot/rpi-busybox-httpd:latest   rpi-03              Running             Running 30 seconds ago
07m1ww7ptxro        web.3               hypriot/rpi-busybox-httpd:latest   rpi-01              Running             Running 28 seconds ago

Lastly, removing the service from our swarm:

1
2
pi@rpi-01:~ $ docker service rm web01
web01

Massive Thanks:

a Massive thanks to Alex Ellis for mentioning me on one of his blogposts:

My PiStack Blog Proudly Hosted on My RaspberryPi Swarm Cluster

This is a repost of my first blogpost which is hosted on my Raspberry Pi Cluster (04 July 2017), that runs Docker Swarm and is served from my Home in South Africa, and can be accessed on http://blog.pistack.co.za

Just Look at It!

  • 3x Raspberry Pi 3 Model B
  • Quad Core 1.2GHz Broadcom BCM2837 64bit CPU
  • 1GB RAM
  • BCM43438 wireless LAN and Bluetooth Low Energy (BLE) on board
  • 3x 32GB Sandisk SD Cards (Replicated GlusterFS Volume for /gluster partition)
  • Upgraded switched Micro USB power source up to 2.5A

My Setup:

I have 3x Raspberrypi 3’s, each with a 32GB SanDisk SD Card, formatted with Raspbian Jessie Lite, powered by a 6 Port USB Hub and networked with a Totolink 5 Port Gigabit Switch, but note that: the Rpi does not support Gigabit Networking

For persistent storage I have setup a Replicated GlusterFS Volume across the 3 nodes.

More details on how I did the setup, can be found from the Setting Up a Docker Swarm Cluster on RaspberryPi Nodes blog post.

Thanks!

Thanks for the visit, I will blog about awesome Docker and RaspberryPi related stuff as my mind stumble along awesome ideas :)