Ruan Bekker's Blog

From a Curious mind to Posts on Github

Running Java Web Applications on Tomcat With Docker Swarm

From this post we used Payara Micro to Setup a Web Application, and a full example was provided on how to create a war file that will be used for the deployment.

Today we will be using Tomcat to deploy the same application. The official repository can be found on hub.docker.com/_/tomcat .

Our Dockerfile for our Own Tomcat Image:

The Dockerfile is modified a bit (CATALINA_OPTS) to be able to pass JVM environment variables, but if you would like to use the standard image you can skip this and just use the image from their repository.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
FROM openjdk:8-jre-alpine

ENV CATALINA_HOME /usr/local/tomcat
ENV PATH $CATALINA_HOME/bin:$PATH
RUN mkdir -p "$CATALINA_HOME"
WORKDIR $CATALINA_HOME
ENV CATALINA_OPTS -Xmx768m -Xms512m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
# let "Tomcat Native" live somewhere isolated
ENV TOMCAT_NATIVE_LIBDIR $CATALINA_HOME/native-jni-lib
ENV LD_LIBRARY_PATH ${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}$TOMCAT_NATIVE_LIBDIR

RUN apk add --no-cache gnupg

# see https://www.apache.org/dist/tomcat/tomcat-$TOMCAT_MAJOR/KEYS
# see also "update.sh" (https://github.com/docker-library/tomcat/blob/master/update.sh)
ENV GPG_KEYS 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
RUN set -ex; \
  for key in $GPG_KEYS; do \
      gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$key"; \
  done

ENV TOMCAT_MAJOR 8
ENV TOMCAT_VERSION 8.5.23
ENV TOMCAT_SHA1 1ba27c1bb86ab9c8404e98068800f90bd662523c

ENV TOMCAT_TGZ_URLS \
# https://issues.apache.org/jira/browse/INFRA-8753?focusedCommentId=14735394#comment-14735394
  https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz \
# if the version is outdated, we might have to pull from the dist/archive :/
  https://www-us.apache.org/dist/tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz \
  https://www.apache.org/dist/tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz \
  https://archive.apache.org/dist/tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz

ENV TOMCAT_ASC_URLS \
  https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz.asc \
# not all the mirrors actually carry the .asc files :'(
  https://www-us.apache.org/dist/tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz.asc \
  https://www.apache.org/dist/tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz.asc \
  https://archive.apache.org/dist/tomcat/tomcat-$TOMCAT_MAJOR/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz.asc

RUN set -eux; \
  \
  apk add --no-cache --virtual .fetch-deps \
      ca-certificates \
      openssl \
  ; \
  \
  success=; \
  for url in $TOMCAT_TGZ_URLS; do \
      if wget -O tomcat.tar.gz "$url"; then \
          success=1; \
          break; \
      fi; \
  done; \
  [ -n "$success" ]; \
  \
  echo "$TOMCAT_SHA1 *tomcat.tar.gz" | sha1sum -c -; \
  \
  success=; \
  for url in $TOMCAT_ASC_URLS; do \
      if wget -O tomcat.tar.gz.asc "$url"; then \
          success=1; \
          break; \
      fi; \
  done; \
  [ -n "$success" ]; \
  \
  gpg --batch --verify tomcat.tar.gz.asc tomcat.tar.gz; \
  tar -xvf tomcat.tar.gz --strip-components=1; \
  rm bin/*.bat; \
  rm tomcat.tar.gz*; \
  \
  nativeBuildDir="$(mktemp -d)"; \
  tar -xvf bin/tomcat-native.tar.gz -C "$nativeBuildDir" --strip-components=1; \
  apk add --no-cache --virtual .native-build-deps \
      apr-dev \
      coreutils \
      dpkg-dev dpkg \
      gcc \
      libc-dev \
      make \
      "openjdk${JAVA_VERSION%%[-~bu]*}"="$JAVA_ALPINE_VERSION" \
      openssl-dev \
  ; \
  ( \
      export CATALINA_HOME="$PWD"; \
      cd "$nativeBuildDir/native"; \
      gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; \
      ./configure \
          --build="$gnuArch" \
          --libdir="$TOMCAT_NATIVE_LIBDIR" \
          --prefix="$CATALINA_HOME" \
          --with-apr="$(which apr-1-config)" \
          --with-java-home="$(docker-java-home)" \
          --with-ssl=yes; \
      make -j "$(nproc)"; \
      make install; \
  ); \
  runDeps="$( \
     scanelf --needed --nobanner --format '%n#p' --recursive "$TOMCAT_NATIVE_LIBDIR" \
         | tr ',' '\n' \
         | sort -u \
         | awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' \
 )"; \
  apk add --virtual .tomcat-native-rundeps $runDeps; \
  apk del .fetch-deps .native-build-deps; \
  rm -rf "$nativeBuildDir"; \
  rm bin/tomcat-native.tar.gz; \
  \
# sh removes env vars it doesn't support (ones with periods)
# https://github.com/docker-library/tomcat/issues/77
  apk add --no-cache bash; \
  find ./bin/ -name '*.sh' -exec sed -ri 's|^#!/bin/sh$|#!/usr/bin/env bash|' '{}' +

# verify Tomcat Native is working properly
RUN set -e \
  && nativeLines="$(catalina.sh configtest 2>&1)" \
  && nativeLines="$(echo "$nativeLines" | grep 'Apache Tomcat Native')" \
  && nativeLines="$(echo "$nativeLines" | sort -u)" \
  && if ! echo "$nativeLines" | grep 'INFO: Loaded APR based Apache Tomcat Native library' >&2; then \
      echo >&2 "$nativeLines"; \
      exit 1; \
  fi

EXPOSE 8080
CMD ["catalina.sh", "run"]

Building our Image and Pusing to our Registry:

1
2
$ docker build -t registry.gitlab.com/<user>/<repo>/<image>:<tag>
$ docker push registry.gitlab.com/<user>/<repo>/<image>:<tag>

Dockerfile for our Application:

Now that we have built our image for Tomcat, we can write our Dockerfile for our application, note that the hello.war file also needs to be in the same working directory, unless written otherwise:

1
2
FROM registry.gitlab.com/<user>/<repo>/<image>:<tag>
COPY hello.war /usr/local/tomcat/webapps/hello.war

Setup the Compose file for our Stack:

We will use docker stack to deploy our application, note that I have Traefik that acts as my reverse proxy.

Below, our app.yml compose file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: '3'

services:
  hello:
    image: registry.gitlab.com/<user>/<repo>/<image>:<tag>
    networks:
      - appnet
    deploy:
      labels:
        - "traefik.port=8080"
        - "traefik.docker.network=appnet"
        - "traefik.frontend.rule=Host:apps.mydomain.com; PathPrefix: /hello/"
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - 'node.role==worker'

networks:
  appnet:
    external: true

Deploy our Application:

From our compose file we defined that our network is external, so if you are using the same name, and you have not yet setup the overlay network:

1
$ docker network create --driver overlay appnet

Now deploy the stack:

1
$ docker stack deploy --compose-file app.yml apps

Testing our Application:

1
2
3
4
5
6
7
8
9
$ curl http://apps.mydomain.com/hello/

<!DOCTYPE html>
<html>
            Hello World!
   Test Page with Docker + Payara Micro</h3>

   Serving From ContainerId: d24f8cd982fc
</html>

Managing Traefik Configuration With Consul on Docker Swarm

Today we will Setup Consul with Traefik on Docker Swarm

Resources:

Create Consul in the Swarm:

Investigate using Consul with Traefik in Docker Swarm.

I have configured consul with constraints to be placed only on my one manager, as I was bind mounting the data directory to /mnt, as this was for testing on a small docker swarm cluster, but with Docker for AWS, we will use cloudstor with EFS, or GlusterFS, NFS for data persistency across any nodes.

Create Compose Files:

  • consul.yml
  • traefik.yml
  • apps.yml

consul.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ cat > consul.yml << EOF
version: '3.3'

services:
  consul:
    image: progrium/consul
    command: -server -bootstrap -log-level debug -ui-dir /ui
    networks:
      - appnet
    deploy:
      placement:
        constraints: [node.role == manager]
    volumes:
      - type: bind
        source: /mnt/consul
        target: /data
    ports:
      - "8400:8400"
      - "8500:8500"
      - "8600:53/udp"

networks:
  appnet:
    external: true

EOF

traefik.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ cat > traefik.yml << EOF
version: '3.3'

services:
  traefik:
    image: traefik
    networks:
      - appnet
    command: --consul --consul.endpoint=consul:8500
    ports:
      - "80:80"
      - "8080:8080"

networks:
  appnet:
    external: true

EOF

apps.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
$ cat > apps.yml << EOF
version: '3.3'

services:
  whoami1:
    image: emilevauge/whoami
    networks:
      - appnet

  whoami2:
    image: emilevauge/whoami
    networks:
      - appnet

  whoami3:
    image: emilevauge/whoami
    networks:
      - appnet

  whoami4:
    image: emilevauge/whoami
    networks:
      - appnet

  whoami5:
    image: rbekker87/flask-containername
    networks:
      - appnet

  whoami6:
    image: rbekker87/flask-containername
    networks:
      - appnet

networks:
  appnet:
    external: true

EOF

Create Overlay Network and Deploy Stacks

Create the overlay network, and deploy the 3 stacks:

1
2
3
4
$ docker network create --driver=overlay appnet
$ docker stack deploy --compose-file consul.yml kvstore
$ docker stack deploy --compose-file traefik.yml proxy
$ docker stack deploy --compose-file apps.yml apps

Populate Configs and Push to Consul KV Store:

Config for Traefik:

1
2
3
4
5
6
7
8
9
$ cat > create_traefik_config.sh << EOF
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/loglevel -d 'DEBUG'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/defaultentrypoints/0 -d 'http'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/entrypoints/http/address -d ':80'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/consul/endpoint -d 'consul:8500'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/consul/watch -d 'true'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/consul/prefix -d 'traefik'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/web/address -d ':8081'
EOF

Config for WhoAmI Web Apps:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ cat > create_whoami_config.sh << EOF
# backend-1
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend1/circuitbreaker/expression -d 'NetworkErrorRatio() > 0.5'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend1/servers/server1/url -d 'http://whoami1:80'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend1/servers/server1/weight -d '10'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend1/servers/server2/url -d 'http://whoami2:80'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend1/servers/server2/weight -d '1'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend1/servers/server2/tags -d 'api,helloworld'

# backend-2
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/maxconn/amount -d '10'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/maxconn/extractorfunc -d 'request.host'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/loadbalancer/method -d 'drr'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/servers/server1/url -d 'http://whoami3:80'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/servers/server1/weight -d '1'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/servers/server2/url -d 'http://whoami4:80'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/servers/server2/weight -d '2'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend2/servers/server2/tags -d 'web'

# frontend-1
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend1/backend -d 'backend2'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend1/routes/test_1/rule -d 'Host:test.localhost'

# frontend-2
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend2/backend -d 'backend1'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend2/passHostHeader -d 'true'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend2/priority -d '10'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend2/entrypoints -d 'http'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend2/routes/test_2/rule -d 'PathPrefix:/test'
EOF

Config for Flask Container Name Web Apps:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ cat > create_flask_config.sh << EOF
# backends
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/amount -d '5'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/maxconn/extractorfunc -d 'request.host'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/loadbalancer/method -d 'drr'

curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/servers/server1/url -d 'http://whoami5:5000'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/servers/server1/weight -d '1'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/servers/server1/tags -d 'flask'

curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/servers/server2/url -d 'http://whoami6:5000'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/servers/server2/weight -d '2'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/servers/server2/tags -d 'flask'

# frontend:
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend3/backend -d 'backend3'
curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend3/routes/test_1/rule -d 'Host:flask.localhost'
EOF

Push Configs to Consul:

1
2
3
$ sh create_traefik_config.sh
$ sh create_whoami_config.sh
$ sh create_flask_config.sh

Testing Applications:

Test Frontend with Host Header: test.localhost

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ curl -H "Host:test.localhost" http://127.0.0.1:80
Hostname: 88c29de3aeb0
IP: 127.0.0.1
IP: 10.0.0.6
IP: 10.0.0.7
IP: 172.18.0.3
GET / HTTP/1.1
Host: test.localhost
User-Agent: curl/7.47.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.255.0.2
X-Forwarded-Host: test.localhost
X-Forwarded-Proto: http
X-Forwarded-Server: 2f827d04fbfb

Test Frontend with PathPrefix: /test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ curl http://127.0.0.1:80/test
Hostname: 14bd4dc0ab00
IP: 127.0.0.1
IP: 10.0.0.12
IP: 10.0.0.13
IP: 172.18.0.4
GET /test HTTP/1.1
Host: 127.0.0.1
User-Agent: curl/7.47.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.255.0.2
X-Forwarded-Host: 127.0.0.1
X-Forwarded-Proto: http
X-Forwarded-Server: 2f827d04fbfb

Test with failure expected:

1
2
3
4
5
$ curl http://127.0.0.1:80
404 page not found

$ curl -H "Host:foo.localhost" http://127.0.0.1:80
404 page not found

Change the frontend rule to foo.localhost and test again:

1
$ curl -XPUT http://127.0.0.1:8500/v1/kv/traefik/frontends/frontend1/routes/test_1/rule -d 'Host:foo.localhost'

Testing with foo.localhost :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ curl -H "Host:foo.localhost" http://127.0.0.1:80
Hostname: 88c29de3aeb0
IP: 127.0.0.1
IP: 10.0.0.6
IP: 10.0.0.7
IP: 172.18.0.3
GET / HTTP/1.1
Host: test.localhost
User-Agent: curl/7.47.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.255.0.2
X-Forwarded-Host: foo.localhost
X-Forwarded-Proto: http
X-Forwarded-Server: 2f827d04fbfb

Test Flask Web Apps, with RoundRobin + Weight:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: c94e41420ec7 , UUID: 8d536277-1041-4140-b28a-91630d69ab15

$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: 786a3b15a32e , UUID: 50bb435f-dac3-4cb0-8ecf-250f13a4d7a5

$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: c94e41420ec7 , UUID: 680ca88c-8048-4044-a31b-1e922545dc8c

$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: c94e41420ec7 , UUID: 5506792c-04e5-4517-b58e-fad57b5d1da5

$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: 786a3b15a32e , UUID: 930056e5-4667-4c2c-976b-246cf891a351

$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: c94e41420ec7 , UUID: 7c757b50-0e3a-47e2-8636-ae0583802afb

$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: c94e41420ec7 , UUID: 918bf337-1476-480c-aacb-72fc89392c45

$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: 786a3b15a32e , UUID: 5c2c6a0e-f2ea-4d5f-a6ce-a7df2ef4886b

Data Persistent Test:

List the Stacks:

1
2
3
4
5
$ docker stack ls
NAME                SERVICES
apps                6
kvstore             1
proxy               1

Kill the Consul Container to ensure data is persistent:

1
$ docker kill $(docker ps -f name=consul -q)

Verify that the replica has been fulfilled to its desired state:

1
2
3
4
5
$ docker stack ps kvstore
ID                  NAME                   IMAGE                    NODE                DESIRED STATE       CURRENT STATE                ERROR                         PORTS
j80kxxei6lyx        kvstore_consul.1       progrium/consul:latest   ip-10-1-4-51        Running             Running about a minute ago
lsvww0z8g24c         \_ kvstore_consul.1   progrium/consul:latest   ip-10-1-4-51        Shutdown            Failed about a minute ago    "task: non-zero exit (137)"
amkmsfodslwk         \_ kvstore_consul.1   progrium/consul:latest   ip-10-1-4-51        Shutdown            Shutdown 33 minutes ago

Read the Value of the Backend3 URL Key:

1
2
$ curl -XGET http://127.0.0.1:8500/v1/kv/traefik/backends/backend3/servers/server1/url?raw
http://whoami5:5000

`

Test The Service:

1
2
$ curl -H "Host:flask.localhost" http://127.0.0.1:80
Container Hostname: 786a3b15a32e , UUID: 4cf985e0-3f47-4320-ac38-42745b00ba1e

Inspect Services:

Consul:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
$ docker service inspect kvstore_consul
[
    {
        "ID": "ppe2c1ld5eyvby6x62fr649z8",
        "Version": {
            "Index": 2097
        },
        "CreatedAt": "2017-10-03T12:40:27.342669337Z",
        "UpdatedAt": "2017-10-03T12:55:22.93080059Z",
        "Spec": {
            "Name": "kvstore_consul",
            "Labels": {
                "com.docker.stack.image": "progrium/consul",
                "com.docker.stack.namespace": "kvstore"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "progrium/consul:latest@sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274",
                    "Labels": {
                        "com.docker.stack.namespace": "kvstore"
                    },
                    "Args": [
                        "-server",
                        "-bootstrap",
                        "-log-level",
                        "debug",
                        "-ui-dir",
                        "/ui"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/mnt/consul",
                            "Target": "/data"
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Constraints": [
                        "node.role == manager"
                    ],
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "5q3puzw9pfxa5gokx2mx1kn9j",
                        "Aliases": [
                            "consul"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8400,
                        "PublishedPort": 8400,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8500,
                        "PublishedPort": 8500,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "udp",
                        "TargetPort": 53,
                        "PublishedPort": 8600,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "PreviousSpec": {
            "Name": "kvstore_consul",
            "Labels": {
                "com.docker.stack.image": "progrium/consul",
                "com.docker.stack.namespace": "kvstore"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "progrium/consul:latest@sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274",
                    "Labels": {
                        "com.docker.stack.namespace": "kvstore"
                    },
                    "Args": [
                        "-server",
                        "-bootstrap",
                        "-log-level",
                        "debug",
                        "-ui-dir",
                        "/ui"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/mnt/consul",
                            "Target": "/data"
                        }
                    ]
                },
                "Resources": {},
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "5q3puzw9pfxa5gokx2mx1kn9j",
                        "Aliases": [
                            "consul"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8400,
                        "PublishedPort": 8400,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8500,
                        "PublishedPort": 8500,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "udp",
                        "TargetPort": 53,
                        "PublishedPort": 8600,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8400,
                        "PublishedPort": 8400,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8500,
                        "PublishedPort": 8500,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "udp",
                        "TargetPort": 53,
                        "PublishedPort": 8600,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 8400,
                    "PublishedPort": 8400,
                    "PublishMode": "ingress"
                },
                {
                    "Protocol": "tcp",
                    "TargetPort": 8500,
                    "PublishedPort": 8500,
                    "PublishMode": "ingress"
                },
                {
                    "Protocol": "udp",
                    "TargetPort": 53,
                    "PublishedPort": 8600,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "zz458844j2msbqa6a1g2es8re",
                    "Addr": "10.255.0.5/16"
                },
                {
                    "NetworkID": "5q3puzw9pfxa5gokx2mx1kn9j",
                    "Addr": "10.0.0.2/24"
                }
            ]
        },
        "UpdateStatus": {
            "State": "completed",
            "StartedAt": "2017-10-03T12:55:09.724524449Z",
            "CompletedAt": "2017-10-03T12:55:22.930760766Z",
            "Message": "update completed"
        }
    }
]

Traefik:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
$ docker service inspect proxy_traefik
[
    {
        "ID": "oqb0lyiprpwby9xkb4n1mn5kl",
        "Version": {
            "Index": 2037
        },
        "CreatedAt": "2017-10-03T12:40:37.696749025Z",
        "UpdatedAt": "2017-10-03T12:40:37.698038856Z",
        "Spec": {
            "Name": "proxy_traefik",
            "Labels": {
                "com.docker.stack.image": "traefik",
                "com.docker.stack.namespace": "proxy"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "traefik:latest@sha256:90697fb79a104520f350a3a1db6402584f473301ab6d1a71d264758b65fa232e",
                    "Labels": {
                        "com.docker.stack.namespace": "proxy"
                    },
                    "Args": [
                        "--consul",
                        "--consul.endpoint=consul:8500"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        },
                        {
                            "OS": "linux"
                        },
                        {
                            "Architecture": "arm64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "5q3puzw9pfxa5gokx2mx1kn9j",
                        "Aliases": [
                            "traefik"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8080,
                        "PublishedPort": 8080,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 8080,
                        "PublishedPort": 8080,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 80,
                    "PublishMode": "ingress"
                },
                {
                    "Protocol": "tcp",
                    "TargetPort": 8080,
                    "PublishedPort": 8080,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "zz458844j2msbqa6a1g2es8re",
                    "Addr": "10.255.0.7/16"
                },
                {
                    "NetworkID": "5q3puzw9pfxa5gokx2mx1kn9j",
                    "Addr": "10.0.0.4/24"
                }
            ]
        }
    }
]

Flask App:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
$ docker service inspect apps_whoami5
[
    {
        "ID": "hy0443u03j8pygaegdorsrjot",
        "Version": {
            "Index": 2052
        },
        "CreatedAt": "2017-10-03T12:41:06.582341297Z",
        "UpdatedAt": "2017-10-03T12:41:06.583398389Z",
        "Spec": {
            "Name": "apps_whoami5",
            "Labels": {
                "com.docker.stack.image": "rbekker87/flask-containername",
                "com.docker.stack.namespace": "apps"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "rbekker87/flask-containername:latest@sha256:fa4dc5905a10130d4309ffbc877155b9f61956980dc51ee2eaa16ac4255bcc2b",
                    "Labels": {
                        "com.docker.stack.namespace": "apps"
                    },
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "5q3puzw9pfxa5gokx2mx1kn9j",
                        "Aliases": [
                            "whoami5"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip"
            },
            "VirtualIPs": [
                {
                    "NetworkID": "5q3puzw9pfxa5gokx2mx1kn9j",
                    "Addr": "10.0.0.8/24"
                }
            ]
        }
    }
}

Basic Example With Python to Create a Module That Consists of Classes and Functions

Just a very basic example how to create a Python Module that consists of a Single Class and 2 basic functions.

Our main app will can our module to print out a word, that we pass to our first function.

The Directory Setup:

Below is a tree view of my current working directory:

1
2
3
4
5
6
7
$ tree
.
├── providers
│   ├── __init__.py
│   ├── test.py
├── README.md
└── main.py

In order to make a python file a module, we need to have a blank __init__.py file in our directory. So any files under our providers directory will be seen as modules from our main.py file.

Our Test Module:

in our providers/test.py file:

1
2
3
4
5
6
7
8
class TestClass:

    def word_to_return(self, word_value):
        return word_value

    def simple_test(self):
        data = self.word_to_return('its me!')
        return data

Then our providers/test.py file will be blank.

Our main.py, we will import our test module, instantiate our class, and call our function within the class that we instantiated:

1
2
3
4
5
6
from providers import test

test_instance = test.TestClass()
response = test_instance.simple_test()

print(response)

instead of response = test_instance.simple_test(), you could also do print(test_instance.simple_test()

Testing it out:

1
2
$ python main.py
its me!

It’s very basic but will post some more topics around this in the future.

Also note, this blog is for quick posts that I come accross during my daily doings, for more details tutorials have a look at my main blog: sysadmins.co.za

Python Script to Decrypt Encrypted Data With AWS KMS

Quick script to decrypt data that was encrypted with your KMS key:

The Script:

The script requires the encrypted scring as an argument:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/usr/bin/env python

import boto3
import sys
from base64 import b64decode

try:
    encrypted_value = sys.argv[1]
except IndexError:
    print("Usage: {} {}".format(sys.argv[0], 'the-encrypted-string'))
    exit(1)

session = boto3.Session(
        region_name='eu-west-1',
        profile_name='default'
    )

kms = session.client('kms')

response = kms.decrypt(CiphertextBlob=b64decode(encrypted_value))['Plaintext']
print("Decrypted Value: {}".format(response))

Change the permissions so that the file is executable:

1
$ chmod +x decrypt.py

Usage:

1
2
$ ./decrypt.py asdlaskjdasidausd09q3uoijad09ujd38u309
Decrypted Value: thisIsMyDecryptedValue

Setup Kerberos Server and Client on Ubuntu

Kerberos is a authentication protocol that provides a centralized authentication server, that works with the concepts of tickets that are encrypted.

Today we will setup a Kerberos Server (KDC) and setup and Kerberos Enabled Client, and then testing our setup by obtaining a Kerberos Ticket from our server.

Setup the Server:

Install Kerberos KDC and Admin Server:

1
2
3
$ apt update && apt upgrade -y
$ apt install krb5-kdc krb5-admin-server krb5-config -y
$ krb5_newrealm

You will be prompted for realm, and hostnames, in my case I have setup the following:

  • REALM: LAN.RUANBEKER.COM
  • HOST: localhost
  • ADMIN_SERVER: localhost

Then our master password:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
This script should be run on the master KDC/admin server to initialize
a Kerberos realm.  It will ask you to type in a master key password.
This password will be used to generate a key that is stored in
/etc/krb5kdc/stash.  You should try to remember this password, but it
is much more important that it be a strong password than that it be
remembered.  However, if you lose the password and /etc/krb5kdc/stash,
you cannot decrypt your Kerberos database.
Loading random data
Initializing database '/var/lib/krb5kdc/principal' for realm 'LAN.RUANBEKKER.COM',
master key name 'K/M@LAN.RUANBEKKER.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

The output:

1
2
3
4
5
6
7
8
9
10
11
12
Now that your realm is set up you may wish to create an administrative
principal using the addprinc subcommand of the kadmin.local program.
Then, this principal can be added to /etc/krb5kdc/kadm5.acl so that
you can use the kadmin program on other computers.  Kerberos admin
principals usually belong to a single user and end in /admin.  For
example, if jruser is a Kerberos administrator, then in addition to
the normal jruser principal, a jruser/admin principal should be
created.

Don't forget to set up DNS information so your clients can find your
KDC and admin servers.  Doing so is documented in the administration
guide.

Uncomment the last line which contains admin:

1
$ vi /etc/krb5kdc/kadm5.acl

a Kerberos principal is a unique identity to which Kerberos can assign tickets, lets add our first principal, james:

1
2
3
4
5
6
7
8
9
$ kadmin.local
Authenticating as principal root/admin@LAN.RUANBEKKER.COM with password.
kadmin.local:  addprinc james

WARNING: no policy specified for james@LAN.RUANBEKKER.COM; defaulting to no policy
Enter password for principal "james@LAN.RUANBEKKER.COM":
Re-enter password for principal "james@LAN.RUANBEKKER.COM":
Principal "james@LAN.RUANBEKKER.COM" created.
kadmin.local:  exit

Setup the Client:

Setup a Host Entry:

1
$ echo '10.1.1.1 kdc.lan.ruanbekker.com kdc' >> /etc/hosts

Setup Kerberos Client:

1
2
3
4
$ apt install krb5-user -y
- realm
- hostname
- hostname

Obtain a Ticket from the Server:

1
2
3
4
5
6
7
8
9
10
$ kinit -p james
Password for james@LAN.RUANBEKKER.COM:

$ klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: james@LAN.RUANBEKKER.COM

Valid starting     Expires            Service principal
10/18/17 22:13:34  10/19/17 08:13:34  krbtgt/LAN.RUANBEKKER.COM@LAN.RUANBEKKER.COM
  renew until 10/19/17 22:13:30

Resources:

Using Python to Build a Dictionary From Data Eg Sports Per Person

I had to achieve a way to provide data in key-value format, where I wanted to see what sports people like, eg: {"ruan": ["rugby", "cricket"]}

The Idea

So my idea was to have the name as the key, and the sports as the value in a list.

Some Catches

So for this post, I will be setting the data statically in the code, while at the time I was working data that was returned via a API.

I am looping through each occurence, adding the name, and when the name exists, I append the sport to the list of the person.

The catch was that, if there was any duplicated data, the person will only exists once in the dictionary that I am building, but the sport will be appended, so if there were 2 occurences of rugby it will show the sport 2 times. So I had to put some logic into the code to handle that.

The Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
"""|Info:

Printing Sports per Person, by looping through data, appending the sports to a list per person, which gets added to our dictionary.

Variables:
 group {dict} -- "the dictionary that we are building up"
 people {list} -- "list of people with their sport choices"
 for sportman in people: {[for-loop]} -- "iterating through our data, if the sport exists, continue, if not, apeend it to the list"
 print(group) {[dict]} -- "printing the results"
"""

group = {}
people = [
  {
      "name": "ruan",
      "sport": "cricket"
  },
  {
      "name": "stefan",
      "sport": "rugby"
  },
  {
      "name": "stefan",
      "sport": "cricket"
  },
  {
      "name": "james",
      "sport": "rugby"
  },
  {
      "name": "james",
      "sport": "golf"
  },
  {
      "name": "stefan",
      "sport": "rugby"
  },
  {
      "name": "james",
      "sport": "hockey"
  }
]

for sportman in people:
    if sportman['name'] in group:
        if sportman['sport'] not in group[sportman['name']]:
            group[sportman['name']].append(sportman['sport'])
        else:
            pass
    else:
        group[sportman['name']] = []
        group[sportman['name']].append(sportman['sport'])

print(group)

Running the Script:

When running the script results in the following:

1
2
$ python sports.py
{'james': ['rugby', 'golf', 'hockey'], 'ruan': ['cricket'], 'stefan': ['rugby', 'cricket']}

Customize Ubuntu 16 Desktop With Arc Dark Theme

So I was running ApricityOS for quite some time, which is a Arch Distibution. But a couple of hours before PyConZa I was trying to do a update and found that their repositories were reporting 404 errors, and turns out they have stopped their project :( . I quite liked ApricityOS, as it’s what you will expect when installing Arch with the basic applications and the numix icon/theme pack.

So I decided to use Ubuntu for a change.

Customizations:

For the Operating System, I am Ubuntu 16:

For my theme I am using Arc Dark:

Moka Icon pack for my Icons:

Terminator for my terminal of choice:

And my config:

~/.config/terminator/config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
[global_config]
  enabled_plugins = LaunchpadCodeURLHandler, APTURLHandler, LaunchpadBugURLHandler
  sticky = True
  window_state = maximise
[keybindings]
[layouts]
  [[default]]
    [[[child1]]]
      parent = window0
      profile = default
      type = Terminal
    [[[window0]]]
      parent = ""
      type = Window
  [[multi]]
    foreground_color = "#ffffff"
    palette = "#62b9d6:#cc0000:#4e9a06:#c4a000:#3465a4:#75507b:#06989a:#d3d7cf:#77c529:#ef2929:#8ae234:#fce94f:#729fcf:#ad7fa8:#34e2e2:#eeeeec"
    [[[child0]]]
      order = 0
      parent = ""
      position = 0:25
      size = 1366, 768
      type = Window
    [[[child1]]]
      order = 0
      parent = child0
      position = 632
      type = HPaned
    [[[child2]]]
      order = 0
      parent = child1
      position = 354
      type = VPaned
    [[[child5]]]
      order = 1
      parent = child1
      position = 354
      type = VPaned
    [[[terminal3]]]
      command = top; bash
      order = 0
      parent = child2
      profile = default
      type = Terminal
    [[[terminal4]]]
      command = uptime; bash
      order = 1
      parent = child2
      profile = default
      type = Terminal
    [[[terminal6]]]
      command = cd ~/workspace; bash
      order = 0
      parent = child5
      profile = default
      type = Terminal
    [[[terminal7]]]
      command = cd ~/workspace; bash
      order = 1
      parent = child5
      profile = default
      type = Terminal
  [[simples]]
    [[[child0]]]
      order = 0
      parent = ""
      position = 0:25
      size = 715, 694
      type = Window
    [[[child1]]]
      order = 0
      parent = child0
      position = 348
      type = VPaned
    [[[terminal2]]]
      command = cd ~/workspace; bash
      order = 0
      parent = child1
      profile = default
      type = Terminal
    [[[terminal3]]]
      command = cd ~/workspace; bash
      order = 1
      parent = child1
      profile = default
      type = Terminal
[plugins]
[profiles]
  [[default]]
    background_image = None
    icon_bell = False
    scrollback_infinite = True

And to force color prompts:

~/.bashrc
1
force_color_prompt=yes

And my Wallpaper:

Other Preferred Applications:

I will update this page as I’m getting new apps or modifications

Display PHP Content Through HTML Files

While I was working with a root index page which is in HTML that had PHP content in, it did not render all of the PHP and some content was displayed as text, instead of rendered.

The Issue:

Some PHP content not rendered, as I am seeing this at the top of the page as plain text:

1
; somePhpFunction(); ?>

My Nginx Config:

nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
http {
    include                     /etc/nginx/mime.types;
    default_type                application/octet-stream;
    sendfile                    on;
    access_log                  /var/log/nginx/access.log;
    keepalive_timeout           3000;

    proxy_cache_path /var/cache/nginx/ levels=1:2 keys_zone=nginx_cache:10m max_size=16m inactive=60m;

    server {
        listen                  80;
        root                    /www;
        index                   index.php index.html index.htm;
        server_name             _;
        client_max_body_size    32m;
        error_page              500 502 503 504  /50x.html;
        proxy_cache       nginx_cache;
        add_header        X-Proxy-Cache "public";

        location = /50x.html {
              root              /var/lib/nginx/html;
        }

        location ~ \.php$ {
              fastcgi_pass      127.0.0.1:9000;
              fastcgi_index     index.php;
              include           fastcgi.conf;
        }
    }
}

The Fix:

a .htaccess had to be placed in my root directory of the website:

.htaccess
1
AddType application/x-httpd-php .html .htm

After that was in place, all of the PHP was rendered

Sending Mail With SSMTP on Alpine Linux

Quick Post on how to use ssmtp on Alpine Linux to Send Mail:

Update and Install SSMTP

1
2
$ apk update
$ apk add ssmtp

Configure SSMTP

1
2
3
4
5
6
$ cat > /etc/ssmtp/ssmtp.conf << EOF
root=postmaster
mailhub=mail.domain.com:25
hostname=`hostname`
FromLineOverride=YES
EOF

Create the Mail Content

1
2
3
4
5
6
7
$ cat > mail.txt << EOF
To: recipient@domain.com
From: sender@domain.com
Subject: Mail with SSMTP

Hello, this is a test mail.
EOF

Testing Mail Delivery

1
$ ssmtp recipient@domain.com < file.txt

Related:

Backup and Restore Mutliple Collections From a Database With MongoDB

From a previous post we’ve Setup a MongoDB Cluster, and in this post we will go through the steps of backing up a database and restoring it to another mongodb cluster.

MLab offers a free Shared MongoDB Hosted Service with a limitation of 500MB, which I will be using to restore my data from my own hosted cluster to the free MLab service.

Create the MongoDB Backup

First we will need to create our backup path, and then backup our database, in my case, I am backing up my rocketchat database:

1
2
$ mkdir -p /opt/backups/mongodb
$ mongodump --host mongodb.example.com --port 27017 -u <mongouser> --authenticationDatabase <authdb> --db rocketchat --out /opt/backups/mongodb/

Change into the backup directory:

1
$ cd /opt/backups/mongodb/rocketchat/

You will find the bson and json metadata files for each collection:

1
2
3
4
5
6
7
8
9
10
$ ls -l | awk '{print $9}' | head -9
custom_emoji.chunks.bson
custom_emoji.chunks.metadata.json
custom_emoji.files.bson
custom_emoji.files.metadata.json
instances.bson
instances.metadata.json
meteor_accounts_loginServiceConfiguration.bson
meteor_accounts_loginServiceConfiguration.metadata.json
...

Restore MongoDB Database

We will need to restore all the collections to our new mongodb service, I have created a bash script (restore-mongodb.sh) that will restore each collection to our rocketchat database:

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/usr/bin/env bash

mongo_user=<mongouser>
mongo_pass=<mongopass>

for file in `ls | grep bson`
  do
    for collection in `echo $file | sed 's/.bson//g'`
  do
    mongorestore --host mymongoid.mlab.com --port 12345 -u $mongo_user -p $mongo_pass -d rocketchat -c $collection $file
    sleep 2
  done
done

Change the permissions of your script to make it executable and execute the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ chmod +x restore-mongodb.sh
$ ./restore-mongodb.sh

2017-10-03T22:05:39.138+0200    checking for collection data in custom_emoji.chunks.bson
2017-10-03T22:05:39.159+0200    reading metadata for rocketchat.custom_emoji.chunks from custom_emoji.chunks.metadata.json
2017-10-03T22:05:39.211+0200    restoring rocketchat.custom_emoji.chunks from custom_emoji.chunks.bson
2017-10-03T22:05:39.900+0200    restoring indexes for collection rocketchat.custom_emoji.chunks from metadata
2017-10-03T22:05:39.922+0200    finished restoring rocketchat.custom_emoji.chunks (20 documents)
2017-10-03T22:05:39.922+0200    done
2017-10-03T22:05:42.188+0200    checking for collection data in custom_emoji.files.bson
2017-10-03T22:05:42.231+0200    reading metadata for rocketchat.custom_emoji.files from custom_emoji.files.metadata.json
2017-10-03T22:05:42.252+0200    restoring rocketchat.custom_emoji.files from custom_emoji.files.bson
2017-10-03T22:05:42.623+0200    restoring indexes for collection rocketchat.custom_emoji.files from metadata
2017-10-03T22:05:42.645+0200    finished restoring rocketchat.custom_emoji.files (20 documents)
2017-10-03T22:05:42.645+0200    done
...

Checkout the New MongoDB Database:

Once the restore has been done, logon to your new mongodb database and have a look at the collections in the database:

1
2
3
4
5
6
7
8
9
10
11
12
$ mongo mymongoid.mlab.com:12345/rocketchat -u <mongouser> -p
MongoDB shell version v3.4.7
Enter password:
connecting to: mongodb://mymongoid.mlab.com:12345/rocketchat
MongoDB server version: 3.4.9

rs-mymongoid:PRIMARY> show collections
_raix_push_app_tokens
_raix_push_notifications
custom_emoji.chunks
custom_emoji.files
instances

Resources: