Ruan Bekker's Blog

From a Curious mind to Posts on Github

Setup Matrix and Element Chat Server

In this tutorial we will setup a Matrix and Element Chat Server using Docker on Ubuntu.

What is Matrix?

Matrix is an open standard and communication protocol for secure, decentralised, real-time communication. For more information on Matrix, see their website

Install Docker

I will assume that docker and docker compose is installed, if not, follow this resource to install them: - https://docs.docker.com/get-docker/

Install Matrix Server

Create the directory structure:

1
2
3

$ docker network create --driver=bridge --subnet=10.10.10.0/24 --gateway=10.10.10.1 matrix_net
$ mkdir matrix
$ cd matrix/

The docker-compose.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

version: '3.8'

services:
  element:
    image: vectorim/element-web:latest
    restart: unless-stopped
    volumes:
      - ./element-config.json:/app/config.json
    networks:
      default:
        ipv4_address: 10.10.10.3

  synapse:
    image: matrixdotorg/synapse:latest
    restart: unless-stopped
    networks:
      default:
        ipv4_address: 10.10.10.4
    volumes:
     - ./synapse:/data

  postgres:
    image: postgres:11
    restart: unless-stopped
    networks:
      default:
        ipv4_address: 10.10.10.2
    volumes:
     - ./postgresdata:/var/lib/postgresql/data
    environment:
     - POSTGRES_DB=synapse
     - POSTGRES_USER=synapse
     - POSTGRES_PASSWORD=STRONGPASSWORD
     - POSTGRES_INITDB_ARGS=--lc-collate C --lc-ctype C --encoding UTF8

networks:
  default:
    external:
      name: matrix

Download a sample config:

1
2

$ wget https://develop.element.io/config.json
$ mv config.json element-config.json

And adjust the bits where needed in element-config.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

{
    "default_server_config": {
        "m.homeserver": {
            "base_url": "https://matrix.domain.co.za",
            "server_name": "matrix.domain.co.za"
        },
        "m.identity_server": {
            "base_url": "https://vector.im"
        }
    },
    "brand": "Element",
    "integrations_ui_url": "https://scalar.vector.im/",
    "integrations_rest_url": "https://scalar.vector.im/api",
    "integrations_widgets_urls": [
        "https://scalar.vector.im/_matrix/integrations/v1",
        "https://scalar.vector.im/api",
        "https://scalar-staging.vector.im/_matrix/integrations/v1",
        "https://scalar-staging.vector.im/api",
        "https://scalar-staging.riot.im/scalar/api"
    ],
    "hosting_signup_link": "https://element.io/matrix-services?utm_source=element-web&utm_medium=web",
    "bug_report_endpoint_url": "https://element.io/bugreports/submit",
    "uisi_autorageshake_app": "element-auto-uisi",
    "showLabsSettings": true,
    "piwik": {
        "url": "https://piwik.riot.im/",
        "siteId": 1,
        "policyUrl": "https://element.io/cookie-policy"
    },
    "roomDirectory": {
        "servers": [
            "matrix.org",
            "gitter.im",
            "libera.chat"
        ]
    },
    "enable_presence_by_hs_url": {
        "https://matrix.org": false,
        "https://matrix-client.matrix.org": false
    },
    "terms_and_conditions_links": [
        {
            "url": "https://element.io/privacy",
            "text": "Privacy Policy"
        },
        {
            "url": "https://element.io/cookie-policy",
            "text": "Cookie Policy"
        }
    ],
    "hostSignup": {
      "brand": "Element Home",
      "cookiePolicyUrl": "https://element.io/cookie-policy",
      "domains": [
          "matrix.org"
      ],
      "privacyPolicyUrl": "https://element.io/privacy",
      "termsOfServiceUrl": "https://element.io/terms-of-service",
      "url": "https://ems.element.io/element-home/in-app-loader"
    },
    "sentry": {
        "dsn": "https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc5@sentry.matrix.org/6",
        "environment": "develop"
    },
    "posthog": {
        "projectApiKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "apiHost": "https://posthog.hss.element.io"
    },
    "features": {},
    "map_style_url": "https://api.maptiler.com/maps/streets/style.json?key=xxxxxxxxxxxxx"
}

Generate the homeserver config:

1

$ docker run -it --rm -v "$HOME/matrix/synapse:/data" -e SYNAPSE_SERVER_NAME=matrix.domain.co.za -e SYNAPSE_REPORT_STATS=yes matrixdotorg/synapse:latest generate

Verify the generated config in synapse/homeserver.yaml (I only changed server name and database):

1
2
3
4
5
6
7
8
9
10

server_name: "matrix.domain.co.za"
database:
  name: psycopg2
  args:
    user: synapse
    password: STRONGPASSWORD
    database: synapse
    host: postgres
    cp_min: 5
    cp_max: 10

Boot the stack:

1

$ docker-compose up -d

Caddy Reverse Proxy

Install caddy as a reverse proxy (includes letsencrypt out of the box):

1
2
3
4
5

$ apt install -y debian-keyring debian-archive-keyring apt-transport-https
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo tee /etc/apt/trusted.gpg.d/caddy-stable.asc
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
$ apt update
$ apt install caddy -y

Create the /etc/caddy/Caddyfile with the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

matrix.domain.co.za {
        reverse_proxy /_matrix/* 10.10.10.4:8008
        reverse_proxy /_synapse/client/* 10.10.10.4:8008

        header {
                X-Content-Type-Options nosniff
                Referrer-Policy strict-origin-when-cross-origin
                Strict-Transport-Security "max-age=63072000; includeSubDomains;"
                Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=(), interest-cohort=()"
                X-Frame-Options SAMEORIGIN
                X-XSS-Protection 1
                X-Robots-Tag none
                -server
        }
}

element.domain.co.za {
        encode zstd gzip
        reverse_proxy 10.10.10.3:80

        header {
                X-Content-Type-Options nosniff
                Referrer-Policy strict-origin-when-cross-origin
                Strict-Transport-Security "max-age=63072000; includeSubDomains;"
                Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=(), interest-cohort=()"
                X-Frame-Options SAMEORIGIN
                X-XSS-Protection 1
                X-Robots-Tag none
                -server
        }
}

Change to the /etc/caddy directory then reload:

1
2
3
4

$ pushd /etc/caddy
$ caddy fmt
$ caddy reload
$ popd

Wait a couple of minutes and visit element on https://element.domain.co.za/

Admin Element User

Create your admin user on the docker container:

1
2
3
4
5
6
7
8
9

$ docker exec -it matrix_synapse_1 bash
> register_new_matrix_user -c /data/homeserver.yaml http://localhost:8008

New user localpart [root]: ruan
Password:
Confirm password:
Make admin [no]: yes
Sending registration request...
Success!

Resources

Thanks to cyberhost.uk for credit on helping me with this post.

Thank You

Thanks for reading, if you like my content, check out my website, read my newsletter or follow me at @ruanbekker on Twitter.

Load Environment Variables From File in Python

In this quick tutorial we will demonstrate how to load additional environment variables from file into your python application.

It loads key value pairs from a file and append it to its current runtime environment variables, so your current environment is unaffected.

python-dotenv

We will make use of the package python-dotenv so we will need to install the python package with pip:

1

python3 -m pip install python-dotenv

The env file

I will create the .env in my current working directory with the content:

1
2

APPLICATION_NAME=foo
APPLICATION_OWNER=bar

The application

This is a basic demonstration of a python application which loads the additional environment variables from file, then we will use json.dumps(.., indent=2) so that we can get a pretty print of all our environment variables:

1
2
3
4
5
6
7

import os
import json
from dotenv import load_dotenv

load_dotenv('.env')

print(json.dumps(dict(os.environ), indent=2))

When we run the application the output will look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
  "SHELL": "/bin/bash",
  "PWD": "/home/ubuntu/env-vars",
  "LOGNAME": "ubuntu",
  "HOME": "/home/ubuntu",
  "LANG": "C.UTF-8",
  "TERM": "xterm-256color",
  "USER": "ubuntu",
  "LC_CTYPE": "C.UTF-8",
  "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin",
  "SSH_TTY": "/dev/pts/0",
  "OLDPWD": "/home/ubuntu",
  "APPLICATION_NAME": "foo",
  "APPLICATION_OWNER": "bar"
}

As we can see our two environment variables was added to the environment. If you would like to access your two environment variables, we can do the following:

1
2
3
4
5
6
7
8
9

import os
from dotenv import load_dotenv

load_dotenv('.env')

APPLICATION_NAME = os.getenv('APPLICATION_NAME')
APPLICATION_OWNER = os.getenv('APPLICATION_OWNER')

print('Name: {0}, Owner: {1}'.format(APPLICATION_NAME, APPLICATION_OWNER))

And when we run that, the output should be the following:

1

Name: foo, Owner: bar

Thank You

Thanks for reading, if you like my content, check out my website, read my newsletter or follow me at @ruanbekker on Twitter.

Run a Basic Python Flask Restful API

In this tutorial we will run a basic api using flask-restful, it will only have two routes which will be a get and post method for the purpose of demonstration.

What is Flask Restful

Flask-RESTful is an extension for Flask that adds support for quickly building REST APIs. It is a lightweight abstraction that works with your existing ORM/libraries. Flask-RESTful encourages best practices with minimal setup.

If you want to see a basic Flask API post, you can follow the link below: - https://blog.ruanbekker.com/blog/2018/11/27/python-flask-tutorial-series-create-a-hello-world-app-p1/

Installation

Install Flask and Flask Restful:

1
2

python3 -m pip install flask
python3 -m pip install flask-restful

Code

The basic code that we have, is to have two methods available (get and post):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

import flask
import flask_restful

app = flask.Flask(__name__)
api = flask_restful.Api(app)

class HelloWorld(flask_restful.Resource):
    def get(self):
        return {'hello': 'world'}

    def post(self):
        json_data = request.get_json(force=True)
        firstname = json_data['firstname']
        lastname = json_data['lastname']
        return jsonify(firstname=firstname, lastname=lastname)

api.add_resource(HelloWorld, '/')

if __name__ == "__main__":
    app.run(debug=True)

Run the Server

Run the server:

1

python api.py

Then make a get request:

1

curl http://localhost:5000/

The response should be the following:

1
2
3

{
    "hello": "world"
}

Then make a post request:

1

curl -XPOST http://localhost:5000/ -d '{"firstname": "ruan", "lastname": "bekker"}'

The response should look something like this:

1
2
3
4

{
  "firstname": "ruan",
  "lastname": "bekker"
}

Integration Tests

We can setup integration tests with unittest by creating test_api.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

import unittest
import app as api

class TestFlaskApi(unittest.TestCase):
    def setUp(self):
        self.app = api.app.test_client()

    def test_get_method(self):
        response = self.app.get("/")
        self.assertEqual(
            response.get_json(),
            {"hello": "world"},
        )

    def test_post_method(self):
        # request payload
        payload = json.dumps({
            "firstname": "ruan",
            "lastname": "bekker"
        })

        # make request
        response = self.app.post("/", data=payload, headers={"Content-Type": "application/json"})

        # assert
        self.assertEqual(str, type(response.json['lastname']))
        self.assertEqual(200, response.status_code)

    def tearDown(self):
        # delete if anything was created
        pass

if __name__ == '__main__':
    unittest.main()

Then we can run our test with:

1

python -m unittest discover -p test_app.py -v

Since our first test is expecting {"hello": "world"} our test will pass, and our second test we are validating that our post request returns a 200 response code and that our lastname field is of string type.

The output of our tests will show something like this:

1
2
3
4
5
6
7

test_get_request (test_app.TestFlaskApi) ... ok
test_post_request (test_app.TestFlaskApi) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.009s

OK

More on Flask-Restful

This was a very basic example and their documentation provides a great tutorial on how to extend from this example. This is also a great blogpost on testing rest api’s.

Thank You

Thanks for reading, if you like my content, check out my website, read my newsletter or follow me at @ruanbekker on Twitter.

Run OpenLDAP With a UI on Docker

In this tutorial we will setup two containers, openldap and a openldap ui to manage our users on openldap.

What is OpenLDAP

OpenLDAP is an open source implementation of the Lightweight Directory Access Protocol, which makes it possible for organizations to use centralized authentication and directory access services over a network.

Configuration

This stack will boot a openldap and openldap-ui container container with the following docker-compose.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

version: "3.8"

services:
  openldap:
    image: osixia/openldap:1.5.0
    container_name: openldap
    volumes:
      - ./storage/ldap_db:/var/lib/ldap
      - ./storage/ldap_config:/etc/ldap/slapd.d
    environment:
      - LDAP_ORGANISATION=example-org
      - LDAP_DOMAIN=example.org
      - LDAP_ADMIN_PASSWORD=admin
      - LDAP_CONFIG_PASSWORD=config
      - LDAP_RFC2307BIS_SCHEMA=true
      - LDAP_REMOVE_CONFIG_AFTER_SETUP=true
      - LDAP_TLS_VERIFY_CLIENT=never
    networks:
      - openldap

  openldap-ui:
    image: wheelybird/ldap-user-manager:v1.5
    container_name: openldap-ui
    environment:
      - LDAP_URI=ldap://openldap
      - LDAP_BASE_DN=dc=example,dc=org
      - LDAP_REQUIRE_STARTTLS=FALSE
      - LDAP_ADMINS_GROUP=admins
      - LDAP_ADMIN_BIND_DN=cn=admin,dc=example,dc=org
      - LDAP_ADMIN_BIND_PWD=admin
      - LDAP_IGNORE_CERT_ERRORS=true
      - NO_HTTPS=TRUE
      - PASSWORD_HASH=SSHA
      - SERVER_HOSTNAME=localhost:18080
    depends_on:
      - openldap
    ports:
      - 18080:80
    networks:
      - openldap

networks:
  openldap:
    name: openldap

Boot

Boot the stack with docker-compose:

1

docker-compose up -d

You can access OpenLDAP-UI on port 18080 and the admin password will be admin. You will have admin access to create users.

Verify Users

Access the openldap container:

1

docker-compose exec openldap bash

You can use ldapsearch to verify our user:

1

ldapsearch -x -h openldap -D "uid=ruan,ou=people,dc=example,dc=org" -b "ou=people,dc=example,dc=org" -w "$PASSWORD" -s base 'uid=ruan'

Or we can use ldapwhoami:

1

ldapwhoami -vvv -h ldap://openldap:389 -p 389 -D 'uid=ruan,ou=people,dc=example,dc=org' -x -w "$PASSWORD"

Which will provide a output with something like:

1
2
3

ldap_initialize( <DEFAULT> )
dn:uid=ruan,ou=people,dc=example,dc=org
Result: Success (0)

Thank You

Thanks for reading, if you like my content, check out my website, read my newsletter or follow me at @ruanbekker on Twitter.

Create DNS Records With Terraform on Cloudflare

In this tutorial we will use Terraform to create DNS records on Cloudflare.

Installing Terraform

I will be installing terraform for linux, but you can follow terraform’s documentation if you are using a different operating system: - https://learn.hashicorp.com/tutorials/terraform/install-cli

1
2
3

> curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
> sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
> sudo apt update && sudo apt install terraform -y

Verify that terraform was installed:

1
2
3

> terraform version
Terraform v1.1.6
on linux_amd64

Cloudflare Authentication

We need to create an API Token in order to authenticate terraform to make the required API calls to create the DNS Record.

They have a great post on this, which you can follow below: - https://developers.cloudflare.com/api/tokens/create

You will need access to “Edit DNS Zones” and also include the Domain that you would like to edit.

Ensure that you save the API Token in a safe place.

Terraform Code

First we will create a project directory:

1
2

> mkdir terraform-cloudflare-dns
> cd terraform-cloudflare-dns

First we will create the providers.tf which we define our provider and the required parameters for the provider:

1
2
3
4
5
6
7
8
9
10
11
12
13

terraform {
  required_providers {
    cloudflare = {
      source = "cloudflare/cloudflare"
      version = "~> 3.0"
    }
  }
}

provider "cloudflare" {
  email   = var.cloudflare_email
  api_token = var.cloudflare_api_token
}

As you can see, we are referencing email and api_token as variables, therefore we need to define those variables in variables.tf:

1
2
3
4
5
6
7
8
9

variable "cloudflare_email" {
  type        = string
  description = "clouflare email address"
}

variable "cloudflare_api_token" {
  type        = string
  description = "cloudflare api token"
}

In our main.tf, we are first using a data resource to query cloudflare for our domain rbkr.xyz and then access the attribute id which we will be using in our cloudflare_record resource so that it knows which domain to add the DNS record for.

Then we are going to create the A record foobar and provide the value of 127.0.0.1:

1
2
3
4
5
6
7
8
9
10
11

data "cloudflare_zone" "this" {
  name = "rbkr.xyz"
}

resource "cloudflare_record" "foobar" {
  zone_id = data.cloudflare_zone.this.id
  name    = "foobar"
  value   = "127.0.0.1"
  type    = "A"
  proxied = false
}

Then we are defining our outputs in outputs.tf:

1
2
3
4
5
6
7
8

output "record" {
  value = cloudflare_record.foobar.hostname
}

output "metadata" {
  value       = cloudflare_record.foobar.metadata
  sensitive   = true
}

Creating the Record

Once our configuration code is in place we can run a init which will download the providers:

1

> terraform init

Once that is done, we can run a plan so we can see what will be deployed, but since our variables.tf has no default values, we will either have to define this in terraform.tfvars or use it in-line.

I will be using it in-line for this demonstration:

1

> terraform plan -var "cloudflare_email=$EMAIL" -var "cloudflare_api_token=$API_TOKEN"

Once you are happy, you can run a apply which will deploy the changes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

> terraform apply -var "cloudflare_email=$EMAIL" -var "cloudflare_api_token=$API_TOKEN"

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # cloudflare_record.foobar will be created
  + resource "cloudflare_record" "foobar" {
      + allow_overwrite = false
      + created_on      = (known after apply)
      + hostname        = (known after apply)
      + id              = (known after apply)
      + metadata        = (known after apply)
      + modified_on     = (known after apply)
      + name            = "foobar"
      + proxiable       = (known after apply)
      + proxied         = false
      + ttl             = (known after apply)
      + type            = "A"
      + value           = "127.0.0.1"
      + zone_id         = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + metadata = (sensitive value)
  + record   = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

cloudflare_record.foobar: Creating...
cloudflare_record.foobar: Creation complete after 4s [id=xxxxxxxxxxxxxxxxxxxxx]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

metadata = <sensitive>
record = "foobar.rbkr.xyz"

Test DNS

We can now test if this is working as expected with a dns utility like dig:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

> dig foobar.rbkr.xyz

; <<>> DiG 9.10.6 <<>> foobar.rbkr.xyz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20800
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;foobar.rbkr.xyz.       IN      A

;; ANSWER SECTION:
foobar.rbkr.xyz. 300    IN      A       127.0.0.1

;; Query time: 262 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Wed Feb 02 13:57:59 SAST 2022
;; MSG SIZE  rcvd: 68

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.

Blockchain Basics

In this tutorial, we will cover the basics of blockchain and why you would want to run a full-node such as bitcoin, ethereum, etc.

Blockchain Basics

Before we start setting up our bitcoin full-node, we first need to get through some blockchain basics, if you already aware of it, you can skip the the setup section of this post.

Block

Transaction data is permanently recorded into files called blocks. You can think of it as a transaction ledger. Blocks are organised into a linear sequence over time.

New transactions are constantly being processed by miners into new blocks which are added to the end of the chain. As blocks are buried deeper and deeper into the blockchain they become harder and harder to change or remove, this gives rise of Bitcoin’s Irreversible Transactions.

The first block added to the blockchain is referred to as the genesis block

Blockchain

A blockchain is a transaction database shared by all nodes participating in a system based on the bitcoin protocol. A full copy of a currency’s blockchain contains every transaction ever executed in the currency. With this information, one can find out how much value belonged to each address at any point in history.

Every block contains a hash of the previous block. This has the effect of creating a chain of blocks from the genesis block to the current block. Each block is guaranteed to come after the previous block chronologically because the previous block’s hash would otherwise not be known. Each block is also computationally impractical to modify once it has been in the chain for a while because every block after it would also have to be regenerated. These properties are what make bitcoins transactions irreversible. The blockchain is the main innovation of Bitcoin.

Mining

Mining is the process of adding transaction records to bitcoin’s public ledger of past transactions. The term “mining rig” is referred to where as a single computer system that performs the necessary computations for “mining”.

The blockchain serves to confirm transactions to the rest of the network as having taken place. Bitcoin nodes use the blockchain to distinguish legitimate Bitcoin transactions from attempts to re-spend coins that have already been spent elsewhere.

Node

Any computer that connects to the bitcoin network is called a node. Nodes that fully verify all of the rules of bitcoin are called full nodes. The most popular software implementation of full nodes is called bitcoin-core, its releases can be found on their github page

What is a Full Node

A full node is a node (computer system with bitcoin-core running on it) which downloads every block and transaction and check them against bitcoin’s consensus rules. which fully validates transactions and blocks. Almost all full nodes also help the network by accepting transactions and blocks from other full nodes, validating those transactions and blocks, and then relaying them to further full nodes.

Some examples of consensus rules:

  • Blocks may only create a certain number of bitcoins. (Currently 6.25 BTC per block.)
  • Transactions must have correct signatures for the bitcoins being spent.
  • Transactions/blocks must be in the correct data format.
  • Within a single blockchain, a transaction output cannot be double-spent.

At minimum, a full node must download every transaction that has ever taken place, all new transactions, and all block headers. Additionally, full nodes must store information about every unspent transaction output until it is spent.

By default full nodes are inefficient in that they download each new transaction at least twice, and they store the entire block chain (more than 165 GB as of 20180214) forever, even though only the unspent transaction outputs (<2 GB) are required. Performance can improved by enabling -blocksonly mode and enabling pruning

Archival Nodes

A subset of full nodes also accept incoming connections and upload old blocks to other peers on the network. This happens if the software is run with -listen=1 as is default.

Contrary to some popular misconceptions, being an archival node is not necessary to being a full node. If a user’s bandwidth is constrained then they can use -listen=0, if their disk space is constrained they can use pruning, all the while still being a fully-validating node that enforces bitcoin’s consensus rules and contributing to bitcoin’s overall security.

Most information was referenced from this wiki.

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.

Run a GETH Ethereum Light Node

ethereum

In this tutorial we will install the Geth implementation of Ethereum on Linux and we will be using the light sync mode which will get you up and running in minutes, which only downloads a couple of GBs.

Once we have our node setup we will be using the API and Web3 to interact with our ethereum node.

To understand the basics of blockchain better, you can read my post: - The Basics of Blockchain

Environment Setup

We require go to be installed on our server, so from golang’s releases page get the latest version of Go and extract it:

1
2
3

cd /tmp
wget https://go.dev/dl/go1.17.4.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.17.4.linux-amd64.tar.gz

Setup environment for Go in /etc/profile.d/go.sh:

1
2
3

export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin

While you are in your session, source the file:

1

source /etc/profile.d/go.sh

And verify that Go is installed:

1
2

go version
go version go1.17.4 linux/amd64

Download Geth

From the geth releases page, get the latest version, extract and setup a symlink to the latest version:

1
2
3
4
5
6

cd /tmp
wget https://gethstore.blob.core.windows.net/builds/geth-linux-amd64-1.10.13-7a0c19f8.tar.gz
tar -xvf geth-linux-amd64-1.10.13-7a0c19f8.tar.gz
sudo mkdir -p /usr/local/geth/1.10.13/bin
sudo mv geth-linux-amd64-1.10.13-7a0c19f8/geth /usr/local/geth/1.10.13/bin/geth
sudo ln -s /usr/local/geth/1.10.13 /usr/local/geth/current

Setup the environment for geth on /etc/profile.d/geth.sh:

1

export PATH=$PATH:/usr/local/geth/current/bin

Then source the file while you are still in your session:

1

source /etc/profile.d/geth.sh

You should be able to verify that geth is installed by running:

1
2
3
4
5
6
7
8
9
10

geth version
Geth
Version: 1.10.13-stable
Git Commit: eae3b1946a276ac099e0018fc792d9e8c3bfda6d
Git Commit Date: 20210929
Architecture: amd64
Go Version: go1.17
Operating System: linux
GOPATH=/home/ubuntu/go
GOROOT=/usr/local/go

Setup Geth

Create the data directory for geth and change the ownership of the directory to our user:

1
2

sudo mkdir -p /blockchain/ethereum/data
sudo chown -R ubuntu:ubuntu /blockchain/ethereum

Run geth in the foreground to test:

1
2
3
4
5

geth --ropsten \
  --datadir /blockchain/ethereum/data --datadir.minfreedisk 1024 \
  --cache 128 --syncmode "light" \
  --http --http.addr 0.0.0.0 --http.port 8545 \
  --metrics --metrics.addr 0.0.0.0 --metrics.port 6060

If everything goes okay, you can stop the process, and remove the ropsten testnet blockchain and state databases:

1

geth --ropsten removedb

Create the systemd unit file in /etc/systemd/system/geth.service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

[Unit]
Description=Geth Node
After=network.target auditd.service
Wants=network.target

[Service]
WorkingDirectory=/home/ubuntu
ExecStart=/usr/local/geth/current/bin/geth \
  --ropsten \
  --datadir /blockchain/ethereum/data \
  --datadir.minfreedisk 1024 \
  --cache 128 \
  --syncmode "light" \
  --http --http.addr 0.0.0.0 --http.port 8545 \
  --http.api "admin,db,debug,eth,miner,net,personal,txpool,web3" \
  --http.corsdomain "*" \
  --metrics --metrics.addr 0.0.0.0 --metrics.port 6060 \
  --whitelist 10920274=0xfd652086d220d506ae5b7cb80fde97d2f3f7028d346cc7d9d384a83d3d638532
User=ubuntu
Group=ubuntu
Restart=on-failure
RestartSec=120s
KillMode=process
KillSignal=SIGINT
TimeoutStopSec=600

[Install]
WantedBy=multi-user.target
Alias=geth.service

The values such as --whitelist can be retrieved from this issue or this post and extracted from the post:

“due to the London upgrade you’ll probably end up on the chain that isn’t tracked by Etherscan and Metamask. To ensure you only retrieve blocks from peers on that chain, include the following string in your geth start command”

Since we created a new systemd unit file, reload the systemd daemon:

1

sudo systemctl daemon-reload

Enable and start geth:

1
2

sudo systemctl enable geth
sudo systemctl restart geth

You can tail the logs to ensure everything runs as it should:

1

sudo journalctl -fu geth

API

Following the JSON-RPC documentation, create your account:

1

curl -H "Content-Type: application/json" -XPOST http://localhost:8545/ -d '{"jsonrpc":"2.0","method":"personal_newAccount","params":["password"],"id":1}'

The response should provide your ropsten testnet address:

1

{"jsonrpc":"2.0","id":1,"result":"0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8"}

We can list all our ethereum addresses by calling the eth_accounts method:

1

curl -H "Content-Type: application/json" -XPOST http://localhost:8545/ -d '{"jsonrpc":"2.0","method":"eth_accounts","params":[],"id":1}'

We can then check our balance with eth_getbalance, where we pass the ethereum address which is in hex format, followed by the block number, but we will use “latest”:

1

curl -H "Content-Type: application/json" -XPOST http://localhost:8545/ -d '{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8", "latest"],"id":1}'

You can use the following faucets to send testnet funds to your account: - https://faucet.dimensions.network/ - https://faucet.ropsten.be/

After sending funds to your account, we can check our balance again:

1

curl -H "Content-Type: application/json" -XPOST http://localhost:8545/ -d '{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8", "latest"],"id":1}'

And our balances should reflect in our account:

1

{"jsonrpc":"2.0","id":1,"result":"0x429d069189e0000"}

Hexadecimal and Wei

As you can notice the value of our balance for our ethereum address is in hexadecimal format, we can convert it to decimal format:

1
2

echo $((0x429d069189e0000))
300000000000000000

We can use python to convert to decimal using the int() function, by passing the hexadecimal value and pass its base to convert it into an integer, the base for hexadecimal is 16:

1
2

>>> print(int('0x429d069189e0000', 16))
300000000000000000

The decimal value that was returned is the value in Wei, and the value of 1 ETH equals to 1,000,000,000,000,000,000 Wei.

Using gwei.io the conversions from 1 ETH are:

1
2
3
4
5
6
7

Wei: 1000000000000000000
Kwei: 1000000000000000
Mwei: 1000000000000
Gwei: 1000000000
Twei: 1000000
Pwei: 1000
ETH: 1

So now we can convert our balance from wei to ethereum:

  • your_balance_in_wei / unit_value_of_wei
  • 300000000000000000 / 1000000000000000000
1
2

python3 -c "print(300000000000000000 / 1000000000000000000)"
0.3

We can use this converter to make sure my math is correct

To get the current gas price in wei

1

curl -H "Content-Type: application/json" -XPOST http://localhost:8545/ -d '{"jsonrpc":"2.0","method":"eth_gasPrice","params":[],"id":1}'{"jsonrpc":"2.0","id":1,"result":"0x73a20d04"}

CLI - Accounts

Create a password in /tmp/pass then:

1
2
3
4
5
6
7
8
9
10
11

geth --datadir /blockchain/ethereum/data --keystore /blockchain/ethereum/data/keystore account new --password /tmp/.pass

Your new key was generated

Public address of the key:   0x5814D945EC909eb1307be4F133AaAB3dEF3572f0
Path of the secret key file: /blockchain/ethereum/data/keystore/UTC--2021-10-06T15-43-23.679655564Z--5814d945ec909eb1307be4f133aaab3def3572f0

- You can share your public address with anyone. Others need it to interact with you.
- You must NEVER share the secret key with anyone! The key controls access to your funds!
- You must BACKUP your key file! Without the key, it's impossible to access account funds!
- You must REMEMBER your password! Without the password, it's impossible to decrypt the key!

Then when you attach your console session, you will be able to see the address that we created:

1
2
3

geth attach /blockchain/ethereum/data/geth.ipc
> eth.accounts[0]
"0x5814d945ec909eb1307be4f133aaab3def3572f0"

CLI - Attach

Run the geth console:

1
2
3
4
5
6
7
8
9
10

geth attach /blockchain/ethereum/data/geth.ipc
Welcome to the Geth JavaScript console!

instance: Geth/v1.10.13-stable-eae3b194/linux-amd64/go1.17
at block: 11173667 (Wed Oct 06 2021 08:00:44 GMT+0200 (SAST))
 datadir: /blockchain/ethereum/data
 modules: admin:1.0 debug:1.0 eth:1.0 ethash:1.0 les:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 vflux:1.0 web3:1.0

To exit, press ctrl-d or type exit
>

We can run net to show us the peercounts:

1
2
3
4
5
6
7
8
9

> net
{
  listening: true,
  peerCount: 1,
  version: "3",
  getListening: function(callback),
  getPeerCount: function(callback),
  getVersion: function(callback)
}

Or if we just want to access the peerCount value:

1
2

> net.peerCount
1

To view the peers thats connected:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

> admin.peers
[{
    caps: ["eth/66", "les/2", "les/3", "les/4", "snap/1"],
    enode: "enode://3b76ec5359e59048721de8b6ff97a064ea280233d37433222ce7efcdac700c987326734983c9b65f8f1914c40e1efd6b43999912a3bca208fcbb540a678db110@93.75.22.22:30308",
    enr: "enr:-KO4QD2mp_FKB4ZDpfOAD_ziVnMXo1Mcd-FQl9Abj__EJKr9As6UE0frpdaiOnWjqzuGLGaabaAkG7e2CvfY8LulI9ENg2V0aMfGhHEZtrOAgmlkgnY0gmlwhF1LFhaDbGVzwQGJc2VjcDI1NmsxoQI7duxTWeWQSHId6Lb_l6Bk6igCM9N0MyIs5-_NrHAMmIRzbmFwwIN0Y3CCdmSDdWRwgnZk",
    id: "a95433e1bcbcc64f5d1ad8bd2535557d1f5ed2191a760f704d42a925656bb8de",
    name: "Geth/v1.10.9-stable-eae3b194/linux-amd64/go1.17",
    network: {
      inbound: false,
      localAddress: "192.168.0.120:55166",
      remoteAddress: "93.75.22.22:30308",
      static: false,
      trusted: false
    },
    protocols: {
      les: {
        difficulty: 35015228630523840,
        head: "1aa1db0e6810f504f1542e8c3c49cecf17b0c3246b41f45bede42723d22b7c0c",
        version: 4
      }
    }
}]

Check if the node is syncing:

1
2
3
4
5
6
7
8

> eth.syncing
{
  currentBlock: 11176044,
  highestBlock: 11176158,
  knownStates: 40043719,
  pulledStates: 39904521,
  startingBlock: 0
}

We can view our accounts:

1
2

> eth.accounts
["0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8"]

And get the balance:

1
2

> eth.getBalance(eth.accounts[0])
300000000000000000

CLI - SendTransaction

The account that will be receiving the funds (host-a):

1
2
3

$ geth attach /blockchain/ethereum/data/geth.ipc
> eth.accounts[0]
"0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8"

It’s current balance:

1
2

> eth.getBalance(eth.accounts[0])
20485608293038927543

On the account that we will be sending from (host-b):

1
2
3

$ geth attach /blockchain/ethereum/data/geth.ipc
> eth.accounts[0]
"0xd490fb53c0e7d3c80153112a4bd135e2cf897282"

It’s current balance:

1
2

> eth.getBalance(eth.accounts[0])
2001712477998186788

When we attempt to send 1ETH to the recipient address:

1
2
3
4
5

> eth.sendTransaction({from: "0xd490fb53c0e7d3c80153112a4bd135e2cf897282", to: "0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8", value: "1000000000000000000"})
Error: authentication needed: password or unlock
  at web3.js:6357:37(47)
  at web3.js:5091:62(37)
  at <eval>:1:20(10)

You will notice that we need to unlock our account first:

1
2
3
4

> web3.personal.unlockAccount(web3.personal.listAccounts[0], null, 60)
Unlock account 0xd490fb53c0e7d3c80153112a4bd135e2cf897282
Passphrase:
true

Now we can send the transaction:

1
2

> eth.sendTransaction({from: "0xd490fb53c0e7d3c80153112a4bd135e2cf897282", to: "0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8", value: "1000000000000000000"})
"0x4bffabf28b71e6f83a48f0accb850b232dc3f482e30d942be3a2eb53d639d4bd"

And the transaction id can be looked up on the ropsten blockexplorer: - https://ropsten.etherscan.io/tx/0x4bffabf28b71e6f83a48f0accb850b232dc3f482e30d942be3a2eb53d639d4bd

And after the confirmations has been confirmed, we can see in our receiving account, we received the funds:

1
2

> eth.getBalance(eth.accounts[0])
21485608293038927543

Web3

Run a python environment:

1

docker run -it python:3.8.5-slim bash

Install dependencies:

1
2
3

apt update
apt install python3-dev gcc -y
pip install web3[tester]

The examples I will be following was extracted from the documentation: - https://ethereum.org/ml/developers/tutorials/a-developers-guide-to-ethereum-part-one/

Instantiate a client and connect to your geth node, [this documentation] provides different methods of connecting, but I will be using the HTTPProvider to connect over the network:

1
2
3
4

>>> from web3 import Web3
>>> w3 = Web3(Web3.HTTPProvider('http://192.168.0.120:8545'))
>>> w3.isConnected()
True

List the accounts:

1
2

>>> w3.eth.accounts
['0x2b1718CdF7dBcc381267CCF43D320C6e194D6aa8']

Get the balance in Wei:

1
2
3

>>> account = w3.eth.accounts[0]
>>> w3.eth.get_balance(account)
300000000000000000

Convert from Wei to ETH:

1
2
3

>>> balance_wei = w3.eth.get_balance(account)
>>> w3.fromWei(balance_wei, 'ether')
Decimal('0.3')

Get the information about the latest block:

1
2

>>> w3.eth.get_block('latest')
AttributeDict({'baseFeePerGas': 9, 'difficulty': 2073277081, 'extraData': HexBytes('0x63732e7064782e65647520676574682076312e31302e38'), 'gasLimit': 8000000, 'gasUsed': 3361330, 'hash': HexBytes('0xd06a7a734413bcffa4d56617b7efb9ebd8e684c5fcc7fd4f3ec85b8b809b1a0b'), 'logsBloom': HexBytes('0x00000004080000000001000000000000000000000000000000000000000000860000000000000400000000010000000000000000001000000000000080240000000000000000000000000018000000000000000000040000000000000000000002000000020000000000000000000800000000000000200000000010000000800000000000000002000000020000000040000000000000000001000000000000020800000000000000000000000000000000000000000000000000000000800000004002008000000001000000800000000000000000000000000000000062004010202800004000000000000000000000000040000000000002200000000000'), 'miner': '0xe9e7034AeD5CE7f5b0D281CFE347B8a5c2c53504', 'mixHash': HexBytes('0x42641ef2d13826f9cb070516f81515464af9c5c0a36edaa7c250fec62d18a193'), 'nonce': HexBytes('0x670ce792aed73895'), 'number': 11173874, 'parentHash': HexBytes('0xe26e265b264e5158a46ee213d39150d90b532db061497027f35ad36e98458895'), 'receiptsRoot': HexBytes('0x08c15b7365caa993a3047a3093ae641d5b97c51aff058952ab48a56bdee9240b'), 'sha3Uncles': HexBytes('0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347'), 'size': 16130, 'stateRoot': HexBytes('0x9e2d99e11b4c71b93f14a563a87945c0ed89e577eefc860a7909bd6f3b8e669f'), 'timestamp': 1633502988, 'totalDifficulty': 35015626783987398, 'transactions': [HexBytes('0x9013555031ca4510e619968814b75ff428c595488c46a387a6b57774313f4686'), HexBytes('0x899f3dfa8cc0ce7eac397500a014dd624d50c0024e112fa3989403da5669b838'), HexBytes('0xf0c08c7e6849be5d23e0c603b405012a1baa4252884f8efac3244d3ed77b8622'), HexBytes('0xb6e03e10e0d6ced0a791f3a9474d760d7248301dc489c5b191aa82b1ef23e677'), HexBytes('0xb424a4da501df145346027c9c839ae9bf9a25f3672bf13fe097c39f46eda5028'), HexBytes('0xcb74ac5580485542ca532f5dc46798b84cb26d34ebc127871d6e2ffead6c32c7'), HexBytes('0xb61cf0eb92798885e4a6309d228e8a31e892e4353810593ba14a2737c1fcd53a'), HexBytes('0x20b27640c1b674be98d3051fac5dcf5ae50d5b7e957defc2336f07b99053fb2c'), HexBytes('0x2929a7384e5b47c4e414142623911a2deca95996e761bc10ccedf607342156af'), HexBytes('0x698af438f73bf384b7c35d4448c0e098d7744b4ce58327dc258a3d5706421c7e')], 'transactionsRoot': HexBytes('0x33cca53eabc2aed8cb0c8a5a7b771b9f14fd2e2aa2561195250411f0714ec768'), 'uncles': []})

Mining

Note: Light clients do not support mining

From another node, im running the fast sync mode on ropsten:

1

--syncmode "fast"

1
2

> eth.hashrate
43949

1
2
3
4

> eth.coinbase
"0x2b1718cdf7dbcc381267ccf43d320c6e194d6aa8"
> eth.getBalance(eth.coinbase).toNumber();
145000000000000000000

Delete Data

If we look at a fully synced ropsten “fast” node:

1
2
3
4
5
6
7
8
9
10

du -h /blockchain/ethereum/
4.0K  /blockchain/ethereum/data/keystore
856K  /blockchain/ethereum/data/geth/triecache
58G   /blockchain/ethereum/data/geth/chaindata/ancient
69G   /blockchain/ethereum/data/geth/chaindata
2.2M  /blockchain/ethereum/data/geth/nodes
188M  /blockchain/ethereum/data/geth/ethash
69G   /blockchain/ethereum/data/geth
69G   /blockchain/ethereum/data
69G   /blockchain/ethereum/

Remove the data with removedb:

1
2
3
4
5
6
7
8
9
10
11

geth --datadir /blockchain/ethereum/data removedb
INFO [10-06|20:01:52.061] Maximum peer count                       ETH=50 LES=0 total=50
INFO [10-06|20:01:52.061] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-06|20:01:52.062] Set global gas cap                       cap=50,000,000
Remove full node state database (/blockchain/ethereum/data/geth/chaindata)? [y/n] y
Remove full node state database (/blockchain/ethereum/data/geth/chaindata)? [y/n] y
INFO [10-06|20:01:57.141] Database successfully deleted            path=/blockchain/ethereum/data/geth/chaindata elapsed=2.482s
Remove full node ancient database (/blockchain/ethereum/data/geth/chaindata/ancient)? [y/n] y
Remove full node ancient database (/blockchain/ethereum/data/geth/chaindata/ancient)? [y/n] y
INFO [10-06|20:02:05.645] Database successfully deleted            path=/blockchain/ethereum/data/geth/chaindata/ancient elapsed=589.737ms
INFO [10-06|20:02:05.645] Light node database missing              path=/blockchain/ethereum/data/geth/lightchaindata

Now when we list the data directory, we can see the data was removed:

1
2
3
4
5
6
7
8
9
10

du -h /blockchain/ethereum/
8.0K  /blockchain/ethereum/data/keystore
868K  /blockchain/ethereum/data/geth/triecache
4.0K  /blockchain/ethereum/data/geth/chaindata/ancient
180K  /blockchain/ethereum/data/geth/chaindata
1.4M  /blockchain/ethereum/data/geth/nodes
188M  /blockchain/ethereum/data/geth/ethash
190M  /blockchain/ethereum/data/geth
190M  /blockchain/ethereum/data
190M  /blockchain/ethereum/

Resources

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.

Run Docker Containers With Terraform

In this post I will demonstrate how to use the terraform docker_container resource from the docker provider to run two docker containers, traefik and nginx and use the random provider to generate a random url for us.

Pre-Requisites

You will require terraform and docker to be installed.

Project Structure

The source code for this post is available on my github repository, but the project structure will look like the following:

image

Our providers.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "2.15.0"
    }
    random = {
      version = "~> 3.0"
    }
  }
}

provider "docker" {
  host = "unix:///var/run/docker.sock"
}

provider "random" {}

Our variables.tf:

1
2
3
4

variable "domain" {
  type    = string
  default = "localdns.xyz"
}

Our outputs.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

output "nginx_container_name" {
  value = docker_container.nginx.name
}

output "traefik_container_name" {
  value = docker_container.traefik.name
}

output "traefik_url" {
  value = "http://traefik.${var.domain}/"
}

output "nginx_url" {
  value = "http://www.${random_string.nginx.result}.${var.domain}/"
}

Our main.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134

resource "random_string" "nginx" {
  length  = 8
  upper   = false
  special = false
}

resource "docker_image" "nginx" {
  name = "nginx:stable-alpine"
}

resource "docker_image" "traefik" {
  name = "traefik:1.7.14"
}

resource "docker_network" "nginx" {
  name   = "docknet"
  driver = "bridge"
}

resource "docker_container" "traefik" {
  name  = "traefik"
  image = docker_image.traefik.name

  networks_advanced {
    name    = docker_network.nginx.name
    aliases = ["docknet"]
  }

  restart = "unless-stopped"
  destroy_grace_seconds = 30
  must_run = true
  memory = 256

  volumes {
    host_path      = "/var/run/docker.sock"
    container_path = "/var/run/docker.sock"
  }

  command = [
    "--api",
    "--docker",
    "--docker.watch",
    "--entrypoints=Name:http Address::80",
    "--logLevel=INFO"
  ]

  ports {
    internal = 80
    external = 80
    ip       = "0.0.0.0"
  }

  labels {
    label = "traefik.enable"
    value = true
  }

  labels {
    label = "traefik.docker.network"
    value = "docknet"
  }

  labels {
    label = "traefik.frontend.rule"
    value = "Host:traefik.${var.domain}"
  }

  labels {
    label = "traefik.port"
    value = 8080
  }

}

resource "docker_container" "nginx" {
  name  = "nginx"
  image = docker_image.nginx.name

  networks_advanced {
    name    = docker_network.nginx.name
    aliases = ["docknet"]
  }

  restart = "unless-stopped"
  destroy_grace_seconds = 30
  must_run = true
  memory = 256

  volumes {
    host_path      = "/Users/ruan/personal/terraform-playground/docker-containers/html"
    container_path = "/usr/share/nginx/html"
  }

  volumes {
    host_path      = "/Users/ruan/personal/terraform-playground/docker-containers/configs/nginx.conf"
    container_path = "/etc/nginx/nginx.conf"
  }

  volumes {
    host_path      = "/Users/ruan/personal/terraform-playground/docker-containers/configs/app.conf"
    container_path = "/etc/nginx/conf.d/app.conf"
  }

  env = [
    "PUID=501",
    "PGID=20"
  ]

  labels {
    label = "traefik.enable"
    value = true
  }

  labels {
    label = "traefik.docker.network"
    value = "docknet"
  }

  labels {
    label = "traefik.frontend.rule"
    value = "Host:www.${random_string.nginx.result}.${var.domain}"
  }

  labels {
    label = "traefik.port"
    value = 80
  }

  depends_on = [
    docker_container.traefik,
    random_string.nginx
  ]

}

Our html/index.html:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

<!doctype html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">

        <title>Welcome</title>

        <!-- Fonts -->
        <link href="https://fonts.googleapis.com/css?family=Nunito:200,600" rel="stylesheet">

        <!-- Styles -->
        <style>
            html, body {
                background-color: #fff;
                color: #636b6f;
                font-family: 'Nunito', sans-serif;
                font-weight: 200;
                height: 100vh;
                margin: 0;
            }

            .full-height {
                height: 100vh;
            }

            .flex-center {
                align-items: center;
                display: flex;
                justify-content: center;
            }

            .position-ref {
                position: relative;
            }

            .top-right {
                position: absolute;
                right: 10px;
                top: 18px;
            }

            .content {
                text-align: center;
            }

            .title {
                font-size: 84px;
            }

            .links > a {
                color: #636b6f;
                padding: 0 25px;
                font-size: 13px;
                font-weight: 600;
                letter-spacing: .1rem;
                text-decoration: none;
                text-transform: uppercase;
            }

            .m-b-md {
                margin-bottom: 30px;
            }
        </style>
    </head>
    <body>
        <div class="flex-center position-ref full-height">
            <div class="content">
                <div class="title m-b-md">
                    Welcome
                </div>

                <div class="links">
                    <a href="https://ruan.dev" target="_blank">About Me</a>
                </div>
            </div>
        </div>
    </body>
</html>

Our configs/nginx.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    include /etc/nginx/conf.d/app.conf;
}

And lastly, our configs/app.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13

server {
  listen 80;
  server_name _;

  location / {
    root   /usr/share/nginx/html;
    index  index.html;
  }

  location /healthz {
    return 200 'up';
  }
}

Deployment

Once everything is in place, or if you want to clone my repository, you can do that by:

1
2

git clone https://github.com/ruanbekker/terraform-docker-container-example
cd terraform-docker-container-example

Then we can initialize terraform by fetching the required plugins:

1

terraform init

Once that completes we can run a plan:

1

terraform plan

And that should output something more or less like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
  + create

Terraform will perform the following actions:

  # docker_container.nginx will be created
  + resource "docker_container" "nginx" {
      + attach                = false
      + bridge                = (known after apply)
      + command               = (known after apply)
      + container_logs        = (known after apply)
      + destroy_grace_seconds = 30
      + entrypoint            = (known after apply)
      + env                   = [
          + "PGID=20",
          + "PUID=501",
        ]
      + exit_code             = (known after apply)
      + gateway               = (known after apply)
      + hostname              = (known after apply)
      + id                    = (known after apply)
      + image                 = "nginx:stable-alpine"
      + init                  = (known after apply)
      + ip_address            = (known after apply)
      + ip_prefix_length      = (known after apply)
      + ipc_mode              = (known after apply)
      + log_driver            = "json-file"
      + logs                  = false
      + memory                = 256
      + must_run              = true
      + name                  = "nginx"
      + network_data          = (known after apply)
      + read_only             = false
      + remove_volumes        = true
      + restart               = "unless-stopped"
      + rm                    = false
      + security_opts         = (known after apply)
      + shm_size              = (known after apply)
      + start                 = true
      + stdin_open            = false
      + tty                   = false

      + healthcheck {
          + interval     = (known after apply)
          + retries      = (known after apply)
          + start_period = (known after apply)
          + test         = (known after apply)
          + timeout      = (known after apply)
        }

      + labels {
          + label = "traefik.docker.network"
          + value = "docknet"
        }
      + labels {
          + label = "traefik.enable"
          + value = "true"
        }
      + labels {
          + label = "traefik.frontend.rule"
          + value = (known after apply)
        }
      + labels {
          + label = "traefik.port"
          + value = "80"
        }

      + networks_advanced {
          + aliases = [
              + "docknet",
            ]
          + name    = "docknet"
        }

      + volumes {
          + container_path = "/etc/nginx/conf.d/app.conf"
          + host_path      = "/Users/ruan/personal/terraform-playground/docker-containers/configs/app.conf"
        }
      + volumes {
          + container_path = "/etc/nginx/nginx.conf"
          + host_path      = "/Users/ruan/personal/terraform-playground/docker-containers/configs/nginx.conf"
        }
      + volumes {
          + container_path = "/usr/share/nginx/html"
          + host_path      = "/Users/ruan/personal/terraform-playground/docker-containers/html"
        }
    }

  # docker_container.traefik will be created
  + resource "docker_container" "traefik" {
      + attach                = false
      + bridge                = (known after apply)
      + command               = [
          + "--api",
          + "--docker",
          + "--docker.watch",
          + "--entrypoints=Name:http Address::80",
          + "--logLevel=INFO",
        ]
      + container_logs        = (known after apply)
      + destroy_grace_seconds = 30
      + entrypoint            = (known after apply)
      + env                   = (known after apply)
      + exit_code             = (known after apply)
      + gateway               = (known after apply)
      + hostname              = (known after apply)
      + id                    = (known after apply)
      + image                 = "traefik:1.7.14"
      + init                  = (known after apply)
      + ip_address            = (known after apply)
      + ip_prefix_length      = (known after apply)
      + ipc_mode              = (known after apply)
      + log_driver            = "json-file"
      + logs                  = false
      + memory                = 256
      + must_run              = true
      + name                  = "traefik"
      + network_data          = (known after apply)
      + read_only             = false
      + remove_volumes        = true
      + restart               = "unless-stopped"
      + rm                    = false
      + security_opts         = (known after apply)
      + shm_size              = (known after apply)
      + start                 = true
      + stdin_open            = false
      + tty                   = false

      + healthcheck {
          + interval     = (known after apply)
          + retries      = (known after apply)
          + start_period = (known after apply)
          + test         = (known after apply)
          + timeout      = (known after apply)
        }

      + labels {
          + label = "traefik.docker.network"
          + value = "docknet"
        }
      + labels {
          + label = "traefik.enable"
          + value = "true"
        }
      + labels {
          + label = "traefik.frontend.rule"
          + value = "Host:traefik.localdns.xyz"
        }
      + labels {
          + label = "traefik.port"
          + value = "8080"
        }

      + networks_advanced {
          + aliases = [
              + "docknet",
            ]
          + name    = "docknet"
        }

      + ports {
          + external = 80
          + internal = 80
          + ip       = "0.0.0.0"
          + protocol = "tcp"
        }

      + volumes {
          + container_path = "/var/run/docker.sock"
          + host_path      = "/var/run/docker.sock"
        }
    }

  # docker_image.nginx will be created
  + resource "docker_image" "nginx" {
      + id          = (known after apply)
      + latest      = (known after apply)
      + name        = "nginx:stable-alpine"
      + output      = (known after apply)
      + repo_digest = (known after apply)
    }

  # docker_image.traefik will be created
  + resource "docker_image" "traefik" {
      + id          = (known after apply)
      + latest      = (known after apply)
      + name        = "traefik:1.7.14"
      + output      = (known after apply)
      + repo_digest = (known after apply)
    }

  # docker_network.nginx will be created
  + resource "docker_network" "nginx" {
      + driver      = "bridge"
      + id          = (known after apply)
      + internal    = (known after apply)
      + ipam_driver = "default"
      + name        = "docknet"
      + options     = (known after apply)
      + scope       = (known after apply)

      + ipam_config {
          + aux_address = (known after apply)
          + gateway     = (known after apply)
          + ip_range    = (known after apply)
          + subnet      = (known after apply)
        }
    }

  # random_string.nginx will be created
  + resource "random_string" "nginx" {
      + id          = (known after apply)
      + length      = 8
      + lower       = true
      + min_lower   = 0
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + result      = (known after apply)
      + special     = false
      + upper       = false
    }

Plan: 6 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + nginx_container_name   = "nginx"
  + nginx_url              = (known after apply)
  + traefik_container_name = "traefik"
  + traefik_url            = "http://traefik.localdns.xyz/"

Which we can see will create 2 containers, traefik and then nginx, map the configs and html in place and also sets the traefik hostname in the labels for our respective containers so that we can reach them via the specific host headers.

The we can deploy our containers:

1

terraform apply -auto-approve

Which will provide us the output detail defined from our outputs.tf:

1
2
3
4
5
6
7
8

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:

nginx_container_name = "nginx"
nginx_url = "http://www.5igjdfq9.localdns.xyz/"
traefik_container_name = "traefik"
traefik_url = "http://traefik.localdns.xyz/"

Access our Containers

We can access our Traefik Dashboard on http://traefik.localdns.xyz and should look something like this:

image

And when we access our Nginx container on http://www.5igjdfq9.localdns.xyz it should look more or less like this:

image

Running a docker ps will show our running containers:

1
2
3
4

docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS                PORTS                NAMES
e45158ae8cba   nginx:stable-alpine    "/docker-entrypoint   3 minutes ago   Up 3 minutes          80/tcp               nginx
ebdbe42a0fcb   traefik:1.7.14         "/traefik --api       3 minutes ago   Up 3 minutes          0.0.0.0:80->80/tcp   traefik

Cleanup

We can delete our containers by running:

1

terraform destroy -auto-approve

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.

Setup Traefik Version 2 on Docker

In this tutorial we will be setting up Traefik v2 as our reverse proxy with port 80 and 443 enabled, and then hook up a example application behind the application load balancer, and route incoiming requests via host headers.

What is Traefik

Traefik is a modern HTTP reverse proxy and load balancer that makes deploying microservices super easy by making use of docker labels to route your traffic based on host headers, path prefixes etc. Please check out their website to find out more about them.

Use Case

In our example we want to route traefik from http://app.selfhosted.co.za to hit our proxy on port 80, then we want traefik to redirect port 80 to the 443 port configured on the proxy which is configured with letsencrypt and reverse proxy the connection to our application.

The application is being configured via docker labels, which we will get into later.

Our Environment

I will be using the domain selfhosted.co.za, so if you are following along, you can just replace this domain with yours.

For this demonstration I have spun up a VM at Civo as you can see below:

image

From the provided public IP address, we will be creating a DNS A record for our domain, and then create a wildcard entry to CNAME to our initial dns name:

image

You might not want to point all the subdomains to that entry, but to simplify things, every application that needs to be routed via traefik, I can manage from a traefik config level, since my dns is already pointing to the public ip where traefik is running on.

So if I spin up a new container, lets say bitwarden, I can just set bitwarden.selfhosted.co.za in the labels of that container and due to the dns already pointing to traefik, traefik will route the connection to the correct container.

Pre-Requisites

In order to follow along you will need docker and docker-compose to be installed, and can be validated using:

1
2
3
4
5

docker -v
Docker version 20.10.7, build f0df350

docker-compose -v
docker-compose version 1.28.6, build 5db8d86f

Traefik on Docker

We will have one docker-compose.yml file which has the proxy and the example application. Be sure to change the following to suite your environment: - traefik.http.routers.api.rule=Host()' - --certificatesResolvers.letsencrypt.acme.email=youremail@yourdomain.net

The compose:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

---
version: '3.8'

services:
  traefik:
    image: traefik:2.4
    container_name: traefik
    restart: unless-stopped
    volumes:
      - ./traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - docknet
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.api.rule=Host(`traefik.selfhosted.co.za`)'
      - 'traefik.http.routers.api.entrypoints=https'
      - 'traefik.http.routers.api.service=api@internal'
      - 'traefik.http.routers.api.tls=true'
      - 'traefik.http.routers.api.tls.certresolver=letsencrypt'
    ports:
      - 80:80
      - 443:443
    command:
      - '--api'
      - '--providers.docker=true'
      - '--providers.docker.exposedByDefault=false'
      - '--entrypoints.http=true'
      - '--entrypoints.http.address=:80'
      - '--entrypoints.http.http.redirections.entrypoint.to=https'
      - '--entrypoints.http.http.redirections.entrypoint.scheme=https'
      - '--entrypoints.https=true'
      - '--entrypoints.https.address=:443'
      - '--certificatesResolvers.letsencrypt.acme.email=youremail@yourdomain.net'
      - '--certificatesResolvers.letsencrypt.acme.storage=acme.json'
      - '--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=http'
      - '--log=true'
      - '--log.level=INFO'
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

  webapp:
    image: traefik/whoami
    container_name: webapp
    restart: unless-stopped
    networks:
      - docknet
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.webapp.rule=Host(`app.selfhosted.co.za`)'
      - 'traefik.http.routers.webapp.entrypoints=https'
      - 'traefik.http.routers.webapp.tls=true'
      - 'traefik.http.routers.webapp.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.webapp.service=webappservice'
      - 'traefik.http.services.webappservice.loadbalancer.server.port=80'
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

networks:
  docknet:
    name: docknet

Prepare the ./traefik/acme.json file:

1
2
3

mkdir traefik
touch traefik/acme.json
chmod 600 traefik/acme.json

As you can see in order to wire a application onto the proxy we need the following labels:

1
2
3
4
5
6
7

  - 'traefik.enable=true'
  - 'traefik.http.routers.webapp.rule=Host(`app.selfhosted.co.za`)'
  - 'traefik.http.routers.webapp.entrypoints=https'
  - 'traefik.http.routers.webapp.tls=true'
  - 'traefik.http.routers.webapp.tls.certresolver=letsencrypt'
  - 'traefik.http.routers.webapp.service=webappservice'
  - 'traefik.http.services.webappservice.loadbalancer.server.port=80'

Now boot our stack using docker-compose:

1

docker-compose up -d

You can follow the logs to ensure everything works as expected:

1
2
3
4
5
6
7
8
9
10
11

docker-compose logs -f
Attaching to webapp, traefik
traefik    | time="2021-07-11T11:02:22Z" level=info msg="Configuration loaded from flags."
traefik    | time="2021-07-11T11:02:22Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
traefik    | time="2021-07-11T11:02:22Z" level=info msg="Starting provider *traefik.Provider {}"
traefik    | time="2021-07-11T11:02:22Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.so                                              ck\",\"defaultRule\":\"Host(``)\",\"swarmModeRefreshSeconds\":\"15s\"}"
traefik    | time="2021-07-11T11:02:22Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}"
traefik    | time="2021-07-11T11:02:22Z" level=info msg="Starting provider *acme.Provider {\"email\":\"youremail@domain.com\",\"caServer\":\"https://                                              acme-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"http\"},\"ResolverNam                                              e\":\"letsencrypt\",\"store\":{},\"TLSChallengeProvider\":{\"Timeout\":4000000000},\"HTTPChallengeProvider\":{}}"
traefik    | time="2021-07-11T11:02:22Z" level=info msg="Testing certificate renew..." providerName=letsencrypt.acme
traefik    | time="2021-07-11T11:02:24Z" level=info msg=Register... providerName=letsencrypt.acme
webapp     | Starting up on port 80

The certificate process might take anything from 5-30s in my experience.

Test the Application

Now that our webapp container is running, make a http request using curl against the configured host rule, which is app.selfhosted.co.za on http so that we can validate if traefik is doing a redirect to https:

1
2
3
4
5
6
7
8
9
10
11
12

curl -IL http://app.selfhosted.co.za:80

HTTP/1.1 308 Permanent Redirect
Location: https://app.selfhosted.co.za/
Date: Sun, 11 Jul 2021 11:05:47 GMT
Content-Length: 18
Content-Type: text/plain; charset=utf-8

HTTP/2 200
content-type: text/plain; charset=utf-8
date: Sun, 11 Jul 2021 11:05:47 GMT
content-length: 343

If we access our webapp service in our web browser, we will see the following:

image

We can also validate that the certificate is valid:

image

We can also access the traefik dashboard using the configured domain, in this case traefik.selfhosted.co.za, and you should see the pretty traefik dashboard:

image

Future Posts

In future posts I will be using this post as the base setup on getting traefik up and running, and future posts that uses traefik will be tagged under #traefik.

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.

Install Nodejs on Linux Using NVM

In this post we will install Nodejs using Node Version Manager (nvm), which allows you to install and use different versions of node via the command line.

For more information on NVM, checkout their github repository

Install

I will be using a debian based linux distribution, so I first will be updating my package manager’s indexes:

1

$ apt update

Then I will install NVM using the instructions from their repository (always ensure that you are aware what you are installing when you curl, pipe, bash):

1

$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash

Verify

You can now log out and log back in for your path to be updated, or you can follow the instructions on your terminal to source your session so that your path to nvm is updated:

1
2
3
4

$ export NVM_DIR="$HOME/.nvm"
$ [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"
$ [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
$ [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

Then you can verify if nvm is in your path:

1
2

$ command -v nvm
nvm

Installing a Node Version

Before we install a specific version of nodejs, let’s first look at the LTS versions from the Fermium release:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$ nvm ls-remote --lts=fermium
       v14.15.0   (LTS: Fermium)
       v14.15.1   (LTS: Fermium)
       v14.15.2   (LTS: Fermium)
       v14.15.3   (LTS: Fermium)
       v14.15.4   (LTS: Fermium)
       v14.15.5   (LTS: Fermium)
       v14.16.0   (LTS: Fermium)
       v14.16.1   (LTS: Fermium)
       v14.17.0   (LTS: Fermium)
       v14.17.1   (LTS: Fermium)
       v14.17.2   (LTS: Fermium)
       v14.17.3   (LTS: Fermium)
       v14.17.4   (LTS: Fermium)
       v14.17.5   (LTS: Fermium)
       v14.17.6   (LTS: Fermium)
       v14.18.0   (Latest LTS: Fermium)

So I want to install v14.8.0:

1

$ nvm install 14.8.0

I also would like to make it my default version of node:

1
2

$ nvm alias default node
default -> node (-> v14.8.0)

Verify Installation

Now we can verify if npm is installed:

1
2

$ npm -v
6.14.7

as well as node:

1
2

$ node -v
v14.8.0

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.