Ruan Bekker's Blog

From a Curious mind to Posts on Github

Delete Old Items With Amazons DynamoDB TTL Feature

As you may know a DynamoDB Table’s Partition Splits on 2 factors, Read/Write Capacity Units and when Storage goes over 10GB.

Automatically Deleting Old Data in DynamoDB:

With the TTL Feature in DynamoDB, we can enable TTL on a Attribute on our Table, the attributes value needs to have an epoc time value, more specifically, when the current time is the same as the value of on of the items attribute value, that item will be expired, which will be deleted.

What we will be doing:

  • Use Boto3 in Python
  • Create DynamoDB Table: ‘session-table’
  • Set TTL Attribute on ‘ExpirationTime’, so whenever the epoch time is equals to the AttributeValue it will delete the item
  • Do one PUT Item with 48 Hours expiry Date from the Write
  • Do 240 PUT Items with 24 Hours expiry Date from the Write
  • Verify after 24 hours if only one item is in our table.

Pre-Requisites:

Install the AWS CLI, Boto3 and configure your credentials, so that boto3 can read from your credential provider:

1
2
3
4
5
6
7
$ pip install awscli
$ pip install boto3
$ aws configure
AWS Access Key ID [****************XYZ]:
AWS Secret Access Key [****************xyz]:
Default region name [eu-west-1]:
Default output format [json]:

Create the Table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ python

import boto3
session = boto3.Session(region_name='eu-west-1', profile_name='default')
dynamodb = session.resource('dynamodb')
table = dynamodb.create_table(
    TableName='session-table',
    KeySchema=[
        {
            'AttributeName': 'sessionid',
            'KeyType': 'HASH'
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'sessionid',
            'AttributeType': 'S'
        }
    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 2,
        'WriteCapacityUnits': 2
    }
)

From the Console, enable TTL and set the TTL Attribute on ExpirationTime

Write Data to DynamoDB

We have 2 functions that will write the current epoch time to the CreationTime attribute and ExpirationTime will have the current time plus the 24 hours in seconds, which will be used for the 240 items that will be written using the for loop and the other function with the 48 hours of seconds, which will be a single write item.

Then we will just write random data to the session data attribute:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
import boto3
import time
import random
from uuid import uuid4

names = ['james', 'john', 'steve', 'peter', 'frank', 'steven', 'jonathan', 'stephen', 'will', 'adam', 'william']
retailer = ['shoprite', 'edgars', 'pnp', 'bestbuy', 'ok', 'grocer-a', 'amazon', 'seveneleven', 'shop-a']

session = boto3.Session(region_name='eu-west-1', profile_name='dev')
ddb = session.resource('dynamodb')
client = ddb.Table('session-table')

def current_time():
    int(time.time())

def current_time():
    return int(time.time())

def expiration_time():
    return int(time.time()) + 86400

def 48h_expiration_time():
    return int(time.time()) + 172800

# expiry on 48 hours
client.put_item(
    Item={
        'sessionid': str(uuid4()),
        'CreationTime': current_time(),
        'ExpirationTime': 48h_expiration_time(),
        'SessionData': {
            'Name': random.choice(names),
            'Retailer': random.choice(retailer),
            'TimeOfTransaction': current_time(),
            'Amount': random.randint(100,9000)
        }
    }
)

# expiry on 24 hours
for x in xrange(240):
    time.sleep(1)
    client.put_item(
        Item={
            'sessionid': str(uuid4()),
            'CreationTime': current_time(),
            'ExpirationTime': expiration_time(),
            'SessionData': {
                'Name': random.choice(names),
                'Retailer': random.choice(retailer),
                'TimeOfTransaction': current_time(),
                'Amount': random.randint(100,9000)
            }
        }
    )

Verify:

Verify after 24 hours if the item with the 48 hour expiration time is still in our table:

1
2
3
4
5
6
7
8
9
10
11
12
13
client.get_item( Key={'sessionid': '69c2a472-f70e-4d72-b25f-e27573696b0c'} )['Item']

{
    u'ExpirationTime': Decimal('1510672221'),
    u'CreationTime': Decimal('1510585821'),
    u'sessionid': u'69c2a472-f70e-4d72-b25f-e27573696b0c',
    u'SessionData': {
        u'Amount': Decimal('3553'),
        u'Retailer': u'amazon',
        u'TimeOfTransaction': Decimal('1510585821'),
        u'Name': u'steve'
    }
}

Which we can see is still there, when doing a GET item on one of our 24 hour expired items, we can see that its no longer there:

1
2
3
4
5
client.get_item( Key={'sessionid': '70b9fc8c-19c4-49d3-bf63-046e992335af'} )['Item']

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'Item'

Doing a SCAN operation, we should see one item:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import json
r = client.scan(TableName='session-table', Limit=10, Select='COUNT', ReturnConsumedCapacity='TOTAL')

print(json.dumps(r, indent=4))
{
    "Count": 1,
    "ScannedCount": 1,
    "ConsumedCapacity": {
        "CapacityUnits": 0.5,
        "TableName": "session-table"
    },
    "ResponseMetadata": {
        "RetryAttempts": 0,
        "HTTPStatusCode": 200,
        "RequestId": "",
        "HTTPHeaders": {
            "x-amzn-requestid": "",
            "content-length": "107",
            "server": "Server",
            "connection": "keep-alive",
            "x-amz-crc32": "2228370918",
            "date": "Tue, 14 Nov 2017 12:02:31 GMT",
            "content-type": "application/x-amz-json-1.0"
        }
    }
}

So we can confirm that the TTL feature expires the data based on the epoch value we provide our item.

Delete the Table:

1
client.delete(TableName='session-table')

Resources:

Update Your Ghost Blog With the Ghost-CLI

If you installed your Ghost Blog with the Ghost-CLI, you can easily upgrade your Ghost version using the CLI.

Backups

Backup your blog by exporting the json via the Ghost Admin Interface, and also update your content directory:

1
2
3
$ sudo su - ghost
$ cd /var/www/ghost
$ tar -zcf /home/ghost/backups/ghost-content-$(date +%F).tar.gz content

Check the Current Version:

1
2
3
4
$ ghost status

Version:
1.17.0

Update Ghost:

1
2
$ npm i -g ghost-cli
$ ghost update

Verify Version:

1
2
3
4
$ ghost status

Version:
1.17.0

No need to restart Ghost as the update function restarted the process already.

Resources:

Use the Reindex API on Elasticsearch to Reindex Your Data

A Basic Example of Reindexing Data with the /_reindex API on Elasticsearch:

Provision Elasticsearch with Docker:

I will be using Elasticsearch on Docker for this Example:

1
$ docker run -itd --name elasticsearch --publish 9200:9200 elasticsearch:alpine

Create Indexes:

Create 3 Indexes and POST 2 Documents to each Index:

1
2
3
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11.20
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11.21
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11.21

Create the Index where we will reindex the data to:

1
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11 -d '{"settings": {"number_of_shards": 5, "number_of_replicas": 0}}'

POST 2 documents to each index:

1
2
3
4
5
6
7
8
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.20/name/ -d '{"name": "max", "type": "labrador"}'
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.20/name/ -d '{"name": "sam", "type": "pooch"}'

$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.21/name/ -d '{"name": "doggie", "type": "bulldog"}'
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.21/name/ -d '{"name": "james", "type": "huskey"}'

$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.22/name/ -d '{"name": "sarah", "type": "poodle"}'
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.22/name/ -d '{"name": "frank", "type": "alsation"}'

View the Indexes:

As you can see we have 2 documents per index, and a empty index for the data that we would like to reindex to:

1
2
3
4
5
6
$ curl -XGET http://127.0.0.1:9200/_cat/indices?v
health status index               uuid                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   animals-2017.11.20  AxRYUfNpQ5ev2mdZf0bYrw   5   1          2            0      8.9kb          8.9kb
green  open   animals-2017.11     1T6TkYWwRuerIZ5_np1B0w   5   0          0            0      1.5kb          1.5kb
yellow open   animals-2017.11.22  fCdaRyBZRiWyQ3tZLhdBrw   5   1          2            0      8.9kb          8.9kb
yellow open   animals-2017.11.21  4Ei9zMDITHy1dI8lIzfjjA   5   1          2            0      8.9kb          8.9kb

Reindex the Data to our Monthly Index:

We will define our query to match all the indexes that has the data and reindex to our new index animals-2017.11:

1
2
$ curl -XPOST http://127.0.0.1:9200/_reindex -d '{"source": {"index": "animals-2017.11.*"}, "dest": {"index": "animals-2017.11"} }'
{"took":219,"timed_out":false,"total":6,"updated":0,"created":6,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}

View the Indexes:

1
2
3
4
5
6
$ curl -XGET http://127.0.0.1:9200/_cat/indices?v
health status index               uuid                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   animals-2017.11.20  AxRYUfNpQ5ev2mdZf0bYrw   5   1          2            0      8.9kb          8.9kb
green  open   animals-2017.11     1T6TkYWwRuerIZ5_np1B0w   5   0          6            0     20.2kb         20.2kb
yellow open   animals-2017.11.22  fCdaRyBZRiWyQ3tZLhdBrw   5   1          2            0      8.9kb          8.9kb
yellow open   animals-2017.11.21  4Ei9zMDITHy1dI8lIzfjjA   5   1          2            0      8.9kb          8.9kb

Delete the Old Indexes:

As your data is now reindexed, we can safely remove our old indexes:

1
$ curl -XDELETE 'http://127.0.0.1:9200/animals-2017.11.*'

To verify:

1
2
3
$ curl -XGET http://127.0.0.1:9200/_cat/indices?v
health status index               uuid                     pri rep docs.count docs.deleted store.size pri.store.size
green  open   animals-2017.11     1T6TkYWwRuerIZ5_np1B0w   5   0          6            0     20.2kb         20.2kb

Resources:

Using Elasticsearch Curator to Reindex Data

Today I was using Elasticsearch Curator to reindex indices that was created on a daily basis, to reindex all the data to one index. I used this route as the old data will not be accessed frequently.

Install Elasticsearch Curator

1
2
$ docker run -it python:2.7-alpine sh
$ pip install elasticsearch-curator

Create Configs:

Create the curator config:

config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - es.endpoint.com
  port: 443
  use_ssl: True
  ssl_no_validate: False
  http_auth: admin:pass
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['urllib3']

Create the action config:

action-reindex.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: "Reindex index-2017.10.{30,31} into new-index-2017.10"
    action: reindex
    options:
      disable_action: False
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: ['index-2017.10.*']
        dest:
          index: new-index-2017.10
    filters:
    - filtertype: none

Create the Elasticsearch Index:

Create the Index where we will reindex the data to:

1
$ curl -XPUT http://es.endpoint.com/new-index-2017.10 -d '{"settings": {"number_of_shards": 5, "number_of_replicas": 1}}'

Run the Curator:

1
2
3
4
5
6
7
$ curator --config curator.yml action-reindex.yml

2017-11-22 14:18:15,138 INFO      Task "reindex from [index-2017.10.*] to [index-2017.10]" with task_id "Za-sn0z3Q9-75xCMRwJ3-A:15782886" has been running for 928.948195354 seconds
2017-11-22 14:18:24,152 INFO      GET https://es.endpoint.com:443/_tasks/Za-sn0z3Q9-75xCMRwJ3-A%3A15782886 [status:200 request:0.005s]
2017-11-22 14:18:24,153 INFO      Task "reindex from [index-2017.10.*] to [new-index-2017.10]" with task_id "Za-sn0z3Q9-75xCMRwJ3-A:15782886" has been running for 937.962740393 seconds
2017-11-22 14:22:23,171 INFO      Action ID: 1, "reindex" completed.
2017-11-22 14:22:23,171 INFO      Job completed.

Resources:

Using Elasticdump to Backup Elasticsearch Indexes to JSON

We will use Elasticdump to dump data from Elasticsearch to json files on disk, then delete the index, then restore data back to elasticsearch

bekker-clothing-developer-tshirts

Install Elasticdump:

1
2
$ docker run -it node:alpine sh
$ npm install elasticdump -g

Create a Index:

1
2
$ curl -XPUT http://10.79.2.193:9200/test-index
{"acknowledged":true}

Ingest Some Data into the Index:

1
2
3
4
5
$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc1 -d '{"name": "ruan", "age": 30}'
{"_index":"test-index","_type":"docs","_id":"doc1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc2 -d '{"name": "stefan", "age": 29}'
{"_index":"test-index","_type":"docs","_id":"doc2","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

Elasticdump to dump the ata

First dump the mappings:

1
2
3
4
5
6
7
$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index_mapping.json --type=mapping
Mon, 26 Jun 2017 14:15:34 GMT | starting dump
Mon, 26 Jun 2017 14:15:34 GMT | got 1 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:34 GMT | sent 1 objects to destination file, wrote 1
Mon, 26 Jun 2017 14:15:34 GMT | got 0 objects from source elasticsearch (offset: 1)
Mon, 26 Jun 2017 14:15:34 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:15:34 GMT | dump complete

Then dump the data:

1
2
3
4
5
6
7
$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index.json --type=data
Mon, 26 Jun 2017 14:15:43 GMT | starting dump
Mon, 26 Jun 2017 14:15:43 GMT | got 2 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:43 GMT | sent 2 objects to destination file, wrote 2
Mon, 26 Jun 2017 14:15:43 GMT | got 0 objects from source elasticsearch (offset: 2)
Mon, 26 Jun 2017 14:15:43 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:15:43 GMT | dump complete

Preview the Metadata

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ cat /opt/backup/elasticsearch/es_test-index_mapping.json | python -m json.tool
{
    "test-index": {
        "mappings": {
            "docs": {
                "properties": {
                    "age": {
                        "type": "long"
                    },
                    "name": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

Preview the Data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cat /opt/backup/elasticsearch/es_test-index.json | jq
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc1",
  "_score": 1,
  "_source": {
    "name": "ruan",
    "age": 30
  }
}
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc2",
  "_score": 1,
  "_source": {
    "name": "stefan",
    "age": 29
  }
}

Restore The Data

Let’s test the restoring part, go ahead and delete The index:

1
2
$ curl -XDELETE http://10.79.2.193:9200/test-index
{"acknowledged":true}

Restore the Index by Importing the Mapping:

1
2
3
4
5
6
7
$ elasticdump --input=/opt/backup/elasticsearch/es_test-index_mapping.json --output=http://10.79.2.193:9200/test-index --type=mapping
Mon, 26 Jun 2017 14:51:48 GMT | starting dump
Mon, 26 Jun 2017 14:51:48 GMT | got 1 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:51:48 GMT | sent 1 objects to destination elasticsearch, wrote 1
Mon, 26 Jun 2017 14:51:48 GMT | got 0 objects from source file (offset: 1)
Mon, 26 Jun 2017 14:51:48 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:51:48 GMT | dump complete

Verify to see if the Index Exist:

1
2
3
$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E '(docs.count|test)'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          0            0       650b           650b

Restore the Data for the Index:

Use elasticdump to restore the data from json to elasticsearch:

1
2
3
4
5
6
7
$ elasticdump --input=/opt/backup/elasticsearch/es_test-index.json --output=http://10.79.2.193:9200/test-index --type=data
Mon, 26 Jun 2017 14:53:56 GMT | starting dump
Mon, 26 Jun 2017 14:53:56 GMT | got 2 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:53:56 GMT | sent 2 objects to destination elasticsearch, wrote 2
Mon, 26 Jun 2017 14:53:56 GMT | got 0 objects from source file (offset: 2)
Mon, 26 Jun 2017 14:53:56 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:53:56 GMT | dump complete

Verify to see if the Documents was Ingested:

1
2
3
$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E '(docs.count|test)'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          2            0       650b           650b

Preview the Data from Elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ curl -s -XGET http://10.79.2.193:9200/test-index/_search?pretty

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc1",
      "_score" : 1.0,
      "_source" : {
        "name" : "ruan",
        "age" : 30
      }
    }, {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc2",
      "_score" : 1.0,
      "_source" : {
        "name" : "stefan",
        "age" : 29
      }
    } ]
  }
}

Resources:

Routing Web Traffic With a SOCKS Tunnel

I wanted to access a Non Standard HTTP Port on one of my RaspberryPi Hosts, which was not directly available to the Internet, so I have chosen to establish a SOCKS Tunnel to achieve that.

Web Application on my LAN

Getting my RaspberryPi’s Private IP Address:

1
2
$ ifconfig eth0 | grep 'inet 192' | awk '{print $2}'
192.168.1.118

For demonstration purposes, I will use Python’s SimpleHTTPServer:

1
2
3
4
5
$ mkdir web
$ cd web
$ echo 'yeehaa' > index.html
$ python -m SimpleHTTPServer 5050
Serving HTTP on 0.0.0.0 port 5050 ...

Establish the SOCKS Tunnel

From my laptop, establishing the SOCKS Tunnel with SSH, you can use -f to fork it in the background:

1
$ ssh -D 8157 -CqN user@home.domain.com

Configure your Browser:

Configure your browser to Proxy via:

  • Host: localhost
  • Port: 8157

Now when you access the destined host’s private ip, you will get a response:

1
2
Browse to http://192.168.1.118:5050/ and in my case my response is:
-> yeehaa

Local Dev Environment With Docker MySQL and Adminer WebUI With Docker Compose

Let’s setup a local development environment with Docker, MySQL and Adminer WebUI using Docker Compose

Docker Compose File:

Let’s look at our docker-compose file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
version: '3.2'

services:
  mysql-client:
    image: alpine:edge
    volumes:
      - type: bind
        source: ./workspace
        target: /root/workspace
    networks:
      - docknet
    command: ping 127.0.0.1

  db:
    image: mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
    networks:
      - docknet
    volumes:
      - type: volume
        source: dbdata
        target: /var/lib/mysql

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080
    networks:
      - docknet

networks:
    docknet:
        external: true

volumes:
  dbdata:
    external: true

Environment Variables for the MySQL Docker image is:

1
2
3
4
5
6
- MYSQL_ROOT_PASSWORD
- MYSQL_DATABASE
- MYSQL_USER, MYSQL_PASSWORD
- MYSQL_ALLOW_EMPTY_PASSWORD
- MYSQL_RANDOM_ROOT_PASSWORD
- MYSQL_ONETIME_PASSWORD

More info can be viewed on this resource: hub.docker.com/_/mysql/

Pre-Requirements:

Let’s create our pre-requirement:

  1. Networks:
1
$ docker network create docknet
  1. Volumes:

Our Volume for MySQL so that we have persistent data:

1
$ docker volume create dbdata

Our workspace directory that will be persistent in our debug-client alpine container:

1
$ mkdir -p workspace/python

Launching our Services:

Let’s launch our services:

1
2
3
4
$ docker-compose -f mysql-compose.yml up -d
Creating mysql_db_1 ...
Creating mysql_adminer_1
Creating mysql_debug-client_1

Listing our Containers:

1
2
3
4
5
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES
e05804ab6d64        alpine:edge         "ping 127.0.0.1"         21 seconds ago      Up 4 seconds                                   mysql_debug-client_1
c052ceeb6d3b        mysql               "docker-entrypoint..."   21 seconds ago      Up 5 seconds        3306/tcp                   mysql_db_1
2b0446daab4c        adminer             "entrypoint.sh doc..."   26 seconds ago      Up 5 seconds        0.0.0.0:8080->8080/tcp     mysql_adminer_1

Using the Debug Container:

I will use the debug container as the client to connect to the internal services, for example, the mysql-client:

1
2
3
4
$ apk update
$ apk add mysql-client
$ mysql -h db -u root -ppassword
MySQL [(none)]>

Also, you will find the persistent data directory for our workspace:

1
2
$ ls /root/workspace/
python

Accessing the MySQL WebUI: Adminer

Access the service via the exposed endpoint:

1
+ http://localhost:8080/

The login view:

Creating the Table:

Deleting the Environment:

The External Resources will not be deleted:

1
2
3
4
5
$ docker-compose -f mysql-compose.yml down
Removing mysql_debug-client_1 ... done
Removing mysql_db_1           ... done
Removing mysql_adminer_1      ... done
Network docknet is external, skipping

Resources:

Setup a Concourse-CI Server on Ubuntu 16

Concourse is a Pipeline Based Continious Integration system written in Go

Resources:

What is Concourse CI:

Concourse CI is a Continious Integration Platform. Concourse enables you to construct pipelines with a yaml configuration that can consist out of 3 core concepts, tasks, resources, and jobs that compose them. For more information about this have a look at their docs

What will we be doing today

We will setup a Concourse Server on Ubuntu 16.04 and run the traditional Hello, World pipeline

Setup the Server:

Concourse needs PostgresSQL 9.3+

1
2
3
$ apt update && apt upgrade -y
$ apt install postgresql postgresql-contrib -y
$ systemctl enable postgresql

Create the Database and User for Concourse on Postgres:

1
2
$ sudo -u postgres createuser concourse
$ sudo -u postgres createdb --owner=concourse atc

Download the Concourse and Fly Cli Binaries:

1
2
3
4
5
$ wget https://github.com/concourse/concourse/releases/download/v4.2.2/concourse_linux_amd64
$ wget https://github.com/concourse/concourse/releases/download/v4.2.2/fly_linux_amd64
$ chmod +x concourse_linux_amd64 fly_linux_amd64
$ mv concourse_linux_amd64 /usr/bin/concourse
$ mv fly_linux_amd64 /usr/bin/fly

Create the Encryption Keys:

1
2
3
4
5
$ mkdir /etc/concourse
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/tsa_host_key
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/worker_key
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/session_signing_key
$ cp /etc/concourse/worker_key.pub /etc/concourse/authorized_worker_keys

Concourse Web Process Configuration:

1
2
3
4
5
6
7
8
9
10
11
12
$ cat /etc/concourse/web_environment

CONCOURSE_ADD_LOCAL_USER=ruan:pass
CONCOURSE_SESSION_SIGNING_KEY=/etc/concourse/session_signing_key
CONCOURSE_TSA_HOST_KEY=/etc/concourse/tsa_host_key
CONCOURSE_TSA_AUTHORIZED_KEYS=/etc/concourse/authorized_worker_keys
CONCOURSE_POSTGRES_HOST=127.0.0.1
CONCOURSE_POSTGRES_USER=concourse
CONCOURSE_POSTGRES_PASSWORD=concourse
CONCOURSE_POSTGRES_DATABASE=atc
CONCOURSE_MAIN_TEAM_LOCAL_USER=ruan
CONCOURSE_EXTERNAL_URL=http://10.20.30.40:8080

Concourse Worker Process Configuration:

1
2
3
4
5
6
$ cat /etc/concourse/worker_environment

CONCOURSE_WORK_DIR=/var/lib/concourse
CONCOURSE_TSA_HOST=127.0.0.1:2222
CONCOURSE_TSA_PUBLIC_KEY=/etc/concourse/tsa_host_key.pub
CONCOURSE_TSA_WORKER_PRIVATE_KEY=/etc/concourse/worker_key

Create a Concourse user:

1
2
3
4
$ mkdir /var/lib/concourse
$ sudo adduser --system --group concourse
$ sudo chown -R concourse:concourse /etc/concourse /var/lib/concourse
$ sudo chmod 600 /etc/concourse/*_environment

Create SystemD Unit Files, first for the Web Service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat /etc/systemd/system/concourse-web.service

[Unit]
Description=Concourse CI web process (ATC and TSA)
After=postgresql.service

[Service]
User=concourse
Restart=on-failure
EnvironmentFile=/etc/concourse/web_environment
ExecStart=/usr/bin/concourse web

[Install]
WantedBy=multi-user.target

Then the SystemD Unit File for the Worker Service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat /etc/systemd/system/concourse-worker.service

[Unit]
Description=Concourse CI worker process
After=concourse-web.service

[Service]
User=root
Restart=on-failure
EnvironmentFile=/etc/concourse/worker_environment
ExecStart=/usr/bin/concourse worker

[Install]
WantedBy=multi-user.target

Create a postgres password for the concourse user:

1
2
3
4
$ cd /home/concourse/
$ sudo -u concourse psql atc
atc=> ALTER USER concourse WITH PASSWORD 'concourse';
atc=> \q

Start and Enable the Services:

1
2
3
4
5
6
7
$ systemctl start concourse-web concourse-worker
$ systemctl enable concourse-web concourse-worker postgresql
$ systemctl status concourse-web concourse-worker

$ systemctl is-active concourse-worker concourse-web
active
active

The listening ports should more or less look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN      4530/concourse
tcp        0      0 127.0.0.1:7788          0.0.0.0:*               LISTEN      4530/concourse
tcp        0      0 127.0.0.1:8079          0.0.0.0:*               LISTEN      4525/concourse
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1283/sshd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      4047/postgres
tcp6       0      0 :::36159                :::*                    LISTEN      4525/concourse
tcp6       0      0 :::46829                :::*                    LISTEN      4525/concourse
tcp6       0      0 :::2222                 :::*                    LISTEN      4525/concourse
tcp6       0      0 :::8080                 :::*                    LISTEN      4525/concourse
tcp6       0      0 :::22                   :::*                    LISTEN      1283/sshd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           918/dhclient
udp        0      0 0.0.0.0:42165           0.0.0.0:*                           4530/concourse

Client Side:

I will be using a the Fly cli from a Mac, so first we need to download the fly-cli for Mac:

1
2
3
$ wget https://github.com/concourse/concourse/releases/download/v4.2.2/fly_darwin_amd64
$ chmod +x fly_darwin_amd64
$ alias fly='./fly_darwin_amd64'

Next, we need to setup our Concourse Target by Authenticating against our Concourse Endpoint, lets setup our target with the name ci:

1
2
3
4
5
6
7
$ fly -t ci login -c http://10.20.30.40:8080
logging in to team 'main'

username: admin
password:

target saved

Lets list our targets:

1
2
3
$ fly targets
name  url                        team  expiry
ci    http://10.20.30.40:8080    main  Wed, 08 Nov 2017 15:32:59 UTC

Listing Registered Workers:

1
2
3
$ fly -t ci workers
name              containers  platform  tags  team  state    version
ip-172-31-12-134  0           linux     none  none  running  1.2

Listing Active Containers:

1
2
$ fly -t ci containers
handle                                worker            pipeline     job            build #  build id  type   name                  attempt

Hello World Pipeline:

Let’s create a basic pipeline, that will print out Hello, World!:

Our hello-world.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
jobs:
- name: my-job
  plan:
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
          tag: edge
      run:
        path: /bin/sh
        args:
        - -c
        - |
          echo "============="
          echo "Hello, World!"
          echo "============="

Applying the configuration to our pipeline:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ fly -t ci set-pipeline -p yeeehaa -c hello-world.yml
jobs:
  job my-job has been added:
    name: my-job
    plan:
    - task: say-hello
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: alpine
            tag: edge
        run:
          path: /bin/sh
          args:
          - -c
          - |
            echo "============="
            echo "Hello, World!"
            echo "============="

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://10.20.30.40:8080/teams/main/pipelines/yeeehaa

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command
  - click play next to the pipeline in the web ui

We can browse to the WebUI to unpause the pipeline, but since I like to do everything on cli as far as possible, I will unpause the pipeline via cli:

1
2
$ fly -t ci unpause-pipeline -p yeeehaa
unpaused 'yeeehaa'

Now our Pipeline is unpaused, but since we did not specify any triggers, we need to manually trigger the pipeline to run, you can either via the WebUI, select your pipeline which in this case will be named yeeehaa and then select the job, which will be my-job then hit the + sign, which will trigger the pipeline.

I will be using the cli:

1
2
$ fly -t ci trigger-job --job yeeehaa/my-job
started yeeehaa/my-job #1

Via the WebUI on http://10.20.30.40:8080/teams/main/pipelines/yeeehaa/jobs/my-job/builds/1 you should see the Hello, World! output, or via the cli, we also have the option to see the output, so let’s trigger it again, but this time passing the --watch flag:

1
2
3
4
5
6
7
8
9
10
11
12
$ fly -t ci trigger-job --job yeeehaa/my-job --watch
started yeeehaa/my-job #2

initializing
running /bin/sh -c echo "============="
echo "Hello, World!"
echo "============="

=============
Hello, World!
=============
succeeded

Listing our Workers and Containers again:

1
2
3
4
5
6
7
$ fly -t ci workers
name              containers  platform  tags  team  state    version
ip-172-31-12-134  2           linux     none  none  running  1.2

$ fly -t ci containers
handle                                worker            pipeline     job         build #  build id  type   name           attempt
36982955-54fd-4c1b-57b8-216486c58db8  ip-172-31-12-134  yeeehaa      my-job      2        729       task   say-hello      n/a

Installing Elastalert for Elasticsearch on Amazon Linux

Elastalert, a service for Alerting with Elasticsearch:

Setting up Elastalert

We will setup Elastalert for Elasticsearch on Amazon Linux which is a RHEL Based Distribution.

Setting up dependencies

1
2
3
4
5
6
7
8
9
10
$ sudo su
# yum update -y
# yum install git python-devel lib-devel libevent-devel bzip2-devel openssl-devel ncurses-devel zlib zlib-devel xz-devel gcc -y
# yum install python-setuptools -y
# easy_install pip
# pip install virtualenv
# virtualenv .venv
# source .venv/bin/activate
# pip install pip --upgrade
# pip install setuptools --upgrade

Clone Elastalert Repository and Install Dependencies:

1
2
3
4
$ cd /opt/
$ git clone https://github.com/Yelp/elastalert
$ cd elastalert/
$ pip install -r requirements.txt

Configs:

1
2
3
$ cp config.yaml.example config.yaml
$ vim config.yaml
$ vim example_rules/example_frequency.yaml

After opening the config, populate the configuration where needed.

Installation of elastalert:

1
2
$ python setup.py install
$ elastalert-create-index

Running elastalert:

1
2
$ python -m elastalert.elastalert --verbose --rule example_frequency.yaml
INFO:elastalert:Starting up

Systemd Unit File:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# /etc/systemd/system/elastalert.service
[Unit]
Description=Elastalert
# executed after this
After=syslog.target
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/elastalert
Environment="SOME_KEY_1=value" "SOME_KEY_2=value2"
# restart on unexpected exits
Restart=always
# first argument must be an absolute path, rest are arguments to it
ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --rule example_frequency.yaml
# startup/shutdown grace period
TimeoutSec=60

[Install]
# executed before this
WantedBy=multi-user.target
# Thanks:
# https://cloudership.com/blog/2016/4/8/init-scripts-for-web-apps-on-linux-and-why-you-should-be-using-them

Reload, enable and start:

1
2
3
$ systemctl daemon-reload
$ systemctl enable elastalert.service
$ systemctl start elastalert.service

Linux Shell Commands With the Python Commands Module

Using Python to Execute Shell Commands in Linux

Status Code and Output:

Getting the Status Code and the Output:

1
2
3
4
5
6
7
8
9
>>> import commands
>>> commands.getstatusoutput('echo foo')
(0, 'foo')

>>> status, output = commands.getstatusoutput('echo foo')
>>> print(status)
0
>>> print(output)
foo

Command Output Only:

Only getting the Shell Output:

1
2
3
>>> import commands
>>> commands.getoutput('echo foo')
'foo'

Basic Script

Test file with a one line of data:

1
2
$ cat file.txt
test-string

Our very basic python script:

1
2
3
4
5
6
7
import commands

status = None
output = None

status, output = commands.getstatusoutput('cat file.txt')
print("Status: {}, Output: {}".format(status, output))

Running the script:

1
2
$ python script.py
Status: 0, Output: test-string