Ruan Bekker's Blog

From a Curious mind to Posts on Github

Using Elasticsearch Curator to Reindex Data

Today I was using Elasticsearch Curator to reindex indices that was created on a daily basis, to reindex all the data to one index. I used this route as the old data will not be accessed frequently.

Install Elasticsearch Curator

1
2
$ docker run -it python:2.7-alpine sh
$ pip install elasticsearch-curator

Create Configs:

Create the curator config:

config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - es.endpoint.com
  port: 443
  use_ssl: True
  ssl_no_validate: False
  http_auth: admin:pass
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['urllib3']

Create the action config:

action-reindex.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: "Reindex index-2017.10.{30,31} into new-index-2017.10"
    action: reindex
    options:
      disable_action: False
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: ['index-2017.10.*']
        dest:
          index: new-index-2017.10
    filters:
    - filtertype: none

Create the Elasticsearch Index:

Create the Index where we will reindex the data to:

1
$ curl -XPUT http://es.endpoint.com/new-index-2017.10 -d '{"settings": {"number_of_shards": 5, "number_of_replicas": 1}}'

Run the Curator:

1
2
3
4
5
6
7
$ curator --config curator.yml action-reindex.yml

2017-11-22 14:18:15,138 INFO      Task "reindex from [index-2017.10.*] to [index-2017.10]" with task_id "Za-sn0z3Q9-75xCMRwJ3-A:15782886" has been running for 928.948195354 seconds
2017-11-22 14:18:24,152 INFO      GET https://es.endpoint.com:443/_tasks/Za-sn0z3Q9-75xCMRwJ3-A%3A15782886 [status:200 request:0.005s]
2017-11-22 14:18:24,153 INFO      Task "reindex from [index-2017.10.*] to [new-index-2017.10]" with task_id "Za-sn0z3Q9-75xCMRwJ3-A:15782886" has been running for 937.962740393 seconds
2017-11-22 14:22:23,171 INFO      Action ID: 1, "reindex" completed.
2017-11-22 14:22:23,171 INFO      Job completed.

Resources:

Using Elasticdump to Backup Elasticsearch Indexes to JSON

We will use Elasticdump to dump data from Elasticsearch to json files on disk, then delete the index, then restore data back to elasticsearch

bekker-clothing-developer-tshirts

Install Elasticdump:

1
2
$ docker run -it node:alpine sh
$ npm install elasticdump -g

Create a Index:

1
2
$ curl -XPUT http://10.79.2.193:9200/test-index
{"acknowledged":true}

Ingest Some Data into the Index:

1
2
3
4
5
$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc1 -d '{"name": "ruan", "age": 30}'
{"_index":"test-index","_type":"docs","_id":"doc1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc2 -d '{"name": "stefan", "age": 29}'
{"_index":"test-index","_type":"docs","_id":"doc2","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

Elasticdump to dump the ata

First dump the mappings:

1
2
3
4
5
6
7
$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index_mapping.json --type=mapping
Mon, 26 Jun 2017 14:15:34 GMT | starting dump
Mon, 26 Jun 2017 14:15:34 GMT | got 1 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:34 GMT | sent 1 objects to destination file, wrote 1
Mon, 26 Jun 2017 14:15:34 GMT | got 0 objects from source elasticsearch (offset: 1)
Mon, 26 Jun 2017 14:15:34 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:15:34 GMT | dump complete

Then dump the data:

1
2
3
4
5
6
7
$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index.json --type=data
Mon, 26 Jun 2017 14:15:43 GMT | starting dump
Mon, 26 Jun 2017 14:15:43 GMT | got 2 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:43 GMT | sent 2 objects to destination file, wrote 2
Mon, 26 Jun 2017 14:15:43 GMT | got 0 objects from source elasticsearch (offset: 2)
Mon, 26 Jun 2017 14:15:43 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:15:43 GMT | dump complete

Preview the Metadata

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ cat /opt/backup/elasticsearch/es_test-index_mapping.json | python -m json.tool
{
    "test-index": {
        "mappings": {
            "docs": {
                "properties": {
                    "age": {
                        "type": "long"
                    },
                    "name": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

Preview the Data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cat /opt/backup/elasticsearch/es_test-index.json | jq
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc1",
  "_score": 1,
  "_source": {
    "name": "ruan",
    "age": 30
  }
}
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc2",
  "_score": 1,
  "_source": {
    "name": "stefan",
    "age": 29
  }
}

Restore The Data

Let’s test the restoring part, go ahead and delete The index:

1
2
$ curl -XDELETE http://10.79.2.193:9200/test-index
{"acknowledged":true}

Restore the Index by Importing the Mapping:

1
2
3
4
5
6
7
$ elasticdump --input=/opt/backup/elasticsearch/es_test-index_mapping.json --output=http://10.79.2.193:9200/test-index --type=mapping
Mon, 26 Jun 2017 14:51:48 GMT | starting dump
Mon, 26 Jun 2017 14:51:48 GMT | got 1 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:51:48 GMT | sent 1 objects to destination elasticsearch, wrote 1
Mon, 26 Jun 2017 14:51:48 GMT | got 0 objects from source file (offset: 1)
Mon, 26 Jun 2017 14:51:48 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:51:48 GMT | dump complete

Verify to see if the Index Exist:

1
2
3
$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E '(docs.count|test)'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          0            0       650b           650b

Restore the Data for the Index:

Use elasticdump to restore the data from json to elasticsearch:

1
2
3
4
5
6
7
$ elasticdump --input=/opt/backup/elasticsearch/es_test-index.json --output=http://10.79.2.193:9200/test-index --type=data
Mon, 26 Jun 2017 14:53:56 GMT | starting dump
Mon, 26 Jun 2017 14:53:56 GMT | got 2 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:53:56 GMT | sent 2 objects to destination elasticsearch, wrote 2
Mon, 26 Jun 2017 14:53:56 GMT | got 0 objects from source file (offset: 2)
Mon, 26 Jun 2017 14:53:56 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:53:56 GMT | dump complete

Verify to see if the Documents was Ingested:

1
2
3
$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E '(docs.count|test)'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          2            0       650b           650b

Preview the Data from Elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ curl -s -XGET http://10.79.2.193:9200/test-index/_search?pretty

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc1",
      "_score" : 1.0,
      "_source" : {
        "name" : "ruan",
        "age" : 30
      }
    }, {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc2",
      "_score" : 1.0,
      "_source" : {
        "name" : "stefan",
        "age" : 29
      }
    } ]
  }
}

Resources:

Routing Web Traffic With a SOCKS Tunnel

I wanted to access a Non Standard HTTP Port on one of my RaspberryPi Hosts, which was not directly available to the Internet, so I have chosen to establish a SOCKS Tunnel to achieve that.

Web Application on my LAN

Getting my RaspberryPi’s Private IP Address:

1
2
$ ifconfig eth0 | grep 'inet 192' | awk '{print $2}'
192.168.1.118

For demonstration purposes, I will use Python’s SimpleHTTPServer:

1
2
3
4
5
$ mkdir web
$ cd web
$ echo 'yeehaa' > index.html
$ python -m SimpleHTTPServer 5050
Serving HTTP on 0.0.0.0 port 5050 ...

Establish the SOCKS Tunnel

From my laptop, establishing the SOCKS Tunnel with SSH, you can use -f to fork it in the background:

1
$ ssh -D 8157 -CqN user@home.domain.com

Configure your Browser:

Configure your browser to Proxy via:

  • Host: localhost
  • Port: 8157

Now when you access the destined host’s private ip, you will get a response:

1
2
Browse to http://192.168.1.118:5050/ and in my case my response is:
-> yeehaa

Local Dev Environment With Docker MySQL and Adminer WebUI With Docker Compose

Let’s setup a local development environment with Docker, MySQL and Adminer WebUI using Docker Compose

Docker Compose File:

Let’s look at our docker-compose file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
version: '3.2'

services:
  mysql-client:
    image: alpine:edge
    volumes:
      - type: bind
        source: ./workspace
        target: /root/workspace
    networks:
      - docknet
    command: ping 127.0.0.1

  db:
    image: mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
    networks:
      - docknet
    volumes:
      - type: volume
        source: dbdata
        target: /var/lib/mysql

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080
    networks:
      - docknet

networks:
    docknet:
        external: true

volumes:
  dbdata:
    external: true

Environment Variables for the MySQL Docker image is:

1
2
3
4
5
6
- MYSQL_ROOT_PASSWORD
- MYSQL_DATABASE
- MYSQL_USER, MYSQL_PASSWORD
- MYSQL_ALLOW_EMPTY_PASSWORD
- MYSQL_RANDOM_ROOT_PASSWORD
- MYSQL_ONETIME_PASSWORD

More info can be viewed on this resource: hub.docker.com/_/mysql/

Pre-Requirements:

Let’s create our pre-requirement:

  1. Networks:
1
$ docker network create docknet
  1. Volumes:

Our Volume for MySQL so that we have persistent data:

1
$ docker volume create dbdata

Our workspace directory that will be persistent in our debug-client alpine container:

1
$ mkdir -p workspace/python

Launching our Services:

Let’s launch our services:

1
2
3
4
$ docker-compose -f mysql-compose.yml up -d
Creating mysql_db_1 ...
Creating mysql_adminer_1
Creating mysql_debug-client_1

Listing our Containers:

1
2
3
4
5
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES
e05804ab6d64        alpine:edge         "ping 127.0.0.1"         21 seconds ago      Up 4 seconds                                   mysql_debug-client_1
c052ceeb6d3b        mysql               "docker-entrypoint..."   21 seconds ago      Up 5 seconds        3306/tcp                   mysql_db_1
2b0446daab4c        adminer             "entrypoint.sh doc..."   26 seconds ago      Up 5 seconds        0.0.0.0:8080->8080/tcp     mysql_adminer_1

Using the Debug Container:

I will use the debug container as the client to connect to the internal services, for example, the mysql-client:

1
2
3
4
$ apk update
$ apk add mysql-client
$ mysql -h db -u root -ppassword
MySQL [(none)]>

Also, you will find the persistent data directory for our workspace:

1
2
$ ls /root/workspace/
python

Accessing the MySQL WebUI: Adminer

Access the service via the exposed endpoint:

1
+ http://localhost:8080/

The login view:

Creating the Table:

Deleting the Environment:

The External Resources will not be deleted:

1
2
3
4
5
$ docker-compose -f mysql-compose.yml down
Removing mysql_debug-client_1 ... done
Removing mysql_db_1           ... done
Removing mysql_adminer_1      ... done
Network docknet is external, skipping

Resources:

Setup a Concourse-CI Server on Ubuntu 16

Concourse is a Pipeline Based Continious Integration system written in Go

Resources:

What is Concourse CI:

Concourse CI is a Continious Integration Platform. Concourse enables you to construct pipelines with a yaml configuration that can consist out of 3 core concepts, tasks, resources, and jobs that compose them. For more information about this have a look at their docs

What will we be doing today

We will setup a Concourse Server on Ubuntu 16.04 and run the traditional Hello, World pipeline

Setup the Server:

Concourse needs PostgresSQL 9.3+

1
2
3
$ apt update && apt upgrade -y
$ apt install postgresql postgresql-contrib -y
$ systemctl enable postgresql

Create the Database and User for Concourse on Postgres:

1
2
$ sudo -u postgres createuser concourse
$ sudo -u postgres createdb --owner=concourse atc

Download the Concourse and Fly Cli Binaries:

1
2
3
4
5
$ wget https://github.com/concourse/concourse/releases/download/v4.2.2/concourse_linux_amd64
$ wget https://github.com/concourse/concourse/releases/download/v4.2.2/fly_linux_amd64
$ chmod +x concourse_linux_amd64 fly_linux_amd64
$ mv concourse_linux_amd64 /usr/bin/concourse
$ mv fly_linux_amd64 /usr/bin/fly

Create the Encryption Keys:

1
2
3
4
5
$ mkdir /etc/concourse
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/tsa_host_key
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/worker_key
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/session_signing_key
$ cp /etc/concourse/worker_key.pub /etc/concourse/authorized_worker_keys

Concourse Web Process Configuration:

1
2
3
4
5
6
7
8
9
10
11
12
$ cat /etc/concourse/web_environment

CONCOURSE_ADD_LOCAL_USER=ruan:pass
CONCOURSE_SESSION_SIGNING_KEY=/etc/concourse/session_signing_key
CONCOURSE_TSA_HOST_KEY=/etc/concourse/tsa_host_key
CONCOURSE_TSA_AUTHORIZED_KEYS=/etc/concourse/authorized_worker_keys
CONCOURSE_POSTGRES_HOST=127.0.0.1
CONCOURSE_POSTGRES_USER=concourse
CONCOURSE_POSTGRES_PASSWORD=concourse
CONCOURSE_POSTGRES_DATABASE=atc
CONCOURSE_MAIN_TEAM_LOCAL_USER=ruan
CONCOURSE_EXTERNAL_URL=http://10.20.30.40:8080

Concourse Worker Process Configuration:

1
2
3
4
5
6
$ cat /etc/concourse/worker_environment

CONCOURSE_WORK_DIR=/var/lib/concourse
CONCOURSE_TSA_HOST=127.0.0.1:2222
CONCOURSE_TSA_PUBLIC_KEY=/etc/concourse/tsa_host_key.pub
CONCOURSE_TSA_WORKER_PRIVATE_KEY=/etc/concourse/worker_key

Create a Concourse user:

1
2
3
4
$ mkdir /var/lib/concourse
$ sudo adduser --system --group concourse
$ sudo chown -R concourse:concourse /etc/concourse /var/lib/concourse
$ sudo chmod 600 /etc/concourse/*_environment

Create SystemD Unit Files, first for the Web Service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat /etc/systemd/system/concourse-web.service

[Unit]
Description=Concourse CI web process (ATC and TSA)
After=postgresql.service

[Service]
User=concourse
Restart=on-failure
EnvironmentFile=/etc/concourse/web_environment
ExecStart=/usr/bin/concourse web

[Install]
WantedBy=multi-user.target

Then the SystemD Unit File for the Worker Service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat /etc/systemd/system/concourse-worker.service

[Unit]
Description=Concourse CI worker process
After=concourse-web.service

[Service]
User=root
Restart=on-failure
EnvironmentFile=/etc/concourse/worker_environment
ExecStart=/usr/bin/concourse worker

[Install]
WantedBy=multi-user.target

Create a postgres password for the concourse user:

1
2
3
4
$ cd /home/concourse/
$ sudo -u concourse psql atc
atc=> ALTER USER concourse WITH PASSWORD 'concourse';
atc=> \q

Start and Enable the Services:

1
2
3
4
5
6
7
$ systemctl start concourse-web concourse-worker
$ systemctl enable concourse-web concourse-worker postgresql
$ systemctl status concourse-web concourse-worker

$ systemctl is-active concourse-worker concourse-web
active
active

The listening ports should more or less look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN      4530/concourse
tcp        0      0 127.0.0.1:7788          0.0.0.0:*               LISTEN      4530/concourse
tcp        0      0 127.0.0.1:8079          0.0.0.0:*               LISTEN      4525/concourse
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1283/sshd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      4047/postgres
tcp6       0      0 :::36159                :::*                    LISTEN      4525/concourse
tcp6       0      0 :::46829                :::*                    LISTEN      4525/concourse
tcp6       0      0 :::2222                 :::*                    LISTEN      4525/concourse
tcp6       0      0 :::8080                 :::*                    LISTEN      4525/concourse
tcp6       0      0 :::22                   :::*                    LISTEN      1283/sshd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           918/dhclient
udp        0      0 0.0.0.0:42165           0.0.0.0:*                           4530/concourse

Client Side:

I will be using a the Fly cli from a Mac, so first we need to download the fly-cli for Mac:

1
2
3
$ wget https://github.com/concourse/concourse/releases/download/v4.2.2/fly_darwin_amd64
$ chmod +x fly_darwin_amd64
$ alias fly='./fly_darwin_amd64'

Next, we need to setup our Concourse Target by Authenticating against our Concourse Endpoint, lets setup our target with the name ci:

1
2
3
4
5
6
7
$ fly -t ci login -c http://10.20.30.40:8080
logging in to team 'main'

username: admin
password:

target saved

Lets list our targets:

1
2
3
$ fly targets
name  url                        team  expiry
ci    http://10.20.30.40:8080    main  Wed, 08 Nov 2017 15:32:59 UTC

Listing Registered Workers:

1
2
3
$ fly -t ci workers
name              containers  platform  tags  team  state    version
ip-172-31-12-134  0           linux     none  none  running  1.2

Listing Active Containers:

1
2
$ fly -t ci containers
handle                                worker            pipeline     job            build #  build id  type   name                  attempt

Hello World Pipeline:

Let’s create a basic pipeline, that will print out Hello, World!:

Our hello-world.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
jobs:
- name: my-job
  plan:
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
          tag: edge
      run:
        path: /bin/sh
        args:
        - -c
        - |
          echo "============="
          echo "Hello, World!"
          echo "============="

Applying the configuration to our pipeline:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ fly -t ci set-pipeline -p yeeehaa -c hello-world.yml
jobs:
  job my-job has been added:
    name: my-job
    plan:
    - task: say-hello
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: alpine
            tag: edge
        run:
          path: /bin/sh
          args:
          - -c
          - |
            echo "============="
            echo "Hello, World!"
            echo "============="

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://10.20.30.40:8080/teams/main/pipelines/yeeehaa

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command
  - click play next to the pipeline in the web ui

We can browse to the WebUI to unpause the pipeline, but since I like to do everything on cli as far as possible, I will unpause the pipeline via cli:

1
2
$ fly -t ci unpause-pipeline -p yeeehaa
unpaused 'yeeehaa'

Now our Pipeline is unpaused, but since we did not specify any triggers, we need to manually trigger the pipeline to run, you can either via the WebUI, select your pipeline which in this case will be named yeeehaa and then select the job, which will be my-job then hit the + sign, which will trigger the pipeline.

I will be using the cli:

1
2
$ fly -t ci trigger-job --job yeeehaa/my-job
started yeeehaa/my-job #1

Via the WebUI on http://10.20.30.40:8080/teams/main/pipelines/yeeehaa/jobs/my-job/builds/1 you should see the Hello, World! output, or via the cli, we also have the option to see the output, so let’s trigger it again, but this time passing the --watch flag:

1
2
3
4
5
6
7
8
9
10
11
12
$ fly -t ci trigger-job --job yeeehaa/my-job --watch
started yeeehaa/my-job #2

initializing
running /bin/sh -c echo "============="
echo "Hello, World!"
echo "============="

=============
Hello, World!
=============
succeeded

Listing our Workers and Containers again:

1
2
3
4
5
6
7
$ fly -t ci workers
name              containers  platform  tags  team  state    version
ip-172-31-12-134  2           linux     none  none  running  1.2

$ fly -t ci containers
handle                                worker            pipeline     job         build #  build id  type   name           attempt
36982955-54fd-4c1b-57b8-216486c58db8  ip-172-31-12-134  yeeehaa      my-job      2        729       task   say-hello      n/a

Installing Elastalert for Elasticsearch on Amazon Linux

Elastalert, a service for Alerting with Elasticsearch:

Setting up Elastalert

We will setup Elastalert for Elasticsearch on Amazon Linux which is a RHEL Based Distribution.

Setting up dependencies

1
2
3
4
5
6
7
8
9
10
$ sudo su
# yum update -y
# yum install git python-devel lib-devel libevent-devel bzip2-devel openssl-devel ncurses-devel zlib zlib-devel xz-devel gcc -y
# yum install python-setuptools -y
# easy_install pip
# pip install virtualenv
# virtualenv .venv
# source .venv/bin/activate
# pip install pip --upgrade
# pip install setuptools --upgrade

Clone Elastalert Repository and Install Dependencies:

1
2
3
4
$ cd /opt/
$ git clone https://github.com/Yelp/elastalert
$ cd elastalert/
$ pip install -r requirements.txt

Configs:

1
2
3
$ cp config.yaml.example config.yaml
$ vim config.yaml
$ vim example_rules/example_frequency.yaml

After opening the config, populate the configuration where needed.

Installation of elastalert:

1
2
$ python setup.py install
$ elastalert-create-index

Running elastalert:

1
2
$ python -m elastalert.elastalert --verbose --rule example_frequency.yaml
INFO:elastalert:Starting up

Systemd Unit File:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# /etc/systemd/system/elastalert.service
[Unit]
Description=Elastalert
# executed after this
After=syslog.target
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/elastalert
Environment="SOME_KEY_1=value" "SOME_KEY_2=value2"
# restart on unexpected exits
Restart=always
# first argument must be an absolute path, rest are arguments to it
ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --rule example_frequency.yaml
# startup/shutdown grace period
TimeoutSec=60

[Install]
# executed before this
WantedBy=multi-user.target
# Thanks:
# https://cloudership.com/blog/2016/4/8/init-scripts-for-web-apps-on-linux-and-why-you-should-be-using-them

Reload, enable and start:

1
2
3
$ systemctl daemon-reload
$ systemctl enable elastalert.service
$ systemctl start elastalert.service

Linux Shell Commands With the Python Commands Module

Using Python to Execute Shell Commands in Linux

Status Code and Output:

Getting the Status Code and the Output:

1
2
3
4
5
6
7
8
9
>>> import commands
>>> commands.getstatusoutput('echo foo')
(0, 'foo')

>>> status, output = commands.getstatusoutput('echo foo')
>>> print(status)
0
>>> print(output)
foo

Command Output Only:

Only getting the Shell Output:

1
2
3
>>> import commands
>>> commands.getoutput('echo foo')
'foo'

Basic Script

Test file with a one line of data:

1
2
$ cat file.txt
test-string

Our very basic python script:

1
2
3
4
5
6
7
import commands

status = None
output = None

status, output = commands.getstatusoutput('cat file.txt')
print("Status: {}, Output: {}".format(status, output))

Running the script:

1
2
$ python script.py
Status: 0, Output: test-string

Using Python to Query MySQL Database With MySQLdb Library

a Quick post to demostrate how to use Python to Query data from MySQL. We will use the MySQL Docker Image for the demonstration.

Provision MySQL

We will use the latest mysql image, and use the environment variable to pass the root password, and also expose the mysql port:

1
$ docker run -itd -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql

Populate some data in MySQL

Connect to MySQL:

1
$ mysql -h 127.0.0.1 -u root -ppasword

Create some test data:

1
2
3
4
5
6
mysql> create database foo;
mysql> use foo;
mysql> create table bar (name VARCHAR(20), surname VARCHAR(20));
mysql> insert into bar values('ruan', 'bekker');
mysql> insert into bar values('stefan', 'bester');
mysql> insert into bar values('peter', 'williams');

Python with MySQL: Setup the Environment

We will use virtualenv to create a virtual environment to keep our installation isolated from the rest of our system. Install virtualenv:

1
$ pip install virtualenv

Create a virtual environment and install the required dependency:

1
2
3
$ virtualenv venv-mysql
$ source venv-mysql/bin/activate
(venv-mysql) pip install MySQL-python

Python with MySQL: Develop the Client

1
2
3
4
5
6
7
8
9
10
11
12
13
>>> import MySQLdb
>>> db = MySQLdb.connect('127.0.0.1', 'root', 'password', 'foo')
>>> con = db.cursor()
>>> con.execute("SELECT * from bar")
4L
>>> rows = con.fetchall()
>>> for row in rows:
...     print(row[0], row[1])
...
('ruan', 'bekker')
('stefan', 'bester')
('peter', 'williams')
>>> exit()

Your First Hello World App With Golang

So everyone has been saying how awesome Golang is, and at this moment, I am quite curious to fiddle with it.

Golang Environment: Golang Docker Image

A quick way to get a Golang Environment, will be to use Docker. We will be using the Alpine tag:

1
$ docker run -it golang:alpine sh

Our Basic App

After we are in our container, lets write our first Hello World App:

app.go
1
2
3
4
5
6
7
package main

import "fmt"

func main() {
  fmt.Println("Hello, World!")
}

Running our App:

Using golang to run our app:

1
2
$ go run app.go
Hello, World!

We can also build our app to create a executable binary:

1
$ go build app.go

You will find that there is a executable binary named app placed in the current working directory. Let’s execute it:

1
2
$ ./app
Hello, World!

This was a very basic example, but will add more examples as I am learning the language

Resources:

New Posts on Github Pages With Octopress Not Showing on Your Blog

So today I had the issue where new posts that was generated and pushed to github, not being displayed on my blog.

I was able to see the markdown pages on my github respository, but via the blog itself, getting 404’s.

The Issue:

When I did a rake generate I found the following error:

1
jekyll 2.5.3 | Error: invalid byte sequence in US-ASCII

Resolving the Issue:

After running the following, I was able to get rid of the error, and posts showing again:

1
2
$ export LC_ALL="en_US.UTF-8"
$ export LANG="en_US.UTF-8"