Ruan Bekker's Blog

From a Curious mind to Posts on Github

Elasticsearch Curator to Manage and Curate Your Elasticsearch Indexes

Elasticsearch Curator helps you to manage and curate your Elasticsearch Indices. I will show how to use the Curator in the following ways:

  • Create Indexes
  • Reindex Indexes
  • Set Replica Counts on Indexes
  • Delete Indexes

Install Elasticsearch Curator

Install Elasticsearch Curator as follows:

1
2
3
$ virtualenv .venv
$ source .venv/bin/activate
$ pip install elasticsearch-curator

Populate the configuration whith your Elasticsearch Host details:

config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
---
client:
  hosts:
    - es.domain.com
  port: 443
  use_ssl: True
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['urllib3']

Action: Create Indices

Use Curator to Create Elasticsearch Indexes:

action-create-indices.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
---
actions:
  create_web-app1-metrics:
    action: create_index
    description: >-
      Create Elasticsearch Index based on Todays Date
      Specify Number of Primary and Replica Shards
      web-app1-metrics-2017.12.04
    options:
      name: '<web-app1-metrics-{now/d}>'
      extra_settings:
        settings:
          number_of_shards: 5
          number_of_replicas: 1
        continue_if_exception: True
        disable_action: False

  create_web-app2-metrics:
    action: create_index
    description: "Create Index with the 1st of this Month in Daily Format - web-app2-metrics-2017.12.01"
    options:
      name: '<web-app2-metrics-{now/M}>'
      extra_settings:
        settings:
          number_of_shards: 5
          number_of_replicas: 2
        continue_if_exception: True
        disable_action: False

  create_web-app3-metrics:
    action: create_index
    description: "Create Index with Last Months Date in Month Format - web-app3-metrics-2017.11"
    options:
      name: '<web-app2-metrics-{now/M-1M{YYYY.MM}}>'
      extra_settings:
        settings:
          number_of_shards: 5
          number_of_replicas: 2
        continue_if_exception: True
        disable_action: False

  create_web-app4-metrics:
    action: create_index
    description: "Create Index with Daily Format 12 Hours from Now - web-app4-metrics-2017.12.05"
    options:
      name: '<web-app2-metrics-{now/d{YYYY.MM.dd|+12:00}}>'
      extra_settings:
        settings:
          number_of_shards: 5
          number_of_replicas: 2
        continue_if_exception: True
        disable_action: False

When Running curator, you can append --dry-run to test your config/action without touching your data. To create these indexes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ curator --config config.yml action-create-indices.yml

2017-12-04 14:22:40,252 INFO      Preparing Action ID: create_web-app1-metrics, "create_index"
2017-12-04 14:22:40,303 INFO      GET https://es.domain.com:443/ [status:200 request:0.036s]
2017-12-04 14:22:40,304 INFO      Trying Action ID: create_web-app1-metrics, "create_index": Create Elasticsearch Index based on Todays Date Specify Number of Primary and Replica Shards web-app1-metrics-2017.12.04
2017-12-04 14:22:40,304 INFO      "<web-app1-metrics-{now/d}>" is using Elasticsearch date math.
2017-12-04 14:22:40,304 INFO      Creating index "<web-app1-metrics-{now/d}>" with settings: {'continue_if_exception': True, 'settings': {'number_of_replicas': 1, 'number_of_shards': 5}, 'disable_action': False}
2017-12-04 14:22:41,490 INFO      PUT https://es.domain.com:443/%3Cweb-app1-metrics-%7Bnow%2Fd%7D%3E [status:200 request:1.185s]
2017-12-04 14:22:41,490 INFO      Action ID: create_web-app1-metrics, "create_index" completed.
2017-12-04 14:22:41,490 INFO      Preparing Action ID: create_web-app2-metrics, "create_index"
2017-12-04 14:22:41,533 INFO      GET https://es.domain.com:443/ [status:200 request:0.033s]
2017-12-04 14:22:41,534 INFO      Trying Action ID: create_web-app2-metrics, "create_index": Create Index with the 1st of this Month in Daily Format - web-app2-metrics-2017.12.01
2017-12-04 14:22:41,534 INFO      "<web-app2-metrics-{now/M}>" is using Elasticsearch date math.
2017-12-04 14:22:41,534 INFO      Creating index "<web-app2-metrics-{now/M}>" with settings: {'continue_if_exception': True, 'settings': {'number_of_replicas': 2, 'number_of_shards': 5}, 'disable_action': False}
2017-12-04 14:22:41,634 INFO      PUT https://es.domain.com:443/%3Cweb-app2-metrics-%7Bnow%2FM%7D%3E [status:200 request:0.099s]
2017-12-04 14:22:41,634 INFO      Action ID: create_web-app2-metrics, "create_index" completed.
2017-12-04 14:22:41,634 INFO      Preparing Action ID: create_web-app3-metrics, "create_index"
2017-12-04 14:22:41,673 INFO      GET https://es.domain.com:443/ [status:200 request:0.028s]
2017-12-04 14:22:41,674 INFO      Trying Action ID: create_web-app3-metrics, "create_index": Create Index with Last Months Date in Month Format - web-app3-metrics-2017.11
2017-12-04 14:22:41,674 INFO      "<web-app2-metrics-{now/M-1M{YYYY.MM}}>" is using Elasticsearch date math.
2017-12-04 14:22:41,674 INFO      Creating index "<web-app2-metrics-{now/M-1M{YYYY.MM}}>" with settings: {'continue_if_exception': True, 'settings': {'number_of_replicas': 2, 'number_of_shards': 5}, 'disable_action': False}
2017-12-04 14:22:41,750 INFO      PUT https://es.domain.com:443/%3Cweb-app2-metrics-%7Bnow%2FM-1M%7BYYYY.MM%7D%7D%3E [status:200 request:0.076s]
2017-12-04 14:22:41,751 INFO      Action ID: create_web-app3-metrics, "create_index" completed.
2017-12-04 14:22:41,751 INFO      Preparing Action ID: create_web-app4-metrics, "create_index"
2017-12-04 14:22:41,785 INFO      GET https://es.domain.com:443/ [status:200 request:0.027s]
2017-12-04 14:22:41,786 INFO      Trying Action ID: create_web-app4-metrics, "create_index": Create Index with Daily Format 12 Hours from Now - web-app4-metrics-2017.12.05
2017-12-04 14:22:41,786 INFO      "<web-app2-metrics-{now/d{YYYY.MM.dd|+12:00}}>" is using Elasticsearch date math.
2017-12-04 14:22:41,786 INFO      Creating index "<web-app2-metrics-{now/d{YYYY.MM.dd|+12:00}}>" with settings: {'continue_if_exception': True, 'settings': {'number_of_replicas': 2, 'number_of_shards': 5}, 'disable_action': False}
2017-12-04 14:22:42,182 INFO      PUT https://es.domain.com:443/%3Cweb-app2-metrics-%7Bnow%2Fd%7BYYYY.MM.dd%7C%2B12%3A00%7D%7D%3E [status:200 request:0.396s]
2017-12-04 14:22:42,183 INFO      Action ID: create_web-app4-metrics, "create_index" completed.
2017-12-04 14:22:42,183 INFO      Job completed.

Lets have a look at our indices to confirm that our indices was created:

1
2
3
4
5
6
$ curl -s -XGET "https://es.domain.com/_cat/indices/web-*?v"
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   web-app2-metrics-2017.12.01 qJHVyft1THemh1qGvA8u0w   5   2          0            0       810b           810b
green  open   web-app2-metrics-2017.11    y5R4vNfOSh2tiC-yGtkgLg   5   2          0            0       810b           810b
green  open   web-app2-metrics-2017.12.05 -ohbgD6-TmmCeJtVv84dPw   5   2          0            0       810b           810b
green  open   web-app1-metrics-2017.12.04 WeGkgB9FSq-cuLVR7ccQFQ   5   1          0            0       810b           810b

Action: Reindex Indices based on Timestring

I would like to reindex a months worth of index data to a monthly index:

action-reindex.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
---
actions:
  re-index_web-app1-metrics:
    action: reindex
    description: "reindex web-app1-metrics to monthly index of last months date - archive-web-app1-metrics-2017.11"
    options:
      continue_if_exception: False
      disable_action: False
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: '<web-app1-metrics-{now/d-31d{YYYY.MM.dd}}>'
        dest:
          index: '<archive-web-app1-metrics-{now/M-1M{YYYY.MM}}>'
    filters:
    - filtertype: none

Running the Curator to reindex all last months data: web-app1-metrics-2017.11.{01-31} to the index: web-app1-metrics-2017.11:

1
$ curator --config config action-reindex.yml

Curator to Change Replica Counts on your Indices:

We will change all our indices settings to replica count of 2, that is matched with our prefix pattern. We are using wait_for_completion so the job will only be completed once the replica count number is updated and data has been replicated to the replica shards.

Our action file:

action-replicas.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
---
actions:
  increase_replica_2:
    action: replicas
    description: >-
      Increase the replica count to 2 for logstash- prefixed indices older than
      10 days (based on index creation_date)
    options:
      count: 2
      max_wait: -1
      wait_interval: 10
      wait_for_completion: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: packet-capture-2017.11.

Using Curator to increase our replica count on all the matched indices:

action-replicas.yml
1
2
3
4
$ curator --config config.yml action-replicas.yml
2017-12-04 13:42:41,322 INFO      Health Check for all provided keys passed.
2017-12-04 13:42:41,323 INFO      Action ID: increase_replica_2, "replicas" completed.
2017-12-04 13:42:41,323 INFO      Job completed.

Curator to Delete your Indices:

action-replicas.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
---
# documentation:
# https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html

actions:
  delete-index_web-app1-metrics:
    action: delete_indices
    description: >-
      Delete indices older than 21 days - based on index name, web-app1-metrics-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: web-app1-metrics-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 21
      exclude:

  delete-index_web-app2-metrics:
    action: delete_indices
    description: >-
      Delete indices older than 1 month - based on index name, web-app2-metrics-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: web-app2-metrics-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: months
      unit_count: 1
      exclude:

First we will execute a Dry Run:

action-replicas.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ curator --config /opt/curator/es-dev/config.yml /opt/curator/es-dev/actions/action-delete.yml --dry-run

2017-12-04 14:43:19,789 INFO      Preparing Action ID: delete-index_web-app1-metrics, "delete_indices"
2017-12-04 14:43:19,850 INFO      GET https://es.domain.com:443/ [status:200 request:0.037s]
2017-12-04 14:43:19,851 INFO      Trying Action ID: delete-index_web-app1-metrics, "delete_indices": Delete indices older than 21 days - based on index name, web-app1-metrics- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2017-12-04 14:43:19,859 INFO      GET https://es.domain.com:443/_all/_settings?expand_wildcards=open%2Cclosed [status:200 request:0.008s]
2017-12-04 14:43:19,862 INFO      GET https://es.domain.com:443/ [status:200 request:0.002s]
2017-12-04 14:43:19,957 INFO      DRY-RUN MODE.  No changes will be made.
2017-12-04 14:43:19,957 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2017-12-04 14:43:19,957 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.01 with arguments: {}
2017-12-04 14:43:19,958 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.02 with arguments: {}
2017-12-04 14:43:19,958 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.03 with arguments: {}
2017-12-04 14:43:19,958 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.04 with arguments: {}
2017-12-04 14:43:19,958 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.05 with arguments: {}
2017-12-04 14:43:19,958 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.06 with arguments: {}
2017-12-04 14:43:19,958 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.07 with arguments: {}
2017-12-04 14:43:19,958 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.08 with arguments: {}
2017-12-04 14:43:19,959 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.09 with arguments: {}
2017-12-04 14:43:19,959 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.10 with arguments: {}
2017-12-04 14:43:19,959 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.11 with arguments: {}
2017-12-04 14:43:19,959 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.12 with arguments: {}
2017-12-04 14:43:19,959 INFO      DRY-RUN: delete_indices: web-app1-metrics-2017.11.13 with arguments: {}
2017-12-04 14:43:19,959 INFO      Action ID: delete-index_web-app1-metrics, "delete_indices" completed.
2017-12-04 14:43:19,959 INFO      Preparing Action ID: delete-index_web-app2-metrics, "delete_indices"
2017-12-04 14:43:20,025 INFO      GET https://es.domain.com:443/ [status:200 request:0.050s]
2017-12-04 14:43:20,026 INFO      Trying Action ID: delete-index_web-app2-metrics, "delete_indices": Delete indices older than 1 month - based on index name, web-app2-metrics- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2017-12-04 14:43:20,034 INFO      GET https://es.domain.com:443/_all/_settings?expand_wildcards=open%2Cclosed [status:200 request:0.008s]
2017-12-04 14:43:20,039 INFO      GET https://es.domain.com:443/ [status:200 request:0.003s]
2017-12-04 14:43:20,090 INFO      DRY-RUN MODE.  No changes will be made.
2017-12-04 14:43:20,090 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2017-12-04 14:43:20,090 INFO      DRY-RUN: delete_indices: web-app2-metrics-2017.11.01 with arguments: {}
2017-12-04 14:43:20,090 INFO      DRY-RUN: delete_indices: web-app2-metrics-2017.11.02 with arguments: {}
2017-12-04 14:43:20,090 INFO      DRY-RUN: delete_indices: web-app2-metrics-2017.11.03 with arguments: {}
2017-12-04 14:43:20,090 INFO      DRY-RUN: delete_indices: web-app2-metrics-2017.11.04 with arguments: {}
2017-12-04 14:43:20,090 INFO      Action ID: delete-index_web-app2-metrics, "delete_indices" completed.
2017-12-04 14:43:20,090 INFO      Job completed.

Everything seems to be as expected, lets run the Curator without the Dry-Run mode:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ curator --config config.yml action-delete.yml

2017-12-04 14:43:40,042 INFO      Deleting selected indices: [u'web-app1-metrics-2017.11.06', u'web-app1-metrics-2017.11.07', u'web-app1-metrics-2017.11.04', u'web-app1-metrics-2017.11.05', u'web-app1-metrics-2017.11.02', u'web-app1-metrics-2017.11.03', u'web-app1-metrics-2017.11.01', u'web-app1-metrics-2017.11.08', u'web-app1-metrics-2017.11.09', u'web-app1-metrics-2017.11.11', u'web-app1-metrics-2017.11.10', u'web-app1-metrics-2017.11.13', u'web-app1-metrics-2017.11.12']
2017-12-04 14:43:40,043 INFO      ---deleting index web-app1-metrics-2017.11.06
2017-12-04 14:43:40,043 INFO      ---deleting index web-app1-metrics-2017.11.07
2017-12-04 14:43:40,043 INFO      ---deleting index web-app1-metrics-2017.11.04
2017-12-04 14:43:40,043 INFO      ---deleting index web-app1-metrics-2017.11.05
2017-12-04 14:43:40,043 INFO      ---deleting index web-app1-metrics-2017.11.02
2017-12-04 14:43:40,043 INFO      ---deleting index web-app1-metrics-2017.11.03
2017-12-04 14:43:40,043 INFO      ---deleting index web-app1-metrics-2017.11.01
2017-12-04 14:43:40,044 INFO      ---deleting index web-app1-metrics-2017.11.08
2017-12-04 14:43:40,044 INFO      ---deleting index web-app1-metrics-2017.11.09
2017-12-04 14:43:40,044 INFO      ---deleting index web-app1-metrics-2017.11.11
2017-12-04 14:43:40,044 INFO      ---deleting index web-app1-metrics-2017.11.10
2017-12-04 14:43:40,044 INFO      ---deleting index web-app1-metrics-2017.11.13
2017-12-04 14:43:40,044 INFO      ---deleting index web-app1-metrics-2017.11.12
2017-12-04 14:43:40,287 INFO      DELETE https://es.domain.com:443/web-app1-metrics-2017.11.01,web-app1-metrics-2017.11.02,web-app1-metrics-2017.11.03,web-app1-metrics-2017.11.04,web-app1-metrics-2017.11.05,web-app1-metrics-2017.11.06,web-app1-metrics-2017.11.07,web-app1-metrics-2017.11.08,web-app1-metrics-2017.11.09,web-app1-metrics-2017.11.10,web-app1-metrics-2017.11.11,web-app1-metrics-2017.11.12,web-app1-metrics-2017.11.13?master_timeout=30s [status:200 request:0.243s]
2017-12-04 14:43:40,417 INFO      Action ID: delete-index_web-app1-metrics, "delete_indices" completed.
2017-12-04 14:43:40,417 INFO      Preparing Action ID: delete-index_web-app2-metrics, "delete_indices"
2017-12-04 14:43:40,453 INFO      Trying Action ID: delete-index_web-app2-metrics, "delete_indices": Delete indices older than 1 month - based on index name, web-app2-metrics- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2017-12-04 14:43:40,491 INFO      Deleting selected indices: [u'web-app2-metrics-2017.11.03', u'web-app2-metrics-2017.11.01', u'web-app2-metrics-2017.11.02', u'web-app2-metrics-2017.11.04']
2017-12-04 14:43:40,492 INFO      ---deleting index web-app2-metrics-2017.11.03
2017-12-04 14:43:40,492 INFO      ---deleting index web-app2-metrics-2017.11.01
2017-12-04 14:43:40,492 INFO      ---deleting index web-app2-metrics-2017.11.02
2017-12-04 14:43:40,492 INFO      ---deleting index web-app2-metrics-2017.11.04
2017-12-04 14:43:40,566 INFO      DELETE https://es.domain.com:443/web-app2-metrics-2017.11.01,web-app2-metrics-2017.11.02,web-app2-metrics-2017.11.03,web-app2-metrics-2017.11.04?master_timeout=30s [status:200 request:0.074s]
2017-12-04 14:43:40,595 INFO      GET https://es.domain.com:443/ [status:200 request:0.002s]
2017-12-04 14:43:40,596 INFO      Action ID: delete-index_web-app2-metrics, "delete_indices" completed.
2017-12-04 14:43:40,596 INFO      Job completed.

Resources:

Basic Concourse Pipeline With Bash and Golang Jobs

From one of my previous posts, we went through the steps to setup a Concourse CI Server on Ubuntu .

What are we doing today?

Today we will setup a basic pipeline that executes 2 jobs, one using a alpine container that runs a couple of shell commands, and the other job will be using a Golang container to build and execute a golang app. I will also be experimenting with auto trigger, that will trigger the pipeline to run its jobs every 60 seconds.

Our Pipeline will look like the following:

Our Pipeline Definition:

bash-and-golang.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
resources:
- name: container-resource
  type: time
  source:
    interval: 60m

jobs:
- name: my-alpine-job
  plan:
  - get: container-resource
    trigger: true
  - task: vanilla-alpine-tasks
    params:
      OWNER: ruan
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
          tag: edge
      run:
        path: /bin/sh
        args:
        - -c
        - |
          apk update > /dev/null
          apk upgrade > /dev/null
          apk add curl > /dev/null
          echo "Public IP is: `curl -s http://ip.ruanbekker.com`"
          echo "Hostname is: $HOSTNAME"
          echo "Owner is: $OWNER"
          echo foo > /tmp/word.txt
          export MAGIC_WORD=`cat /tmp/word.txt`
          echo "Magic word is $MAGIC_WORD"
          cat > app.sh << EOF
          #!/usr/bin/env sh
          echo "Hello, World!"
          EOF
          chmod +x app.sh
          echo "Shell Script Executing:"
          ./app.sh

- name: my-golang-job
  plan:
  - get: container-resource
    trigger: true
  - task: golang-tasks
    params:
      OWNER: james
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: golang
          tag: '1.6'
      run:
        path: /bin/sh
        args:
        - -c
        - |
          echo "User: `whoami`"
          echo "Go Version: `go version`"
          echo "Hostname is: $HOSTNAME"
          echo "Owner is: $OWNER"
          echo bar > /tmp/word.txt
          export MAGIC_WORD=`cat /tmp/word.txt`
          echo "Magic word is $MAGIC_WORD"
          cat > app.go << EOF
          package main

          import "fmt"

          func main() {
            fmt.Println("Hello, World!")
          }
          EOF
          go build app.go
          echo "Go App Executing:"
          ./app

Login to Concourse:

Logon to concourse and set your target:

1
2
3
4
5
6
7
$ fly -t ci login --concourse-url=http://10.20.30.40:8080
logging in to team 'main'

username: admin
password:

target saved

List your targets:

1
2
3
$ fly targets
name      url                       team  expiry
ci        http://10.20.30.40:8080   main  Sat, 25 Nov 2017 23:30:55 UTC

Apply Configuration

Apply your Configuration:

1
2
3
4
5
6
7
8
9
$ fly -t ci set-pipeline -p bash-and-golang -c bash-and-golang.yml

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://10.20.30.40:8080/teams/main/pipelines/bash-and-golang

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command
  - click play next to the pipeline in the web ui

Unpause

Unpause your Pipeline:

1
2
$ fly -t ci unpause-pipeline -p bash-and-golang
unpaused 'bash-and-golang'

Trigger

Trigger your first job, which will be the Alpine job:

1
2
$ fly -t ci trigger-job --job bash-and-golang/my-alpine-job
started bash-and-golang/my-alpine-job #2

Trigger your second job, which will be the Golang job:

1
2
$ fly -t ci trigger-job --job bash-and-golang/my-golang-job
started bash-and-golang/my-golang-job #2

Remember, we can also monitor the output from the shell:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
$ fly -t ci trigger-job --job bash-and-golang/my-golang-job --watch
started bash-and-golang/my-golang-job #3

initializing
running /bin/sh -c echo "User: `whoami`"
echo "Go Version: `go version`"
echo "Hostname is: $HOSTNAME"
echo "Owner is: $OWNER"
echo bar > /tmp/word.txt
export MAGIC_WORD=`cat /tmp/word.txt`
echo "Magic word is $MAGIC_WORD"
cat > app.go << EOF
package main

import "fmt"

func main() {
  fmt.Println("Hello, World!")
}
EOF
go build app.go
echo "Go App Executing:"
./app

User: root
Go Version: go version go1.6.4 linux/amd64
Hostname is:
Owner is: james
Magic word is bar
Go App Executing:
Hello, World!
succeeded

Use Docker Secrets With MySQL on Docker Swarm

Today we will use Docker Secrets, more specifically store our MySQL Passwords in Secrets, which will be passed to our containers, so that we don’t use clear text passwords in our Compose files.

What is Docker Secrets:

In Docker, Docker Secrets are encrypted during transit and at rest in a Docker Swarm Cluster. The great thing about Docker Secrets is that you can manage these secrets from a central place, and the fact that it encrypts the data and transfers the data securely to the containers that needs the secrets. So you authorize which containers needs access to these secrets.

So instead of setting the MySQL Root Passwords in clear text, you will create the secrets, then in your docker-compose file, you will reference the secret name.

Deploy MySQL with Docker Secrets

We will deploy a Stack that contains MySQL and Adminer (WebUI for MySQL).

We will make the MySQL Service Persistent by setting a constraint to only run on the Manager node, as we will create the volume path on the host, and then map the host to the container so that the container can have persistent data. We will also create secrets for our MySQL Service so that we dont expose any plaintext passwords in our compose file.

Our Docker Compose file:

docker-compose.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
version: '3.3'

services:
  db:
    image: mysql
    secrets:
      - db_root_password
      - db_dba_password
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == manager]
      resources:
        reservations:
          memory: 128M
        limits:
          memory: 256M
    ports:
      - 3306:3306
    environment:
      MYSQL_USER: dba
      MYSQL_DATABASE: mydb
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_root_password
      MYSQL_PASSWORD_FILE: /run/secrets/db_dba_password
    networks:
      - appnet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - type: bind
        source: /opt/docker/volumes/mysql
        target: /var/lib/mysql

  adminer:
    image: adminer
    ports:
      - 8080:8080
    networks:
      - appnet

secrets:
  db_root_password:
    external: true
  db_dba_password:
    external: true

networks:
  appnet:
    external: true

Dependencies:

As we specified our secrets and networks as external resources, it needs to exist before we deploy our stack. We also need to create the directory for our mysql data, as the data will be mapped from our host to our container.

Create the Overlay Network:

1
$ docker network create --driver overlay appnet

Create the Secrets:

1
2
$ openssl rand -base64 12 | docker secret create db_root_password -
$ openssl rand -base64 12 | docker secret create db_dba_password -

List the Secrets:

1
2
3
4
$ docker secret ls
ID                          NAME                CREATED             UPDATED
jzhrwyxkiqt8v81ow0xjktqnw   db_root_password    12 seconds ago      12 seconds ago
plr6rbrqkqy7oplrd21pja3ol   db_dba_password     4 seconds ago       4 seconds ago

Inspect the secret, so that we can see that theres not value exposed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ docker secret inspect db_root_password
[
    {
        "ID": "jzhrwyxkiqt8v81ow0xjktqnw",
        "Version": {
            "Index": 982811
        },
        "CreatedAt": "2017-11-23T14:33:17.005968748Z",
        "UpdatedAt": "2017-11-23T14:33:17.005968748Z",
        "Spec": {
            "Name": "db_root_password",
            "Labels": {}
        }
    }
]

Create the Directory for MySQL:

1
$ mkdir -p /opt/docker/volumes/mysql

Deployment Time!

Deploy the stack:

1
2
3
$ docker stack deploy -c docker-compose.yml apps
Creating service apps_adminer
Creating service apps_db

As you can see the data of our MySQL container resides on our host, which makes the data persistent for the container:

1
2
$ ls /opt/docker/volumes/mysql/
auto.cnf  ca-key.pem  ca.pem  client-cert.pem  client-key.pem  ib_buffer_pool  ibdata1  ib_logfile0  ib_logfile1  ibtmp1  mydb  mysql  performance_schema  private_key.pem  public_key.pem  server-cert.pem  server-key.pem  sys

Connect to MySQL

The value of our secrets will reside under /run/secrets/ in our container, as we have mapped it to our mysql container, lets have a look at them:

1
2
$ docker exec -it $(docker ps -f name=apps_db -q) ls /run/secrets/
db_dba_password  db_root_password

View the actual value of the db_root_password:

1
2
$ docker exec -it $(docker ps -f name=apps_db -q) cat /run/secrets/db_root_password
mRpcY1eY2+wimf10

Connecting to MySQL:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ docker exec -it $(docker ps -f name=apps_db -q) mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mydb               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

As we have deployed adminer, you can access the Adminer WebUI on the Host’s IP and the Defined Port.

Testing Data Persistance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ docker exec -it $(docker ps -f name=apps_db -q) mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.

mysql> create database ruan;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mydb               |
| mysql              |
| performance_schema |
| ruan               |
| sys                |
+--------------------+
6 rows in set (0.00 sec)

mysql> exit;
Bye

Verify the hostname of our container, before we kill the container:

1
2
$ docker exec -it $(docker ps -f name=apps_db -q) hostname
bdedb54bbc2b

Kill the container:

1
2
$ docker kill $(docker ps -f name=apps_db -q)
bdedb54bbc2b

Verify the status of the MySQL Service, as we can see the service count is 0, so the container was succesfully killed.

1
2
3
$ docker service ls -f name=apps_db
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
nzf96q05fktm        apps_db             replicated          0/1                 mysql:latest        *:3306->3306/tcp

After waiting for a couple of seconds, we can see the service is in service again, then check the hostname so that we can confirm that its a new container:

1
2
3
4
5
6
$ docker service ls -f name=apps_db
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
nzf96q05fktm        apps_db             replicated          1/1                 mysql:latest        *:3306->3306/tcp

$ docker exec -it $(docker ps -f name=apps_db -q) hostname
95c15c89f891

Logong to MySQL again and verify if our perviously created database is still there:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ docker exec -it $(docker ps -f name=apps_db -q) mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mydb               |
| mysql              |
| performance_schema |
| ruan               |
| sys                |
+--------------------+
6 rows in set (0.01 sec)

By design docker is stateless, but as we mapped the host’s path to the container our data is persistent. As we have set a constraint so that the container must only spin up on this node, the container will always have access to the data path.

Setup a 3 Node Galera MariaDB Cluster on Ubuntu 16

Today we will setup a 3-Node Galera MariaDB Cluster which is a Multi Master MySQL/MariaDB Cluster on Ubuntu 16.04

Our Server Details:

1
2
3
172.31.11.174     mysql-1
172.31.13.206     mysql-2
172.31.6.93       mysql-3

Update Repo Index and Upgrade:

Update the repository indexes and install the needed packages:

1
$ sudo apt update && sudo apt upgrade -y

Install the needed repository and packages:

1
2
3
4
5
$ apt install software-properties-common -y
$ apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
$ add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mirror.lstn.net/mariadb/repo/10.1/ubuntu xenial main'
$ apt update
$ apt install mariadb-server rsync -y

Configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat > /etc/mysql/conf.d/galera.cnf << EOF
[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

# Galera Cluster Configuration
wsrep_cluster_name="my-galera-cluster"
wsrep_cluster_address="gcomm://172.31.11.174,172.31.13.206,172.31.6.93"
# Galera Synchronization Configuration
wsrep_sst_method=rsync

# Galera Node Configuration
wsrep_node_address="172.31.11.174"
wsrep_node_name="mysql-1"
EOF

Comment out bind-address, so that MariaDB process is reachable from other nodes, by default it wont be in the config, but just to make sure, if it is uncommented, comment the config:

/etc/mysql/my.cnf
1
# bind-address = 127.0.0.1

Stop the MariaDB Process:

1
$ systemctl stop mariadb

Note: Repeat the above steps on all 3 nodes.

Initialize the Cluster:

On the First Node, Initialize the Galera Cluster:

1
2
$ /usr/bin/galera_new_cluster
$ systemctl enable mariadb

Check how many nodes are active in the Cluster:

1
2
3
4
5
6
7
$ mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
Enter password:
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 1     |
+--------------------+-------+

Node-2: Start and Enable MariaDB

1
2
$ systemctl start mariadb
$ systemctl enable mariadb

Verify that the Node has checked in with the Cluster:

1
2
3
4
5
6
7
$ mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
Enter password:
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 2     |
+--------------------+-------+

Node-3: Start and Enable MariaDB

1
2
$ systemctl start mariadb
$ systemctl enable mariadb

Verify that the Node has checked in with the Cluster:

1
2
3
4
5
6
7
$ mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
Enter password:
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+

Create a Database, Table and Record:

Write some data to the table, then reboot the node, in this example on node-1, then logon to node-2 check the number of nodes that’s active in the cluster, which should be 2, then at the same time, look if the data is replicated:

Node-1: Writing the Data to Our Galera Cluster

1
2
3
4
5
6
7
8
9
10
11
MariaDB [(none)]> create database test;
MariaDB [(none)]> use test;
MariaDB [test]>   create database test;
MariaDB [test]>   create table foo (name VARCHAR(20));
MariaDB [test]>   insert into foo values('ruan');
MariaDB [test]>   select * from foo;
+------+
| name |
+------+
| ruan |
+------+

Now that our data is in our database, reboot the node, logon to node-2 and check if the data is replicated:

1
2
3
4
5
6
7
8
$ mysql -u root -p
MariaDB [(none)]> use test;
MariaDB [test]>   select * from foo;
+------+
| name |
+------+
| ruan |
+------+

While the one node is rebooting, check how many nodes are checked into our cluster:

1
2
3
4
5
6
7
$ mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
Enter password:
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 2     |
+--------------------+-------+

Our data is replicated, and after waiting for a couple of seconds, we retry our command to see if the rebooted node checked into the cluster:

1
2
3
4
5
6
7
$ mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
Enter password:
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+

We can confirm that the node that was rebooted, has checked in with the cluster again.

Firewall Rules opened while testing:

TCP: 3306, 4567, 4568, 4444 UDP: 4567

Delete Old Items With Amazons DynamoDB TTL Feature

As you may know a DynamoDB Table’s Partition Splits on 2 factors, Read/Write Capacity Units and when Storage goes over 10GB.

Automatically Deleting Old Data in DynamoDB:

With the TTL Feature in DynamoDB, we can enable TTL on a Attribute on our Table, the attributes value needs to have an epoc time value, more specifically, when the current time is the same as the value of on of the items attribute value, that item will be expired, which will be deleted.

What we will be doing:

  • Use Boto3 in Python
  • Create DynamoDB Table: ‘session-table’
  • Set TTL Attribute on ‘ExpirationTime’, so whenever the epoch time is equals to the AttributeValue it will delete the item
  • Do one PUT Item with 48 Hours expiry Date from the Write
  • Do 240 PUT Items with 24 Hours expiry Date from the Write
  • Verify after 24 hours if only one item is in our table.

Pre-Requisites:

Install the AWS CLI, Boto3 and configure your credentials, so that boto3 can read from your credential provider:

1
2
3
4
5
6
7
$ pip install awscli
$ pip install boto3
$ aws configure
AWS Access Key ID [****************XYZ]:
AWS Secret Access Key [****************xyz]:
Default region name [eu-west-1]:
Default output format [json]:

Create the Table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ python

import boto3
session = boto3.Session(region_name='eu-west-1', profile_name='default')
dynamodb = session.resource('dynamodb')
table = dynamodb.create_table(
    TableName='session-table',
    KeySchema=[
        {
            'AttributeName': 'sessionid',
            'KeyType': 'HASH'
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'sessionid',
            'AttributeType': 'S'
        }
    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 2,
        'WriteCapacityUnits': 2
    }
)

From the Console, enable TTL and set the TTL Attribute on ExpirationTime

Write Data to DynamoDB

We have 2 functions that will write the current epoch time to the CreationTime attribute and ExpirationTime will have the current time plus the 24 hours in seconds, which will be used for the 240 items that will be written using the for loop and the other function with the 48 hours of seconds, which will be a single write item.

Then we will just write random data to the session data attribute:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
import boto3
import time
import random
from uuid import uuid4

names = ['james', 'john', 'steve', 'peter', 'frank', 'steven', 'jonathan', 'stephen', 'will', 'adam', 'william']
retailer = ['shoprite', 'edgars', 'pnp', 'bestbuy', 'ok', 'grocer-a', 'amazon', 'seveneleven', 'shop-a']

session = boto3.Session(region_name='eu-west-1', profile_name='dev')
ddb = session.resource('dynamodb')
client = ddb.Table('session-table')

def current_time():
    int(time.time())

def current_time():
    return int(time.time())

def expiration_time():
    return int(time.time()) + 86400

def 48h_expiration_time():
    return int(time.time()) + 172800

# expiry on 48 hours
client.put_item(
    Item={
        'sessionid': str(uuid4()),
        'CreationTime': current_time(),
        'ExpirationTime': 48h_expiration_time(),
        'SessionData': {
            'Name': random.choice(names),
            'Retailer': random.choice(retailer),
            'TimeOfTransaction': current_time(),
            'Amount': random.randint(100,9000)
        }
    }
)

# expiry on 24 hours
for x in xrange(240):
    time.sleep(1)
    client.put_item(
        Item={
            'sessionid': str(uuid4()),
            'CreationTime': current_time(),
            'ExpirationTime': expiration_time(),
            'SessionData': {
                'Name': random.choice(names),
                'Retailer': random.choice(retailer),
                'TimeOfTransaction': current_time(),
                'Amount': random.randint(100,9000)
            }
        }
    )

Verify:

Verify after 24 hours if the item with the 48 hour expiration time is still in our table:

1
2
3
4
5
6
7
8
9
10
11
12
13
client.get_item( Key={'sessionid': '69c2a472-f70e-4d72-b25f-e27573696b0c'} )['Item']

{
    u'ExpirationTime': Decimal('1510672221'),
    u'CreationTime': Decimal('1510585821'),
    u'sessionid': u'69c2a472-f70e-4d72-b25f-e27573696b0c',
    u'SessionData': {
        u'Amount': Decimal('3553'),
        u'Retailer': u'amazon',
        u'TimeOfTransaction': Decimal('1510585821'),
        u'Name': u'steve'
    }
}

Which we can see is still there, when doing a GET item on one of our 24 hour expired items, we can see that its no longer there:

1
2
3
4
5
client.get_item( Key={'sessionid': '70b9fc8c-19c4-49d3-bf63-046e992335af'} )['Item']

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'Item'

Doing a SCAN operation, we should see one item:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import json
r = client.scan(TableName='session-table', Limit=10, Select='COUNT', ReturnConsumedCapacity='TOTAL')

print(json.dumps(r, indent=4))
{
    "Count": 1,
    "ScannedCount": 1,
    "ConsumedCapacity": {
        "CapacityUnits": 0.5,
        "TableName": "session-table"
    },
    "ResponseMetadata": {
        "RetryAttempts": 0,
        "HTTPStatusCode": 200,
        "RequestId": "",
        "HTTPHeaders": {
            "x-amzn-requestid": "",
            "content-length": "107",
            "server": "Server",
            "connection": "keep-alive",
            "x-amz-crc32": "2228370918",
            "date": "Tue, 14 Nov 2017 12:02:31 GMT",
            "content-type": "application/x-amz-json-1.0"
        }
    }
}

So we can confirm that the TTL feature expires the data based on the epoch value we provide our item.

Delete the Table:

1
client.delete(TableName='session-table')

Resources:

Update Your Ghost Blog With the Ghost-CLI

If you installed your Ghost Blog with the Ghost-CLI, you can easily upgrade your Ghost version using the CLI.

Backups

Backup your blog by exporting the json via the Ghost Admin Interface, and also update your content directory:

1
2
3
$ sudo su - ghost
$ cd /var/www/ghost
$ tar -zcf /home/ghost/backups/ghost-content-$(date +%F).tar.gz content

Check the Current Version:

1
2
3
4
$ ghost status

Version:
1.17.0

Update Ghost:

1
2
$ npm i -g ghost-cli
$ ghost update

Verify Version:

1
2
3
4
$ ghost status

Version:
1.17.0

No need to restart Ghost as the update function restarted the process already.

Resources:

Use the Reindex API on Elasticsearch to Reindex Your Data

A Basic Example of Reindexing Data with the /_reindex API on Elasticsearch:

Provision Elasticsearch with Docker:

I will be using Elasticsearch on Docker for this Example:

1
$ docker run -itd --name elasticsearch --publish 9200:9200 elasticsearch:alpine

Create Indexes:

Create 3 Indexes and POST 2 Documents to each Index:

1
2
3
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11.20
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11.21
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11.21

Create the Index where we will reindex the data to:

1
$ curl -XPUT http://127.0.0.1:9200/animals-2017.11 -d '{"settings": {"number_of_shards": 5, "number_of_replicas": 0}}'

POST 2 documents to each index:

1
2
3
4
5
6
7
8
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.20/name/ -d '{"name": "max", "type": "labrador"}'
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.20/name/ -d '{"name": "sam", "type": "pooch"}'

$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.21/name/ -d '{"name": "doggie", "type": "bulldog"}'
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.21/name/ -d '{"name": "james", "type": "huskey"}'

$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.22/name/ -d '{"name": "sarah", "type": "poodle"}'
$ curl -XPOST http://127.0.0.1:9200/animals-2017.11.22/name/ -d '{"name": "frank", "type": "alsation"}'

View the Indexes:

As you can see we have 2 documents per index, and a empty index for the data that we would like to reindex to:

1
2
3
4
5
6
$ curl -XGET http://127.0.0.1:9200/_cat/indices?v
health status index               uuid                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   animals-2017.11.20  AxRYUfNpQ5ev2mdZf0bYrw   5   1          2            0      8.9kb          8.9kb
green  open   animals-2017.11     1T6TkYWwRuerIZ5_np1B0w   5   0          0            0      1.5kb          1.5kb
yellow open   animals-2017.11.22  fCdaRyBZRiWyQ3tZLhdBrw   5   1          2            0      8.9kb          8.9kb
yellow open   animals-2017.11.21  4Ei9zMDITHy1dI8lIzfjjA   5   1          2            0      8.9kb          8.9kb

Reindex the Data to our Monthly Index:

We will define our query to match all the indexes that has the data and reindex to our new index animals-2017.11:

1
2
$ curl -XPOST http://127.0.0.1:9200/_reindex -d '{"source": {"index": "animals-2017.11.*"}, "dest": {"index": "animals-2017.11"} }'
{"took":219,"timed_out":false,"total":6,"updated":0,"created":6,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}

View the Indexes:

1
2
3
4
5
6
$ curl -XGET http://127.0.0.1:9200/_cat/indices?v
health status index               uuid                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   animals-2017.11.20  AxRYUfNpQ5ev2mdZf0bYrw   5   1          2            0      8.9kb          8.9kb
green  open   animals-2017.11     1T6TkYWwRuerIZ5_np1B0w   5   0          6            0     20.2kb         20.2kb
yellow open   animals-2017.11.22  fCdaRyBZRiWyQ3tZLhdBrw   5   1          2            0      8.9kb          8.9kb
yellow open   animals-2017.11.21  4Ei9zMDITHy1dI8lIzfjjA   5   1          2            0      8.9kb          8.9kb

Delete the Old Indexes:

As your data is now reindexed, we can safely remove our old indexes:

1
$ curl -XDELETE 'http://127.0.0.1:9200/animals-2017.11.*'

To verify:

1
2
3
$ curl -XGET http://127.0.0.1:9200/_cat/indices?v
health status index               uuid                     pri rep docs.count docs.deleted store.size pri.store.size
green  open   animals-2017.11     1T6TkYWwRuerIZ5_np1B0w   5   0          6            0     20.2kb         20.2kb

Resources:

Using Elasticsearch Curator to Reindex Data

Today I was using Elasticsearch Curator to reindex indices that was created on a daily basis, to reindex all the data to one index. I used this route as the old data will not be accessed frequently.

Install Elasticsearch Curator

1
2
$ docker run -it python:2.7-alpine sh
$ pip install elasticsearch-curator

Create Configs:

Create the curator config:

config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - es.endpoint.com
  port: 443
  use_ssl: True
  ssl_no_validate: False
  http_auth: admin:pass
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['urllib3']

Create the action config:

action-reindex.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: "Reindex index-2017.10.{30,31} into new-index-2017.10"
    action: reindex
    options:
      disable_action: False
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: ['index-2017.10.*']
        dest:
          index: new-index-2017.10
    filters:
    - filtertype: none

Create the Elasticsearch Index:

Create the Index where we will reindex the data to:

1
$ curl -XPUT http://es.endpoint.com/new-index-2017.10 -d '{"settings": {"number_of_shards": 5, "number_of_replicas": 1}}'

Run the Curator:

1
2
3
4
5
6
7
$ curator --config curator.yml action-reindex.yml

2017-11-22 14:18:15,138 INFO      Task "reindex from [index-2017.10.*] to [index-2017.10]" with task_id "Za-sn0z3Q9-75xCMRwJ3-A:15782886" has been running for 928.948195354 seconds
2017-11-22 14:18:24,152 INFO      GET https://es.endpoint.com:443/_tasks/Za-sn0z3Q9-75xCMRwJ3-A%3A15782886 [status:200 request:0.005s]
2017-11-22 14:18:24,153 INFO      Task "reindex from [index-2017.10.*] to [new-index-2017.10]" with task_id "Za-sn0z3Q9-75xCMRwJ3-A:15782886" has been running for 937.962740393 seconds
2017-11-22 14:22:23,171 INFO      Action ID: 1, "reindex" completed.
2017-11-22 14:22:23,171 INFO      Job completed.

Resources:

Using Elasticdump to Backup Elasticsearch Indexes to JSON

We will use Elasticdump to dump data from Elasticsearch to json files on disk, then delete the index, then restore data back to elasticsearch

Install Elasticdump:

1
2
$ docker run -it node:alpine sh
$ npm install elasticdump -g

Create a Index:

1
2
$ curl -XPUT http://10.79.2.193:9200/test-index
{"acknowledged":true}

Ingest Some Data into the Index:

1
2
3
4
5
$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc1 -d '{"name": "ruan", "age": 30}'
{"_index":"test-index","_type":"docs","_id":"doc1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc2 -d '{"name": "stefan", "age": 29}'
{"_index":"test-index","_type":"docs","_id":"doc2","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

Elasticdump to dump the ata

First dump the mappings:

1
2
3
4
5
6
7
$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index_mapping.json --type=mapping
Mon, 26 Jun 2017 14:15:34 GMT | starting dump
Mon, 26 Jun 2017 14:15:34 GMT | got 1 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:34 GMT | sent 1 objects to destination file, wrote 1
Mon, 26 Jun 2017 14:15:34 GMT | got 0 objects from source elasticsearch (offset: 1)
Mon, 26 Jun 2017 14:15:34 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:15:34 GMT | dump complete

Then dump the data:

1
2
3
4
5
6
7
$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index.json --type=data
Mon, 26 Jun 2017 14:15:43 GMT | starting dump
Mon, 26 Jun 2017 14:15:43 GMT | got 2 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:43 GMT | sent 2 objects to destination file, wrote 2
Mon, 26 Jun 2017 14:15:43 GMT | got 0 objects from source elasticsearch (offset: 2)
Mon, 26 Jun 2017 14:15:43 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:15:43 GMT | dump complete

Preview the Metadata

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ cat /opt/backup/elasticsearch/es_test-index_mapping.json | python -m json.tool
{
    "test-index": {
        "mappings": {
            "docs": {
                "properties": {
                    "age": {
                        "type": "long"
                    },
                    "name": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

Preview the Data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cat /opt/backup/elasticsearch/es_test-index.json | jq
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc1",
  "_score": 1,
  "_source": {
    "name": "ruan",
    "age": 30
  }
}
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc2",
  "_score": 1,
  "_source": {
    "name": "stefan",
    "age": 29
  }
}

Restore The Data

Let’s test the restoring part, go ahead and delete The index:

1
2
$ curl -XDELETE http://10.79.2.193:9200/test-index
{"acknowledged":true}

Restore the Index by Importing the Mapping:

1
2
3
4
5
6
7
$ elasticdump --input=/opt/backup/elasticsearch/es_test-index_mapping.json --output=http://10.79.2.193:9200/test-index --type=mapping
Mon, 26 Jun 2017 14:51:48 GMT | starting dump
Mon, 26 Jun 2017 14:51:48 GMT | got 1 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:51:48 GMT | sent 1 objects to destination elasticsearch, wrote 1
Mon, 26 Jun 2017 14:51:48 GMT | got 0 objects from source file (offset: 1)
Mon, 26 Jun 2017 14:51:48 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:51:48 GMT | dump complete

Verify to see if the Index Exist:

1
2
3
$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E '(docs.count|test)'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          0            0       650b           650b

Restore the Data for the Index:

Use elasticdump to restore the data from json to elasticsearch:

1
2
3
4
5
6
7
$ elasticdump --input=/opt/backup/elasticsearch/es_test-index.json --output=http://10.79.2.193:9200/test-index --type=data
Mon, 26 Jun 2017 14:53:56 GMT | starting dump
Mon, 26 Jun 2017 14:53:56 GMT | got 2 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:53:56 GMT | sent 2 objects to destination elasticsearch, wrote 2
Mon, 26 Jun 2017 14:53:56 GMT | got 0 objects from source file (offset: 2)
Mon, 26 Jun 2017 14:53:56 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:53:56 GMT | dump complete

Verify to see if the Documents was Ingested:

1
2
3
$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E '(docs.count|test)'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          2            0       650b           650b

Preview the Data from Elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ curl -s -XGET http://10.79.2.193:9200/test-index/_search?pretty

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc1",
      "_score" : 1.0,
      "_source" : {
        "name" : "ruan",
        "age" : 30
      }
    }, {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc2",
      "_score" : 1.0,
      "_source" : {
        "name" : "stefan",
        "age" : 29
      }
    } ]
  }
}

Resources:

Routing Web Traffic With a SOCKS Tunnel

I wanted to access a Non Standard HTTP Port on one of my RaspberryPi Hosts, which was not directly available to the Internet, so I have chosen to establish a SOCKS Tunnel to achieve that.

Web Application on my LAN

Getting my RaspberryPi’s Private IP Address:

1
2
$ ifconfig eth0 | grep 'inet 192' | awk '{print $2}'
192.168.1.118

For demonstration purposes, I will use Python’s SimpleHTTPServer:

1
2
3
4
5
$ mkdir web
$ cd web
$ echo 'yeehaa' > index.html
$ python -m SimpleHTTPServer 5050
Serving HTTP on 0.0.0.0 port 5050 ...

Establish the SOCKS Tunnel

From my laptop, establishing the SOCKS Tunnel with SSH, you can use -f to fork it in the background:

1
$ ssh -D 8157 -CqN user@home.domain.com

Configure your Browser:

Configure your browser to Proxy via:

  • Host: localhost
  • Port: 8157

Now when you access the destined host’s private ip, you will get a response:

1
2
Browse to http://192.168.1.118:5050/ and in my case my response is:
-> yeehaa