Ruan Bekker's Blog

From a Curious mind to Posts on Github

Generate Random Characters With Python Using Random and String Modules

When generating random characters for whatever reason, passwords, secrets-keys etc, you could use the uuid module, which looks like this:

Random String with UUID
1
2
3
>>> from uuid import uuid4
>>> print("Your string is: {0}".format(uuid4()) )
Your string is: 53a6e1a7-a2c7-488e-bed9-d76662de9c5f

But if you want to be more specific, like digits, letters, capitalization etc, you can use the string and random modules to do so. First we will generate a random string containing only letters:

Random String with letters
1
2
3
4
5
6
7
8
9
>>> from string import ascii_letters, punctuation, digits
>>> from random import choice, randint
>>> min = 12
>>> max = 15
>>> string_format = ascii_letters
>>> generated_string = "".join(choice(string_format) for x in range(randint(min, max)))

>>> print("Your String is: {0}".format(generated_string))
Your String is: zNeUFluvZwED

As you can see, you have a randomized string which will be always at least 12 characters and max 15 characters, which is lower and upper case. You can also use the lower and upper functions if you want to capitalize or lower case your string:

1
2
3
4
5
>>> generated_string.lower()
'zneufluvzwed'

>>> generated_string.upper()
'ZNEUFLUVZWED'

Let’s add some logic so that we can have a more randomized characters with digits, punctuations etc:

Random String with Letters, Punctuations and Digits
1
2
3
4
5
6
7
8
>>> from string import ascii_letters, punctuation, digits
>>> from random import choice, randint
>>> min = 12
>>> max = 15
>>> string_format = ascii_letters + punctuation + digits
>>> generated_string = "".join(choice(string_format) for x in range(randint(min, max)))
>>> print("Your String is: {0}".format(generated_string))
Your String is: Bu>}x_/-H5)fLAr

More Python related blog posts.

Manage Scaleway Instances via Their API Like a Boss With Their Command Line Tool Scw

Let’s set things straight: I am a command line fan boy, If I can do the things I have to do with a command line interface, i’m happy! And that means automation ftw! :D

Scaleway Command Line Interface:

I have been using Scaleway for about 2 years now, and absolutely loving their services! So I recently found their command line interface utility: scw, which is written in golang and has a very similar feel to docker.

Install the SCW CLI Tool:

A golang environment is needed and I will be using docker to drop myself into a golang environment and then install the scw utility:

1
2
3
4
$ docker run -it golang:alpine sh
$ apk update
$ apk add openssl git openssh curl
$ go get -u github.com/scaleway/scaleway-cli/cmd/scw

Verify that it was installed:

1
2
$ scw --version
scw version v1.16+dev, build

Awesome sauce!

Authentication:

When we authenticate to Scaleway, it will prompt you to upload your public ssh key, as I am doing this in a container I have no ssh keys, so therefore will generate one before I authenticate.

Generate the SSH Key:

1
2
3
4
5
6
7
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

Now loging to Scaleway using the cli tools:

1
2
3
4
5
6
7
8
9
10
11
12
$ scw login
Login (cloud.scaleway.com): <youremail@domain.com>
Password:
Do you want to upload an SSH key ?
[0] I don't want to upload a key !
[1] id_rsa.pub
Which [id]: 1

You are now authenticated on Scaleway.com as Ruan.
You can list your existing servers using `scw ps` or create a new one using `scw run ubuntu-xenial`.
You can get a list of all available commands using `scw -h` and get more usage examples on github.com/scaleway/scaleway-cli.
Happy cloud riding.

Sweeet!

Getting Info from Scaleway

List Instance Types:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ scw products servers
COMMERCIAL TYPE     ARCH     CPUs      RAM  BAREMETAL
ARM64-128GB        arm64       64   137 GB      false
ARM64-16GB         arm64       16    17 GB      false
ARM64-2GB          arm64        4   2.1 GB      false
ARM64-32GB         arm64       32    34 GB      false
ARM64-4GB          arm64        6   4.3 GB      false
ARM64-64GB         arm64       48    69 GB      false
ARM64-8GB          arm64        8   8.6 GB      false
C1                   arm        4   2.1 GB       true
C2L               x86_64        8    34 GB       true
C2M               x86_64        8    17 GB       true
C2S               x86_64        4   8.6 GB       true
START1-L          x86_64        8   8.6 GB      false
START1-M          x86_64        4   4.3 GB      false
START1-S          x86_64        2   2.1 GB      false
START1-XS         x86_64        1   1.1 GB      false
VC1L              x86_64        6   8.6 GB      false
VC1M              x86_64        4   4.3 GB      false
VC1S              x86_64        2   2.1 GB      false
X64-120GB         x86_64       12   129 GB      false
X64-15GB          x86_64        6    16 GB      false
X64-30GB          x86_64        8    32 GB      false
X64-60GB          x86_64       10    64 GB      false

Get a list of available Images, in my case I am just looking for Ubuntu:

1
2
3
$ scw images | grep -i ubuntu
Ubuntu_Bionic               latest              a21bb700            11 days             [ams1 par1]         [x86_64]
Ubuntu_Mini_Xenial_25G      latest              bc75c00b            13 days             [ams1 par1]         [x86_64]

List Running Instances:

1
2
3
4
5
$ scw ps
SERVER ID           IMAGE                       ZONE                CREATED             STATUS              PORTS               NAME                  COMMERCIAL TYPE
abc123de            Ubuntu_Xenial_16_04_lates   ams1                5 weeks             running             xx.xx.xx.xx         scw-elasticsearch-01  ARM64-4GB
abc456de            ruan-docker-swarm-17_03     par1                10 months           running             xx.xx.xxx.xxx       scw-swarm-manager-01  VC1M
...

List All Instances (Running, Stopped, Started, etc):

1
2
3
4
$ scw ps -a
SERVER ID           IMAGE                       ZONE                CREATED             STATUS              PORTS               NAME                  COMMERCIAL TYPE
abc123df            Ubuntu_Xenial_16_04_lates   ams1                5 weeks             stopped             xx.xx.xx.xx         scw-elasticsearch-02  ARM64-4GB
...

List Instances with a filter based on its name:

1
2
3
$ scw ps -f name=scw-swarm-worker-02
SERVER ID           IMAGE               ZONE                CREATED             STATUS              PORTS               NAME                COMMERCIAL TYPE
1234abcd            Ubuntu_Xenial       par1                8 minutes           running             xx.xx.xxx.xxx       scw-swarm-worker-2  START1-XS

List the Latest Instance that was created:

1
2
3
$ scw ps -l
SERVER ID           IMAGE               ZONE                CREATED             STATUS              PORTS               NAME                COMMERCIAL TYPE
1234abce            Ubuntu_Xenial       par1                6 minutes           running             xx.xx.xxx.xxx       scw-swarm-worker-3  START1-XS

Create Instances:

In my scenario, I would like to create a instance named docker-swarm-worker-4 with the instance type START1-XS in the Paris datacenter, and I will be using my key that I have uploaded, also the image id that I passed, was retrieved when listing for images:

1
2
$ scw --region=par1 create --commercial-type=START1-XS --ip-address=dynamic --ipv6=false --name="docker-swarm-worker-4" --tmp-ssh-key=false  bc75c00b
<response: random uuid string>

Now that the instance is created, we can start it by calling either the name or the id:

1
$ scw start docker-swarm-worker-4

To verify the status of the instance, we can do:

1
2
3
$ scw ps -l
SERVER ID           IMAGE               ZONE                CREATED             STATUS              PORTS               NAME                   COMMERCIAL TYPE
102abc34            Ubuntu_Xenial                           28 seconds          starting                                docker-swarm-worker-4  START1-XS

At this moment it is still starting, after waiting a minute or so, run it again:

1
2
3
$ scw ps -l
SERVER ID           IMAGE               ZONE                CREATED             STATUS              PORTS               NAME                   COMMERCIAL TYPE
102abc34            Ubuntu_Xenial       par1                About a minute      running             xx.xx.xx.xx         docker-swarm-worker-4  START1-XS

As we can see its in a running state, so we are good to access our instance. You have 2 options to access your server, via exec and ssh.

1
2
$ scw exec docker-swarm-worker-4 /bin/bash
root@docker-swarm-worker-4:~

or via SSH:

1
2
$ ssh root@xx.xx.xx.xx
root@docker-swarm-worker-4:~

If you would like to access your server without uploading your SSH key to your account, you can pass --tmp-ssh-key=true as in:

1
$ scw --region=par1 create --commercial-type=START1-XS --ip-address=dynamic --ipv6=false --name="scw-temp-instance" --tmp-ssh-key=true  bc75c00b

Terminating Resources:

This wil stop, terminate the instance with the associated volumes and reserved ip

1
2
$ scw stop --terminate=true scw-temp-instance
scw-temp-instance

If you had to remove a volume that is not needed, or unused:

1
$ scw rmi test-1-snapshot-<long-string>--2018-04-26_12:42

To logout:

1
$ scw logout

Resources:

Have a look at Scaleway-CLI Documentation and their Website for more info, and have a look at their new START1-XS instance types, that is only 1.99 Euro’s, that is insane!

Personally love what they are doing, feel free to head over to their pricing page to see some sweet deals!

Temporary IAM Credentials From EC2 Instance Metadata Using Python

From a Best Practice Perspective its good not having to pass sensitive information around, and especially not hard coding them.

Best Practice: Security

One good way is to use SSM with KMS to Encrypt/Decrypt them, but since EC2 has a Metadata Service available, we can make use of that to retrieve temporary credentials. One requirement though, is that the instance will require an IAM Role where the code will be executed on. The IAM Role also needs to have sufficient privileges to be able to execute, whatever you need to do.

The 12 Factor Methodology however states to use config in your environment variables, but from the application logic, its easy to save it in our environment.

Scenario: Applications on AWS EC2

When you run applications on Amazon EC2 the nodes has access to the EC2 Metadata Service, so in this case our IAM Role has a Policy that authorizes GetItem on our DynamoDB table, therefore we can define our code with no sensitive information, as the code will do all the work to get the credentials and use the credentials to access DynamoDB.

Use Temporary Credentials to Read from DynamoDB

In this example we will get the temporary credentials from the metadata service, then define the temporary credentials in our session to authorize our request against dynamodb to read from our table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
>>> import boto3
>>> from botocore.utils import InstanceMetadataFetcher
>>> from botocore.credentials import InstanceMetadataProvider
>>> provider = InstanceMetadataProvider(iam_role_fetcher=InstanceMetadataFetcher(timeout=1000, num_attempts=2))
>>> creds = provider.load()

>>> session = boto3.Session(
    aws_access_key_id=creds.access_key,
    aws_secret_access_key=creds.secret_key,
    aws_session_token=creds.token
)

>>> ddb = session.client('dynamodb')

>>> response = ddb.get_item(
    TableName='my-dynamodb-table',
    Key={
        'node_type': {
            'S': 'primary_manager'
        }
    }
)

>>> print(response['Item']['ip']['S'])
'10.0.0.32

Also, when you are logged onto the EC2 instance, you can use curl to see the temporary credentials information:

1
2
3
4
5
6
7
8
9
10
11
$ iam_role_name=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/)
$ curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/${iam_role_name}
{
  "Code" : "Success",
  "LastUpdated" : "2018-05-09T14:25:48Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "",
  "SecretAccessKey" : "",
  "Token" : "",
  "Expiration" : "2018-05-09T20:46:55Z"
}

Use Python Requests to Interact With the iTunes API to Search for Music Info

Tutorial on using Python Requests and using Apple iTunes Music API, where we will be doing the following:

  • Basics of using the Requests module
  • Query iTunes API on Songs by Artist
  • Query iTunes API on Artists Info
  • Query iTunes API on All Albums by Artist
  • Query iTunes API on Top 5 Albums
  • Query iTunes API on Multipe Artists

Resources:

Install the Request Module:

1
2
3
$ virtualenv -p /usr/bin/python .venv
$ source .venv/bin/activate
$ pip install requests

Basic Usage of Requests:

In this demonstration we will only use the GET HTTP Method.

Make the GET Request to the endpoint:

1
2
>>> import requests
>>> response = requests.get('https://itunes.apple.com/search?term=guns+and+roses&limit=1')

View the HTTP Status Code of the Response:

1
2
>>> response.status_code
200

To view some of the status codes of the request library:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
>>> requests.codes.ok
200
>>> requests.codes.no_content
204
>>> requests.codes.temporary_redirect
307
>>> requests.codes.permanent_redirect
308
>>> requests.codes.bad
400
>>> requests.codes.not_found
404
>>> requests.codes.bad_gateway
502

Call .ok for the status lookup, the boolean answer will indicate if it responded with a 200 OK:

1
2
>>> response.ok
True

Measure the amount of time the request took:

1
2
>>> response.elapsed.total_seconds()
0.706043

View the content of the response:

1
2
>>> response.content
'\n\n\n{\n "resultCount":1,\n "results": [\n{"wrapperType":"track", "kind":"song", "artistId":106621, "collectionId":5669937, "trackId":5669911, "artistName":"Guns N\' Roses", "collectionName":"Greatest Hits", "trackName":"Sweet Child O\' Mine", "collectionCensoredName":"Greatest Hits", "trackCensoredName":"Sweet Child O\' Mine", "artistViewUrl":"https://itunes.apple.com/us/artist/guns-n-roses/106621?uo=4", "collectionViewUrl":"https://itunes.apple.com/us/album/sweet-child-o-mine/5669937?i=5669911&uo=4", "trackViewUrl":"https://itunes.apple.com/us/album/sweet-child-o-mine/5669937?i=5669911&uo=4", \n"previewUrl":"https://audio-ssl.itunes.apple.com/apple-assets-us-std-000001/Music6/v4/f2/7d/73/f27d7346-de92-bdc6-e148-56a3da406005/mzaf_2747902348777129728.plus.aac.p.m4a", "artworkUrl30":"https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/30x30bb.jpg", "artworkUrl60":"https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/60x60bb.jpg", "artworkUrl100":"https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/100x100bb.jpg", "collectionPrice":9.99, "trackPrice":1.29, "releaseDate":"1987-07-21T07:00:00Z", "collectionExplicitness":"notExplicit", "trackExplicitness":"notExplicit", "discCount":1, "discNumber":1, "trackCount":14, "trackNumber":2, "trackTimeMillis":355267, "country":"USA", "currency":"USD", "primaryGenreName":"Rock", "isStreamable":true}]\n}\n\n\n'

View the content in json format:

1
2
>>> response.json()
{u'resultCount': 1, u'results': [{u'collectionExplicitness': u'notExplicit', u'releaseDate': u'1987-07-21T07:00:00Z', u'currency': u'USD', u'artistId': 106621, u'previewUrl': u'https://audio-ssl.itunes.apple.com/apple-assets-us-std-000001/Music6/v4/f2/7d/73/f27d7346-de92-bdc6-e148-56a3da406005/mzaf_2747902348777129728.plus.aac.p.m4a', u'trackPrice': 1.29, u'isStreamable': True, u'trackViewUrl': u'https://itunes.apple.com/us/album/sweet-child-o-mine/5669937?i=5669911&uo=4', u'collectionName': u'Greatest Hits', u'collectionId': 5669937, u'trackId': 5669911, u'collectionViewUrl': u'https://itunes.apple.com/us/album/sweet-child-o-mine/5669937?i=5669911&uo=4', u'trackCount': 14, u'trackNumber': 2, u'discNumber': 1, u'collectionPrice': 9.99, u'trackCensoredName': u"Sweet Child O' Mine", u'trackName': u"Sweet Child O' Mine", u'trackTimeMillis': 355267, u'primaryGenreName': u'Rock', u'artistViewUrl': u'https://itunes.apple.com/us/artist/guns-n-roses/106621?uo=4', u'kind': u'song', u'country': u'USA', u'wrapperType': u'track', u'artworkUrl100': u'https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/100x100bb.jpg', u'collectionCensoredName': u'Greatest Hits', u'artistName': u"Guns N' Roses", u'artworkUrl60': u'https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/60x60bb.jpg', u'trackExplicitness': u'notExplicit', u'artworkUrl30': u'https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/30x30bb.jpg', u'discCount': 1}]}

View the request headers:

1
2
>>> response.headers
{'Content-Length': '650', 'x-apple-translated-wo-url': '/WebObjects/MZStoreServices.woa/ws/wsSearch?term=guns+and+roses&limit=1&urlDesc=', 'Access-Control-Allow-Origin': '*', 'x-webobjects-loadaverage': '0', 'X-Cache': 'TCP_MISS from a2-21-98-60.deploy.akamaitechnologies.com (AkamaiGHost/9.3.0.3-22245996) (-)', 'x-content-type-options': 'nosniff', 'x-apple-orig-url': 'https://itunes.apple.com/search?term=guns+and+roses&limit=1', 'x-apple-jingle-correlation-key': 'GUOFR25MGUUK5J7LUKI6UUFUWM', 'x-apple-application-site': 'ST11', 'Date': 'Tue, 08 May 2018 20:50:39 GMT', 'apple-tk': 'false', 'content-disposition': 'attachment; filename=1.txt', 'Connection': 'keep-alive', 'apple-seq': '0', 'x-apple-application-instance': '2001318', 'X-Apple-Partner': 'origin.0', 'Content-Encoding': 'gzip', 'strict-transport-security': 'max-age=31536000', 'Vary': 'Accept-Encoding', 'apple-timing-app': '109 ms', 'X-True-Cache-Key': '/L/itunes.apple.com/search ci2=limit=1&term=guns+and+roses__', 'X-Cache-Remote': 'TCP_MISS from a23-57-75-64.deploy.akamaitechnologies.com (AkamaiGHost/9.3.0.3-22245996) (-)', 'Cache-Control': 'max-age=86400', 'x-apple-request-uuid': '351c58eb-ac35-28ae-a7eb-a291ea50b4b3', 'Content-Type': 'text/javascript; charset=utf-8', 'apple-originating-system': 'MZStoreServices'}

Python Requests and the iTunes API:

Search for the Artist Guns and Roses and limit the output to 1 Song:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
>>> import requests
>>> import json
>>> a = 'https://itunes.apple.com/search?term=guns+and+roses&limit=1'
>>> b = requests.get(a).json()
>>> print(json.dumps(b, indent=2))
{
  "resultCount": 1,
  "results": [
    {
      "collectionExplicitness": "notExplicit",
      "releaseDate": "1987-07-21T07:00:00Z",
      "currency": "USD",
      "artistId": 106621,
      "previewUrl": "https://audio-ssl.itunes.apple.com/apple-assets-us-std-000001/Music6/v4/f2/7d/73/f27d7346-de92-bdc6-e148-56a3da406005/mzaf_2747902348777129728.plus.aac.p.m4a",
      "trackPrice": 1.29,
      "isStreamable": true,
      "trackViewUrl": "https://itunes.apple.com/us/album/sweet-child-o-mine/5669937?i=5669911&uo=4",
      "collectionName": "Greatest Hits",
      "collectionId": 5669937,
      "trackId": 5669911,
      "collectionViewUrl": "https://itunes.apple.com/us/album/sweet-child-o-mine/5669937?i=5669911&uo=4",
      "trackCount": 14,
      "trackNumber": 2,
      "discNumber": 1,
      "collectionPrice": 9.99,
      "trackCensoredName": "Sweet Child O' Mine",
      "trackName": "Sweet Child O' Mine",
      "trackTimeMillis": 355267,
      "primaryGenreName": "Rock",
      "artistViewUrl": "https://itunes.apple.com/us/artist/guns-n-roses/106621?uo=4",
      "kind": "song",
      "country": "USA",
      "wrapperType": "track",
      "artworkUrl100": "https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/100x100bb.jpg",
      "collectionCensoredName": "Greatest Hits",
      "artistName": "Guns N' Roses",
      "artworkUrl60": "https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/60x60bb.jpg",
      "trackExplicitness": "notExplicit",
      "artworkUrl30": "https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/30x30bb.jpg",
      "discCount": 1
    }
  ]
}

From the response we got a "artistId": 106621, lets query the API on the ArtistId, to get info of the Artist:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
>>> a = 'https://itunes.apple.com/lookup?id=106621'
>>> b = requests.get(a).json()
>>> print(json.dumps(b, indent=2))
{
  "resultCount": 1,
  "results": [
    {
      "artistType": "Artist",
      "amgArtistId": 4416,
      "wrapperType": "artist",
      "artistId": 106621,
      "artistLinkUrl": "https://itunes.apple.com/us/artist/guns-n-roses/106621?uo=4",
      "artistName": "Guns N' Roses",
      "primaryGenreId": 21,
      "primaryGenreName": "Rock"
    }
  ]
}

Query all the Albums by Artist by using the ArtistId and Entity for Album:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
>>> a = 'https://itunes.apple.com/lookup?id=106621&entity=album'
>>> b = requests.get(a).json()
>>> print(json.dumps(b, indent=2))
{
  "resultCount": 13,
  "results": [
    {
      "artistType": "Artist",
      "amgArtistId": 4416,
      "wrapperType": "artist",
      "artistId": 106621,
      "artistLinkUrl": "https://itunes.apple.com/us/artist/guns-n-roses/106621?uo=4",
      "artistName": "Guns N' Roses",
      "primaryGenreId": 21,
      "primaryGenreName": "Rock"
    },
    {
      "artistViewUrl": "https://itunes.apple.com/us/artist/guns-n-roses/106621?uo=4",
      "releaseDate": "2004-01-01T08:00:00Z",
      "collectionType": "Compilation",
      "collectionName": "Greatest Hits",
      "amgArtistId": 4416,
      "copyright": "\u2117 2004 Geffen Records",
      "collectionId": 5669937,
      "artworkUrl60": "https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/60x60bb.jpg",
      "wrapperType": "collection",
      "collectionViewUrl": "https://itunes.apple.com/us/album/greatest-hits/5669937?uo=4",
      "artistId": 106621,
      "collectionCensoredName": "Greatest Hits",
      "artworkUrl100": "https://is3-ssl.mzstatic.com/image/thumb/Music/v4/3c/18/87/3c188735-e462-3c99-92eb-50fb06afa73f/source/100x100bb.jpg",
      "trackCount": 14,
      "currency": "USD",
      "artistName": "Guns N' Roses",
      "country": "USA",
      "primaryGenreName": "Rock",
      "collectionExplicitness": "notExplicit",
      "collectionPrice": 9.99
    },

Get the Top 5 Albums by the Artist:

1
a = 'https://itunes.apple.com/lookup?id=106621&entity=album&limit=5'

How to get AMG ID (all music id):

1
2
3
4
5
6
7
8
9
10
11
>>> a = 'https://itunes.apple.com/search?term=jack+johnson&limit=2'
>>> b = requests.get(a).json()
>>> print(json.dumps(b, indent=2))
{
  "resultCount": 2,
  "results": [
    {
      "collectionExplicitness": "notExplicit",
      "releaseDate": "2005-03-01T08:00:00Z",
      "currency": "USD",
      "artistId": 909253,

Query Multiple Artists by using the amgId’s:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
>>> a = 'https://itunes.apple.com/lookup?amgArtistId=468749,5723'
>>> b = requests.get(a).json()
>>> print(json.dumps(b, indent=2))
{
  "resultCount": 2,
  "results": [
    {
      "artistType": "Artist",
      "amgArtistId": 468749,
      "wrapperType": "artist",
      "artistId": 909253,
      "artistLinkUrl": "https://itunes.apple.com/us/artist/jack-johnson/909253?uo=4",
      "artistName": "Jack Johnson",
      "primaryGenreId": 21,
      "primaryGenreName": "Rock"
    },
    {
      "artistType": "Artist",
      "amgArtistId": 5723,
      "wrapperType": "artist",
      "artistId": 78500,
      "artistLinkUrl": "https://itunes.apple.com/us/artist/u2/78500?uo=4",
      "artistName": "U2",
      "primaryGenreId": 21,
      "primaryGenreName": "Rock"
    }
  ]
}

If we Query the ArtistId from the previous response we will get the same artist:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
>>> a = 'https://itunes.apple.com/lookup?id=909253'
>>> b = requests.get(a).json()
>>> print(json.dumps(b, indent=2))
{
  "resultCount": 1,
  "results": [
    {
      "artistType": "Artist",
      "amgArtistId": 468749,
      "wrapperType": "artist",
      "artistId": 909253,
      "artistLinkUrl": "https://itunes.apple.com/us/artist/jack-johnson/909253?uo=4",
      "artistName": "Jack Johnson",
      "primaryGenreId": 21,
      "primaryGenreName": "Rock"
    }
  ]
}

Only get the Artist Name:

1
2
3
4
5
>> b
{u'resultCount': 1, u'results': [{u'artistType': u'Artist', u'amgArtistId': 468749, u'wrapperType': u'artist', u'artistId': 909253, u'artistLinkUrl': u'https://itunes.apple.com/us/artist/jack-johnson/909253?uo=4', u'artistName': u'Jack Johnson', u'primaryGenreId': 21, u'primaryGenreName': u'Rock'}]}

>>> b['results'][0]['artistName']
u'Jack Johnson'

Printing out the Artist Name and Genre with String Formatting:

1
2
>>> print('Artist: {artist_name}, Genre: {genre_name}'.format(artist_name=b['results'][0]['artistName'], genre_name=b['results'][0]['primaryGenreName']))
Artist: Jack Johnson, Genre: Rock

Setup the Elasticsearch Log Driver on Docker Swarm

Today we will look at a Elasticsearch logging driver for Docker.

Why a Log Driver?

By default the log output can be retrieved when using the docker service logs -f service_name, where log output of that service is shown via stdout. When having a lot of services in your swarm, it becomes useful logging all of your log output to a database service.

This is not just for Swarm but Docker stand alone as well.

In this tutorial we will use the Elasticsearch Log Driver, to log our logs for all our docker swarm services to Elasticsearch.

Installing to Elasticsearch Log Driver:

If you are running Docker Swarm, run this on all the nodes:

1
$ docker plugin install rchicoli/docker-log-elasticsearch:latest --alias elasticsearch_latest

Verify that the log driver has been installed:

1
2
3
$ docker plugin ls
ID                  NAME                          DESCRIPTION                          ENABLED
eadf06ad3d2a        elasticsearch_latest:latest   Send log messages to elasticsearch   true

Test the Log Driver:

Run a container of Alpine and echo a string of text:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ docker run --rm -ti \
    --log-driver elasticsearch_latest \
    --log-opt elasticsearch-url=http://192.168.0.235:9200 \
    --log-opt elasticsearch-insecure=false \
    --log-opt elasticsearch-sniff=false \
    --log-opt elasticsearch-index=docker-%F \
    --log-opt elasticsearch-type=log \
    --log-opt elasticsearch-timeout=10 \
    --log-opt elasticsearch-version=5 \
    --log-opt elasticsearch-fields=containerID,containerName,containerImageID,containerImageName,containerCreated \
    --log-opt elasticsearch-bulk-workers=1 \
    --log-opt elasticsearch-bulk-actions=1000 \
    --log-opt elasticsearch-bulk-size=1024 \
    --log-opt elasticsearch-bulk-flush-interval=1s \
    --log-opt elasticsearch-bulk-stats=false \
        alpine echo -n "this is a test logging message"

Have a look at your Elasticsearch indexes, and you will find the index which was specified in the log-options:

1
2
3
$ curl http://192.168.0.235:9200/_cat/indices?v
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   docker-2018.05.01 8FTqWq6nQlSGpYjD9M5qSg   5   1          1            0      8.9kb          8.9kb

Lets have a look at the Elasticsearch Document which holds the data of the log entry:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ curl http://192.168.0.235:9200/docker-2018.05.01/_search?pretty
{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "docker-2018.05.01",
        "_type" : "log",
        "_id" : "hMTUG2MBIFc8kAgSNkYo",
        "_score" : 1.0,
        "_source" : {
          "containerID" : "cee0dc758528",
          "containerName" : "jolly_goodall",
          "containerImageID" : "sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353",
          "containerImageName" : "alpine",
          "containerCreated" : "2018-05-01T13:11:20.819447101Z",
          "message" : "this is a test logging message",
          "source" : "stdout",
          "timestamp" : "2018-05-01T13:11:21.119861767Z",
          "partial" : true
        }
      }
    ]
  }
}

Using Swarm and Docker Compose:

We will deploy a stack with a whoami golang web app, which will use the elasticsearch log driver:

docker-compose.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
version: '3.4'

services:
  whoami:
    image: rbekker87/golang-whoami:latest
    networks:
      - appnet
    deploy:
      labels:
        - "traefik.port=80"
        - "traefik.backend.loadbalancer.swarm=true"
        - "traefik.docker.network=appnet"
        - "traefik.frontend.rule=Host:whoami.homecloud.mydomain.com"
      mode: replicated
      replicas: 10
      restart_policy:
        condition: any
      update_config:
        parallelism: 1
        delay: 70s
        order: start-first
        failure_action: rollback
      placement:
        constraints:
          - 'node.role==worker'
      resources:
        limits:
          cpus: '0.01'
          memory: 128M
        reservations:
          cpus: '0.001'
          memory: 64M
    logging:
      driver: elasticsearch_latest
      options:
        elasticsearch-url: "http://192.168.0.235:9200"
        elasticsearch-sniff: "false"
        elasticsearch-index: "docker-whoami-%F"
        elasticsearch-type: "log"
        elasticsearch-timeout: "10"
        elasticsearch-version: "6"
        elasticsearch-fields: "containerID,containerName,containerImageID,containerImageName,containerCreated"
        elasticsearch-bulk-workers: "1"
        elasticsearch-bulk-actions: "1000"
        elasticsearch-bulk-size: "1024"
        elasticsearch-bulk-flush-interval: "1s"
        elasticsearch-bulk-stats: "false"
networks:
  appnet:
    external: true

Deploy the Stack:

1
$ docker stack deploy -c docker-compose.yml web

Give it some time to launch and have a look at your indexes, and you will find the index which it wrote to:

1
2
3
4
$ curl http://192.168.0.235:9200/_cat/indices?v
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   docker-2018.05.01         8FTqWq6nQlSGpYjD9M5qSg   5   1          1            0      8.9kb          8.9kb
yellow open   docker-whoami-2018.05.01  YebUtKa1RnCy86iP5_ylgg   5   1         11            0     54.4kb         54.4kb

Having a look at the data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ curl 'http://192.168.0.235:9200/docker-whoami-2018.05.01/_search?pretty&size=1'
{
  "took" : 18,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 11,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "docker-whoami-2018.05.01",
        "_type" : "log",
        "_id" : "acbgG2MBIFc8kAgShQa7",
        "_score" : 1.0,
        "_source" : {
          "containerID" : "97c3b337735f",
          "containerName" : "web_whoami.6.t2prjiexkym14isbx3yfxa99w",
          "containerImageID" : "sha256:0f7762d2ce569fc2ccf95fbc4c7191dde727551a180253fac046daecc580c7e9",
          "containerImageName" : "rbekker87/golang-whoami:latest@sha256:5a55c5de9cc16fbdda376791c90efb7c704c81b8dba949dce21199945c14cc88",
          "containerCreated" : "2018-05-01T13:24:43.089365528Z",
          "message" : "Starting up on port 80",
          "source" : "stdout",
          "timestamp" : "2018-05-01T13:24:48.636773709Z",
          "partial" : false
        }
      }
    ]
  }
}

For more info about this, have a look at the referenced documentation below.

Resources:

Forwarding the Docker Socket via a SSH Tunnel to Execute Docker Commands Locally

With automation in mind, when you want to execute docker commands remotely, you want to do it in a secure manner, as you don’t want to expose your Docker port to the whole world.

One way in doing that, is forwarding the remote docker socket via a local port over a SSH Tunnel. With this way, you can execute docker commands locally on your workstation, as if the swarm is running on your workstation/laptop/node/bastion host etc.

Without the tunnel, I have a swarm on my laptop with no running services:

1
2
$ docker service ls
ID                  NAME                   MODE                REPLICAS            IMAGE                                                               PORTS

As you can see, we have no services running, but the remote swarm has a couple, so after forwarding the connection, we should see our remote services.

Setting up the SSH Tunnel:

Here we will forward the remote docker socket: /var/run/docker.sock to a local port bound to localhost: localhost:2377:

1
2
$ screen -S docker
$ ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -i ~/path/to/key.pem -NL localhost:2377:/var/run/docker.sock root@docker-managers.mydomain.com

Now the SSH Tunnel will be established, and you can detach your screen session, or open a new shell session. To detach your screen session: 'ctrl + a' then d

Verifying that the tunnel is established:

You can use netstat to verify that the port is listening:

1
2
$ netstat -ant | grep 2377
tcp4       0      0  127.0.0.1.2377         *.*                    LISTEN

Inform the Docker Client to use the Port:

Now we need to inform the docker client, to use the new port to talk to the docker daemon. We do that by setting the DOCKER_HOST environment variable to point to localhost:2377:

1
$ export DOCKER_HOST="localhost:2377"

This will remain for the lifetime of the shell session.

Testing it Out:

Now we can run our commands locally, and we should see the output of our remote swarm:

1
2
3
4
5
$ docker service ls
ID                  NAME                   MODE                REPLICAS            IMAGE                                                               PORTS
xjta8e3ek2u2        apps_flask_reminders   replicated          3/3                 rbekker87/flask-reminders:debian
0l7ruktbqj99        apps_kibana            replicated          1/1                 kibana:latest
...

Terminating our SSH Tunnel:

To terminate our SSH Tunnel, reconnect to your shell session, and hit ctrl + c:

1
2
3
4
$ screen -ls
There is a screen on:
  50413.docker    (Detached)
$ screen -r 50413

Hit ctrl + c :

1
CKilled by signal 2.

And exit the screen session:

1
$ exit

With this way, you can do lots of automation with docker swarm, not limited to swarm, but one of them.

Encryption and Decryption With the PyCrypto Module Using the AES Cipher in Python

While I’m learning a lot about encryption at the moment, I wanted to test out encryption with the PyCrypto module in Python using the Advanced Encryption Standard (AES) Symmetric Block Cipher.

Installing PyCrypto:

1
$ pip install pycrypto --user

PyCrypto Example:

Our AES Key needs to be either 16, 24 or 32 bytes long and our Initialization Vector needs to be 16 Bytes long. That will be generated using the random and string modules.

Encrypting:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
>>> from Crypto.Cipher import AES
>>> import random, string, base64

>>> key = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase + string.digits) for x in range(32))
>>> iv = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase + string.digits) for x in range(16))

>>> print(key, len(key))
('BLhgpCL81fdLBk23HkZp8BgbT913cqt0', 32)
>>> print(iv, len(iv))
('OWFJATh1Zowac2xr', 16)

>>> enc_s = AES.new(key, AES.MODE_CFB, iv)
>>> cipher_text = enc_s.encrypt('this is a super important message')
>>> encoded_cipher_text = base64.b64encode(cipher_text)
>>> print(encoded_cipher_text)
'AtBa6zVB0UQ3U/50ogOb6g09FlyPdpmJB7UzoCqxhsQ6'

Decrypting:

1
2
3
4
5
6
7
8
9
>>> from Crypto.Cipher import AES
>>> import base64
>>> key = 'BLhgpCL81fdLBk23HkZp8BgbT913cqt0'
>>> iv = 'OWFJATh1Zowac2xr'

>>> decryption_suite = AES.new(key, AES.MODE_CFB, iv)
>>> plain_text = decryption_suite.decrypt(base64.b64decode(encoded_cipher_text))
>>> print(plain_text)
this is a super important message

It’s not needed to use base64, but to have the ability to stay away from strange characters I decided to encode them with base64 :D

References:

Running a 3 Node Elasticsearch Cluster With Docker Compose on Your Laptop for Testing

Having a Elasticsearch cluster on your laptop with Docker for testing is great. And in this post I will show you how quick and easy it is, to have a 3 node elasticsearch cluster running on docker for testing.

Pre-Requisites

We need to set the vm.max_map_count kernel parameter:

1
$ sudo sysctl -w vm.max_map_count=262144

To set this permanently, add it to /etc/sysctl.conf and reload with sudo sysctl -p

Docker Compose:

The docker compose file that we will reference:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/home/ruan/workspace/docker/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/home/ruan/workspace/docker/elasticsearch/data
    networks:
      - esnet
  elasticsearch3:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
    container_name: elasticsearch3
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata3:/home/ruan/workspace/docker/elasticsearch/data
    networks:
      - esnet


volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local
  esdata3:
    driver: local

networks:
  esnet:

Now make sure the paths exist that we referenced in the compose file, in my case /home/ruan/workspace/docker/elasticsearch/data

Deploy

Deploy your elasticsearch cluster with docker compose:

1
$ docker-compose up

This will run in the foreground, and you should see console output.

Testing Elasticsearch

Let’s run a couple of queries, first up, check the cluster health api:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ curl http://127.0.0.1:9200/_cluster/health?pretty
{
  "cluster_name" : "docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Create a index with replication count of 2:

1
$ curl -H "Content-Type: application/json" -XPUT http://127.0.0.1:9200/test -d '{"number_of_replicas": 2}'

Ingest a document to elasticsearch:

1
2
$ curl -H "Content-Type: application/json" -XPUT http://127.0.0.1:9200/test/docs/1 -d '{"name": "ruan"}'
{"_index":"test","_type":"docs","_id":"1","_version":1,"result":"created","_shards":{"total":3,"successful":3,"failed":0},"_seq_no":0,"_primary_term":1}

View the indices:

1
2
3
4
$ curl http://127.0.0.1:9200/_cat/indices?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   test                        w4p2Q3fTR4uMSYBfpNVPqw   5   2          1            0      3.3kb          1.1kb
green  open   .monitoring-es-6-2018.04.29 W69lql-rSbORVfHZrj4vug   1   1       1601           38        4mb            2mb

Deleting the Cluster:

As its running in the foreground, you can just hit ctrl + c and as we persisted data in our compose, when you spin up the cluster again, the data will still be there.

Resources:

Using the Bulk API With Elasticsearch

This tutorial will guide you how to use the Bulk API with Elasticsearch, this is great for when having a dataset that contains a lot of documents, where you want to insert them into elasticsearch in bulk uploads.

The Dataset

We will be using a dataset from elastic that contains 1000 documents that holds account data.

Getting the Dataset:

1
$ wget -O accounts.json https://github.com/elastic/elasticsearch/blob/master/docs/src/test/resources/accounts.json?raw=true

Preview the data:

1
2
3
4
5
6
7
8
9
10
11
$ head -10  accounts.json
{"index":{"_id":"1"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
{"index":{"_id":"6"}}
{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"}
{"index":{"_id":"13"}}
{"account_number":13,"balance":32838,"firstname":"Nanette","lastname":"Bates","age":28,"gender":"F","address":"789 Madison Street","employer":"Quility","email":"nanettebates@quility.com","city":"Nogal","state":"VA"}
{"index":{"_id":"18"}}
{"account_number":18,"balance":4180,"firstname":"Dale","lastname":"Adams","age":33,"gender":"M","address":"467 Hutchinson Court","employer":"Boink","email":"daleadams@boink.com","city":"Orick","state":"MD"}
{"index":{"_id":"20"}}
{"account_number":20,"balance":16418,"firstname":"Elinor","lastname":"Ratliff","age":36,"gender":"M","address":"282 Kings Place","employer":"Scentric","email":"elinorratliff@scentric.com","city":"Ribera","state":"WA"}

Using the Bulk API:

We will ingest the data to our bank_accounts index, and to the account type:

1
$ curl -s -H "Content-Type: application/json" -XPOST localhost:9200/accounts/docs/_bulk --data-binary "@accounts.json"

When it’s done, have a look at the indices:

1
2
3
$ curl http://127.0.0.1:9200/_cat/indices/bank_accounts?v
health status index         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   bank_accounts BK_OJYOFTD67tqsQBUWSuQ   5   1       1000            0    950.3kb        475.1kb

Doing a search and display one document:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ curl -XGET 'http://127.0.0.1:9200/bank_accounts/_search?pretty&size=1'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank_accounts",
        "_type" : "account",
        "_id" : "25",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 25,
          "balance" : 40540,
          "firstname" : "Virginia",
          "lastname" : "Ayala",
          "age" : 39,
          "gender" : "F",
          "address" : "171 Putnam Avenue",
          "employer" : "Filodyne",
          "email" : "virginiaayala@filodyne.com",
          "city" : "Nicholson",
          "state" : "PA"
        }
      }
    ]
  }
}

Demo Recording:

This has also been reccored, which can be viewed here:

Using Bulk with Auto Generated ID’s

As you might know when you do a POST request to the type, the _id field gets auto populated. Timo, one of my friends had the requirement to use the Bulk API to post auto generated Id’s and not the static id’s that is given in the example dataset.

I have answered this on Elastic’s discuss page: https://discuss.elastic.co/t/looking-for-working-example-data-set-to-bulk-index-into-es6/128678/3

I will provide the steps below as well:

convert.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/usr/bin/env python

src_file = 'src_file.json'
dest_file = 'dest_file.json'
metadata = '{"index": {"_index": "bank_accounts", "_type": "account"}}'

with open(src_file) as open_file:
    lines = open_file.readlines()

lines = [line.replace(' ', '') for line in lines]

with open(dest_file, 'w') as f:
    for each_line in lines:
        f.write(metadata + '\n')
        f.writelines(each_line)

The original file:

1
2
3
4
5
$ head -4 file.json
{"index":{"_id":"1"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
{"index":{"_id":"6"}}
{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"}

Removing the initial metadata:

1
2
$ cat file.json | grep account_number >> src_file.json
$ ./convert.py

Previewing the destination file:

1
2
3
4
5
$ head -4 dest_file.json
{"index": {"_index": "bank_accounts", "_type": "account"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880HolmesLane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
{"index": {"_index": "bank_accounts", "_type": "account"}}
{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671BristolStreet","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"}

Looking at my current indices:

1
2
3
$ curl http://localhost:9200/_cat/indices?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-es-6-2018.05.06 3OgdIbDWQWCR8WJlQTXr9Q   1   1     114715            6      104mb           50mb

Ingesting the data via Bulk API:

1
$ curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_bulk --data-binary @dest_file.json

Looking at my indices to verify that the index exist:

1
2
3
4
$ curl http://localhost:9200/_cat/indices?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   bank_accounts               u37MQvzhSPe97BJzp1u49Q   5   1       1000            0    296.4kb           690b
green  open   .monitoring-es-6-2018.05.06 3OgdIbDWQWCR8WJlQTXr9Q   1   1     114750            6    103.9mb         49.9mb

Looking at one document: :smiley:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ curl 'http://localhost:9200/bank_accounts/_search?pretty&size=1'
{
  "took" : 641,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank_accounts",
        "_type" : "account",
        "_id" : "cohJN2MBCa89A-FEmiJs",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 6,
          "balance" : 5686,
          "firstname" : "Hattie",
          "lastname" : "Bond",
          "age" : 36,
          "gender" : "M",
          "address" : "671BristolStreet",
          "employer" : "Netagy",
          "email" : "hattiebond@netagy.com",
          "city" : "Dante",
          "state" : "TN"
        }
      }
    ]
  }
}

Resources:

Encryption and Decryption With Simple Crypt Using Python

Today I wanted to encrypt sensitive information to not expose passwords, hostnames etc. I wanted to have a way to encrypt my strings with a master password and stumbled upon Simple Crypt.

Simple Crypt

Why simple-crypt? Referenced from their docs:

  • Simple Crypt uses standard, well-known algorithms following the recommendations from this link.
  • The PyCrypto library provides the algorithm implementation, where AES256 cipher is used.
  • It includes a check (an HMAC with SHA256) to warn when ciphertext data are modified.
  • It tries to make things as secure as possible when poor quality passwords are used (PBKDF2 with SHA256, a 256 bit random salt, and 100,000 rounds).
  • Using a library, rather than writing your own code, means that we have less solutions to the same problem.

Installing Simple-Crypt:

From a base alpine image:

1
2
3
4
$ apk update
$ apk add python python-dev py2-pip
$ apk add gcc g++ make libffi-dev openssl-dev
$ pip install simple-crypt

Simple Examples:

Two simple examples to encrypt and decrypt data with simple-crypt. We will use a password sekret and we will encrypt the string: this is a secure message:

1
2
3
4
5
6
7
>>> from simplecrypt import encrypt, decrypt
>>> password = 'sekret'
>>> message = 'this is a secret message'
>>> ciphertext = encrypt(password, message)
>>>
>>> print(ciphertext)
sc#$%^&*(..........

Now that we have our encrypted string, lets decrypt it. First we will use the wrong password, so that you will see how the expected output should look when using a different password, than was used when it was encrypted:

1
2
3
4
5
6
7
8
>>> print(decrypt('badpass', ciphertext))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/simplecrypt/__init__.py", line 72, in decrypt
    _assert_hmac(hmac_key, hmac, hmac2)
  File "/usr/lib/python2.7/site-packages/simplecrypt/__init__.py", line 116, in _assert_hmac
    raise DecryptionException('Bad password or corrupt / modified data.')
simplecrypt.DecryptionException: Bad password or corrupt / modified data.

Now using the correct password to decrypt:

1
2
>>> print(decrypt('sekret', ciphertext))
this is a secret message

SimpleCrypt Base64 and Getpass

I wanted to store the encrypted string in a database, but the ciphertext has a combination of random special characters, so I decided to encode the ciphertext with base64. And the password input will be used with the getpass module.

Our encryption app:

encrypt.py
1
2
3
4
5
6
7
8
9
10
11
import sys
from simplecrypt import encrypt, decrypt
from base64 import b64encode, b64decode
from getpass import getpass

password = getpass()
message = sys.argv[1]

cipher = encrypt(password, message)
encoded_cipher = b64encode(cipher)
print(encoded_cipher)

Our decryption app:

1
2
3
4
5
6
7
8
9
10
11
import sys
from simplecrypt import encrypt, decrypt
from base64 import b64encode, b64decode
from getpass import getpass

password = getpass()
encoded_cipher = sys.argv[1]

cipher = b64decode(encoded_cipher)
plaintext = decrypt(password, cipher)
print(plaintext)

Encrypt and Decrypting Data using our Scripts:

Encrypting the string this is a secret message:

1
2
3
$ python encrypt.py "this is a secret message"
Password:
c2MAAnyfWIfOBV43vxo3sVCEYMG4C6hx69hv2Ii1JKlVHJUgBAlADJPOsD5cJO6MMI9faTDm1As/VfesvBzIe5S16mNyg2q7xfnP5iJ0RlK92vMNRbKOvNibg3M=

Now that we have our encoded ciphertext, lets decrypt it with the password that we encrypted it with:

1
2
3
$ python decrypt.py 'c2MAAnyfWIfOBV43vxo3sVCEYMG4C6hx69hv2Ii1JKlVHJUgBAlADJPOsD5cJO6MMI9faTDm1As/VfesvBzIe5S16mNyg2q7xfnP5iJ0RlK92vMNRbKOvNibg3M='
Password:
this is a secret message

This is one way of working with sensitive info that you would like to encrypt/decrypt.