Ruan Bekker's Blog

From a Curious mind to Posts on Github

Using IAM Authentication With Amazon Elasticsearch Service

Today I will demonstrate how to allow access to Amazons Elasticsearch Service using IAM Authenticationi using AWS Signature Version4.

Elasticsearch Service Authentication Support:

When it comes to security, Amazons Elasticsearch Service supports three types of access policies:

  • Resource Based
  • Identity Based
  • IP Access Based

More information on this can be found below: - https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html

Securing your Amazon Elasticsearch Search Domain:

To secure your domain with IAM Based Authentication, the following steps will be neeed:

  • Create IAM Policy to be associated with a IAM User or Role
  • On Elasticsearch Access Policy, associate the ARN to the Resource
  • Use the AWS4Auth package to sign the requests as AWS supports Signature Version 4
1
2
3
4
5
6
7
8
9
10
11
12
13

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "es:*"
            ],
            "Resource": "arn:aws:es:eu-west-1:<ACCOUNT-ID>:domain/<ES-DOMAIN>"
        }
    ]
}

Create the IAM Role with EC2 Identity Provider as a Trusted Relationship eg. es-role and associate the IAM Policy es-policy to the role.

Create/Moodify the Elasticsearch Access Policy, in this example we will be using a combination of IAM Role, IAM User and IP Based access:

  • IAM Role for EC2 Role Based Services
  • IAM User for User/System Account
  • IP Based for cients that needs to be whitelisted via IP (ip-based just for demonstration, as the tests will be used only for IAM)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT-ID>:role/<IAM-ROLE-NAME>",
          "arn:aws:iam::<ACCOUNT-ID>:user/<IAM-USER-NAME>"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:<ACCOUNT-ID>:domain/<ES-DOMAIN>/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:<ACCOUNT-ID>:domain/<ES-DOMAIN>/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "x.x.x.x",
            "x.x.x.x"
          ]
        }
      }
    }
  ]
}

After the Access Policy has been updated, the Elasticsearch Domain Status will show Active

Testing from EC2 using IAM Instance Profile:

Launch a EC2 Instance with the IAM Role eg. es-role, then using Python, we will make a request to our Elasticsearch Domain using boto3, aws4auth and the native elasticsearch client for python via our IAM Role, which we will get the temporary credentials from boto3.Session.

Installing the dependencies:

1
2
3
4

$ pip install virtualenv
$ virtualenv .venv
$ source .venv/bin/activate
$ pip install boto3 elasticsearch requests_aws4auth

Our code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import boto3, json
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

my_region = 'eu-west-1'
my_service = 'es'
my_eshost = 'search-replaceme.eu-west-1.es.amazonaws.com'

session = boto3.Session(region_name=my_region) # thanks Leon
credentials = session.get_credentials()
credentials = credentials.get_frozen_credentials()
access_key = credentials.access_key
secret_key = credentials.secret_key
token = credentials.token

aws_auth = AWS4Auth(
    access_key,
    secret_key,
    my_region,
    my_service,
    session_token=token
)

es = Elasticsearch(
    hosts = [{'host': my_eshost, 'port': 443}],
    http_auth=aws_auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(json.dumps(es.info(), indent=2))

Running our piece of code, will result in this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ python get-info-from-role.py
{
  "cluster_name": "<ACCOUNT-ID>:<ES-DOMAIN>",
  "cluster_uuid": "sLUnqFSsQdCMlBLrn7BTUA",
  "version": {
    "lucene_version": "6.6.0",
    "build_hash": "Unknown",
    "build_snapshot": false,
    "number": "5.5.2",
    "build_date": "2017-10-18T04:35:01.381Z"
  },
  "name": "KXSwBvT",
  "tagline": "You Know, for Search"
}

Testing using IAM Credentials from Credentials Provider:

Configure your credentials provider:

1
2
3
4
5
6

$ pip install awscli
$ aws configure --profile ruan
AWS Access Key ID [None]: xxxxxxxxx
AWS Secret Access Key [None]: xxxxxx
Default region name [None]: eu-west-1
Default output format [None]: json

Using Python, we will get the credentials from the Credential Provider, using our profile name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

import boto3, json
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

my_service = 'es'
my_region = 'eu-west-1'
my_eshost = 'search-replaceme.eu-west-1.es.amazonaws.com'

session = boto3.Session(
    region_name=my_region,
    profile_name='ruan'
)

credentials = session.get_credentials()
access_key = credentials.access_key
secret_key = credentials.secret_key

aws_auth = AWS4Auth(
    access_key,
    secret_key,
    my_region,
    my_service
)

es = Elasticsearch(
    hosts = [{'host': my_eshost, 'port': 443}],
    http_auth=aws_auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(json.dumps(es.info(), indent=2))

Running it will result in:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ python get-info-from-user.py
{
  "cluster_name": "<ACCOUNT-ID>:<ES-DOMAIN>",
  "cluster_uuid": "sLUnqFSsQdCMlBLrn7BTUA",
  "version": {
    "lucene_version": "6.6.0",
    "build_hash": "Unknown",
    "build_snapshot": false,
    "number": "5.5.2",
    "build_date": "2017-10-18T04:37:21.381Z"
  },
  "name": "KXSwBvT",
  "tagline": "You Know, for Search"
}

For more blog posts on Elasticsearch have a look at: - blog.ruanbekker.com:elasticsearch - sysadmins.co.za:elasticsearch

Get Blogpost Titles Links and Tags From a RSS Link Using Python Feedparser

I wanted to get metadata from my other blog sysadmins.co.za, such as each post’s title, link and tags using the RSS link. I stumbled upon feedparser, where I will use it to scrape all the posts details from the link and append it to a list, which I can then use to ingest it into a database or something like that.

Installing Dependencies:

Install feedparser and requests:

1

$ pip install feedparser requests

The Python Code:

I’m not too sure at this point how to get pagination going, so I’ve set a range to check, and if a status code of 200 is received, it will check if the title is in the list that I defined, if not, it will append it to the list.

At the end of the loop, the script will return the list that was defined, which will provide the info mentioned earlier:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

import feedparser
import time
import requests

rss_url = "https://sysadmins.co.za/rss"

posts = []

def get_posts_for_ghost(rss_url):
    response = feedparser.parse(rss_url)
    for each in response['entries']:
        if each['title'] in [x['title'] for x in posts]:
            pass
        else:
            posts.append({
                "title": each['title'],
                "link": each['links'][0]['href'],
                "tags": [x['term'] for x in each['tags']],
                "date": time.strftime('%Y-%m-%d', each['published_parsed'])
            })
    return posts

count = 12

for x in range(count):
    if requests.get("{0}/{1}/".format(rss_url, count)).status_code == 200:
        print("get succeeded, count at: {}".format(count))
        get_posts_for_ghost("{0}/{1}/".format(rss_url, count))
        count -= 1
    else:
        print("got 404, count at: {}".format(count))
        count -= 1

    get_posts_for_ghost(rss_url)

print(posts)

Running the script:

Running the script will look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

$ python rssfeed.py
got 404, count at: 12
got 404, count at: 11
got 404, count at: 10
get succeeded, count at: 9
get succeeded, count at: 8
get succeeded, count at: 7
get succeeded, count at: 6
get succeeded, count at: 5
get succeeded, count at: 4
get succeeded, count at: 3
get succeeded, count at: 2
get succeeded, count at: 1
[
  {
    'title': 'Tutorial on DynamoDB using Bash and the AWS CLI Tools to Interact with a Music Dataset',
    'link': 'https://sysadmins.co.za/tutorial-on-dynamodb-using-bash-and-the-aws-cli-tools-to-interact-with-a-music-dataset/',
    'tags': ['DynamoDB', 'Bash', 'AWS'],
    'date': '2018-08-15'
  },
  {
    'title': 'Setup a PPTP VPN on Ubuntu 16',
    'link': 'https://sysadmins.co.za/setup-a-pptp-vpn-on-ubuntu-16/',
    'tags': ['Networking', 'VPN'],
    'date': '2018-06-27'
  },
  ...

Tutorial on DynamoDB Using Bash and the AWS CLI Tools to Interact With a Music Dataset

In this tutorial we will be using Amazons DynamoDB (DynamoDB Local) to host a sample dataset consisting of music data that I retrieved from the iTunes API, which we will be using the aws cli tools to interact with the data.

We will be doing the following:

  • Use Docker to provision a Local DynamoDB Server
  • Create a DynamoDB Table with a Hash and Range Key
  • List the Table
  • Create a Item in DynamoDB
  • Read a Item from DynamoDB
  • Read a Item from DynamoDB by specifying the details you would like to read
  • Batch Write multiple items to DynamoDB
  • Scan all your Items from DynamoDB
  • Query by Artist
  • Query by Artist and Song
  • Query all the Songs from an Artist starting with a specific letter
  • Indexes
  • Delete the Table

If you are just getting started with DynamoDB, I recommend having a look at Amazons DynamoDB Documentation Page first.

The Music Dataset:

I used the iTunes API to get the music metadata, but I also have a post on how to query the iTunes API to get data from them to use.

a Quick way in Python to get the top 10 songs from Guns and Roses, will look like this:

1
2
3

>>> a = 'https://itunes.apple.com/search?term=guns+and+roses&limit=10'
>>> b = requests.get(a).json()
>>> print(json.dumps(b, indent=2))

Create the DynamoDB Local Server on Docker:

If you have a AWS Account you can provision your table from there, but if you want to test it locally, you can provision a local DynamoDB Server using Docker:

1

$ docker run -it -p 8000:8000 --name dynamodb-local rbekker87/dynamodb-local

Install the AWS CLI Tools:

1
2
3

$ pip install awscli
$ aws configure
# you can enter random data if you are using dynamodb-local

Create the DynamoDB Table:

Create a DynamoDB Table named MusicCollection with a Artist (HASH) and SongTitle (RANGE) key attributes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

$ aws dynamodb create-table --table-name MusicCollection \
  --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S \
  --key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \
  --endpoint-url http://localhost:8000

Response:
{
    "TableDescription": {
        "TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/MusicCollection",
        "AttributeDefinitions": [
            {
                "AttributeName": "Artist",
                "AttributeType": "S"
            },
            {
                "AttributeName": "SongTitle",
                "AttributeType": "S"
            }
        ],
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "WriteCapacityUnits": 5,
            "LastIncreaseDateTime": 0.0,
            "ReadCapacityUnits": 5,
            "LastDecreaseDateTime": 0.0
        },
        "TableSizeBytes": 0,
        "TableName": "MusicCollection",
        "TableStatus": "ACTIVE",
        "KeySchema": [
            {
                "KeyType": "HASH",
                "AttributeName": "Artist"
            },
            {
                "KeyType": "RANGE",
                "AttributeName": "SongTitle"
            }
        ],
        "ItemCount": 0,
        "CreationDateTime": 1525339294.186
    }
}

Listing the Tables:

List the DynamoDB Table that you created:

1
2
3
4
5
6
7

$ aws dynamodb list-tables --endpoint-url http://localhost:8000

{
    "TableNames": [
        "MusicCollection"
    ]
}

Create a Item in DynamoDB:

Add a song from the band Bring me the Horizon called Sleepwalking from the album Sempiternal to the table by using the PutItem call:

1
2

$ aws dynamodb --endpoint-url http://localhost:8000 put-item --table-name MusicCollection \
  --item '{"Artist": {"S": "Bring me the Horizon"}, "SongTitle": {"S": "Sleepwalking"}, "AlbumTitle": {"S": "Sempiternal"}}'

Read a Item from DynamoDB

Get the Song Details from the Table by using the GetItem call:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ aws dynamodb --endpoint-url http://localhost:8000 get-item --table-name MusicCollection \
  --key  '{"Artist": {"S": "Bring me the Horizon"}, "SongTitle": {"S": "Sleepwalking"}}'

{
    "Item": {
        "Artist": {
            "S": "Bring me the Horizon"
        },
        "SongTitle": {
            "S": "Sleepwalking"
        },
        "AlbumTitle": {
            "S": "Sempiternal"
        }
    }
}

To only get specific attributes we can use --aatributes-to-get:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ aws dynamodb --endpoint-url http://localhost:8000 get-item --table-name MusicCollection \
  --attributes-to-get '["AlbumTitle", "SongTitle"]' \
  --key  '{"Artist": {"S": "Bring me the Horizon"}, "SongTitle": {"S": "Sleepwalking"}}'

{
    "Item": {
        "SongTitle": {
            "S": "Sleepwalking"
        },
        "AlbumTitle": {
            "S": "Sempiternal"
        }
    }
}

However, AWS Recommends to use the --projection-expression parameter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ aws dynamodb --endpoint-url http://localhost:8000 get-item --table-name MusicCollection \
  --projection-expression "AlbumTitle, SongTitle" \
  --key  '{"Artist": {"S": "Bring me the Horizon"}, "SongTitle": {"S": "Sleepwalking"}}'

{
    "Item": {
        "SongTitle": {
            "S": "Sleepwalking"
        },
        "AlbumTitle": {
            "S": "Sempiternal"
        }
    }
}

Batch Write

Now lets use the iTunes API to get a collection of some songs, which I will dump into a json file on github. So now that we have a json file with a collection of songs from multiple artists, we can go ahead and write it into our table using the BatchWriteItem call:

1
2

$ wget https://raw.githubusercontent.com/ruanbekker/dynamodb-local-docker/master/demo/batch-write-songs.json
$ aws dynamodb batch-write-item --request-items file://music-table/batch-write-songs.json --endpoint-url http://localhost:8000

Scan the Table:

This can be a very expensive call, as a Scan will return all the items from your table, and depending on the size of your table, you could be throttled, but since we are using dynamodb local and only having 16 items in our table, we can do a scan to return all the items in our table:

1
2
3

$ aws dynamodb --endpoint-url http://localhost:8000 scan --table-name MusicCollection
{
    "Count": 16,

Query

Let’s start using the Query call to get all the songs from the Artist: AC/DC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

$ aws dynamodb --endpoint-url http://localhost:8000 query --select ALL_ATTRIBUTES \
  --table-name MusicCollection \
  --key-condition-expression "Artist = :a" \
  --expression-attribute-values  '{":a":{"S":"AC/DC"}}'

{
    "Count": 3,
    "Items": [
        {
            "Artist": {
                "S": "AC/DC"
            },
            "SongTitle": {
                "S": "Back In Black"
            },
            "AlbumTitle": {
                "S": "Back In Black"
            }
        },
        {
            "Artist": {
                "S": "AC/DC"
            },
            "SongTitle": {
                "S": "Thunderstruck"
            },
            "AlbumTitle": {
                "S": "The Razors Edge"
            }
        },
        {
            "Artist": {
                "S": "AC/DC"
            },
            "SongTitle": {
                "S": "You Shook Me All Night Long"
            },
            "AlbumTitle": {
                "S": "Back in Black"
            }
        }
    ],
    "ScannedCount": 3,
    "ConsumedCapacity": null
}

Query to get the details of a specific Song from a specific Artist:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

$ aws dynamodb --endpoint-url http://localhost:8000 query --select ALL_ATTRIBUTES \
  --table-name MusicCollection \
  --key-condition-expression "Artist = :a and SongTitle = :t" \
  --expression-attribute-values  '{ ":a": {"S": "AC/DC"}, ":t": {"S": "You Shook Me All Night Long"}}'

{
    "Count": 1,
    "Items": [
        {
            "Artist": {
                "S": "AC/DC"
            },
            "SongTitle": {
                "S": "You Shook Me All Night Long"
            },
            "AlbumTitle": {
                "S": "Back in Black"
            }
        }
    ],
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

Query to get all the songs from the Beatles that starts with the letter ‘H’:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

$ aws dynamodb --endpoint-url http://localhost:8000 query --select ALL_ATTRIBUTES \
  --table-name MusicCollection \
  --key-condition-expression "Artist = :a and begins_with(SongTitle, :t)" \
  --expression-attribute-values  '{":a":{"S":"The Beatles"}, ":t": {"S": "h"}}'

{
    "Count": 2,
    "Items": [
        {
            "Artist": {
                "S": "The Beatles"
            },
            "SongTitle": {
                "S": "Happy Day"
            },
            "AlbumTitle": {
                "S": "The Beatles 1967-1970 (The Blue Album)"
            }
        },
        {
            "Artist": {
                "S": "The Beatles"
            },
            "SongTitle": {
                "S": "Help!"
            },
            "AlbumTitle": {
                "S": "The Beatles Box Set"
            }
        }
    ],
    "ScannedCount": 2,
    "ConsumedCapacity": null
}

So our table consists of Artist (HASH) and SongTitle (RANGE), so we can only query based on those attributes. You will find when you try to query on a attribute that is not part of the KeySchema, a exception will be received:

1
2
3

$ aws dynamodb --endpoint-url http://localhost:8000 query --select ALL_ATTRIBUTES --table-name MusicCollection --key-condition-expression "Artist = :a and AlbumTitle = :t" --expression-attribute-values  '{":a":{"S":"AC/DC"}, ":t": {"S": "Back in Black"}}'

An error occurred (ValidationException) when calling the Query operation: Query condition missed key schema element

So how do we query on a attribute that is not part of the KeySchema? Let’s say you want to query all the songs from a Artist and a specific Album.

Global Secondary Indexes:

Add Global Secondary Index, with the Attributes: Artist and AlbumTitle.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

$ aws dynamodb --endpoint-url http://localhost:8000 update-table --table-name MusicCollection \
  --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S AttributeName=AlbumTitle,AttributeType=S \
  --global-secondary-index-updates "Create={"IndexName"="album-index", "KeySchema"=[ {"AttributeName"="Artist", "KeyType"="HASH"}, {"AttributeName"="AlbumTitle", "KeyType"="RANGE" }], "Projection"={"ProjectionType"="INCLUDE", "NonKeyAttributes"="AlbumTitle"}, "ProvisionedThroughput"= {"ReadCapacityUnits"=1, "WriteCapacityUnits"=1} }"

{
    "TableDescription": {
        "TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/MusicCollection",
        "AttributeDefinitions": [
            {
                "AttributeName": "Artist",
                "AttributeType": "S"
            },
            {
                "AttributeName": "SongTitle",
                "AttributeType": "S"
            },
            {
                "AttributeName": "AlbumTitle",
                "AttributeType": "S"
            }
        ],
        "GlobalSecondaryIndexes": [
            {
                "IndexName": "album-index",
                "Projection": {
                    "ProjectionType": "INCLUDE",
                    "NonKeyAttributes": [
                        "AlbumTitle"
                    ]
                },
                "ProvisionedThroughput": {
                    "WriteCapacityUnits": 1,
                    "ReadCapacityUnits": 1
                },
                "IndexStatus": "CREATING",
                "Backfilling": false,
                "KeySchema": [
                    {
                        "KeyType": "HASH",
                        "AttributeName": "Artist"
                    },
                    {
                        "KeyType": "RANGE",
                        "AttributeName": "AlbumTitle"
                    }
                ],
                "IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/MusicCollection/index/album-index"
            }
        ],
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "WriteCapacityUnits": 5,
            "LastIncreaseDateTime": 0.0,
            "ReadCapacityUnits": 5,
            "LastDecreaseDateTime": 0.0
        },
        "TableSizeBytes": 984,
        "TableName": "MusicCollection",
        "TableStatus": "ACTIVE",
        "KeySchema": [
            {
                "KeyType": "HASH",
                "AttributeName": "Artist"
            },
            {
                "KeyType": "RANGE",
                "AttributeName": "SongTitle"
            }
        ],
        "ItemCount": 15,
        "CreationDateTime": 1525339294.186
    }
}

Now when we use the same query, but we specify our index, we will get the data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

$ aws dynamodb --endpoint-url http://localhost:8000 query \
  --select ALL_ATTRIBUTES \
  --table-name MusicCollection \
  --index-name album-index \
  --key-condition-expression "Artist = :a and AlbumTitle = :t" \
  --expression-attribute-values  '{":a":{"S":"AC/DC"}, ":t": {"S": "Back in Black"}}'

{
    "Count": 1,
    "Items": [
        {
            "Artist": {
                "S": "AC/DC"
            },
            "SongTitle": {
                "S": "You Shook Me All Night Long"
            },
            "AlbumTitle": {
                "S": "Back in Black"
            }
        }
    ],
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

Delete the Table:

Delete the Table that we created:

1

$ aws dynamodb --endpoint-url http://localhost:8000 delete-table --table-name MusicCollection

Resources:

Ruby Tutorial Series Setup and Variables

In this post we will setup our Ruby environment, then start printing out values to the console and will also be touching on variables.

Ruby Environment:

I have a Docker image built on Alpine, the resources can be found via:

To setup a Ruby environment on your workstation, I would recommend using https://github.com/rbenv/rbenv.

Drop into a Ruby Shell:

I will be using Docker to drop into a ruby container:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$ docker run -it rbekker87/alpine-ruby:2.5 sh

       ______       _____
______ ___  /__________(_)___________
_  __ `/_  /___  __ \_  /__  __ \  _ \
/ /_/ /_  / __  /_/ /  / _  / / /  __/
\__,_/ /_/  _  .___//_/  /_/ /_/\___/
            /_/

Alpine Build:
Container Hostname: 8a4dfc590dd0
Checkout my Docker Blogs:
- https://sysadmins.co.za/tag/docker
- http://blog.ruanbekker.com/blog/categories/docker

$ irb
irb(main):001:0>

If you have the irb output, you should be good to go.

Strings and Integers

You will find when you enter a string, which is represented as one or more characters enclosed within quotation marks:

1
2

irb(main):001:0> "hello"
=> "hello"

The integers will be without the quotation marks, when we introduce anything within quotation marks, ruby will read it as a string. So for a integer, lets provide ruby with a number and the number will be returned to the shell:

1
2

irb(main):002:0> 1
=> 1

Using mathematical symbols like the + will either sum the two values when they are integers, or concatenate when they are strings.

Let’s start with strings: we will add the string hello and world

1
2

irb(main):003:0> "hello" + "world"
=> "helloworld"

Now let’s add two numbers together, 10 and 20:

1
2

irb(main):004:0> 10 + 20
=> 30

As you can see, it did a calculation on the two numbers as they were treated as integeres. But what happens when we add them as strings?

1
2

irb(main):005:0> "10" + "20"
=> "1020"

Adding them as strings, will concatenate them.

String Methods

Ruby’s strings has many built in methods, which makes it convenient manipulating data, let me go through a couple that I am working with:

Getting the length of the string:

1
2

irb(main):006:0> "hello".length
5

Is the string empty?

1
2

irb(main):007:0> "hello".empty?
=> false

Getting the index position of 0 of the string:

1
2

irb(main):008:0> "hello"[0]
=> "h"

Getting a array of your string:

1
2

irb(main):009:0> "hello".chars
=> ["h", "e", "l", "l", "o"]

Returning your string in Uppercase:

1
2

irb(main):010:0> "hello".upcase
=> "HELLO"

Returning your string in Lowercase:

1
2

irb(main):011:0> "HELLO".downcase
=> "hello"

Capitalize your String:

1
2

irb(main):012:0> "hello".capitalize
=> "Hello"

Swap the case of your string:

1
2

irb(main):013:0> "Hello".swapcase
=> "hELLO"

Variables

Let’s define variables to the static content that we used above.

Let’s define our key: word to the value: of hello, world:

1
2

irb(main):019:0> word = "hello, world"
=> "hello, world"

Accessing the variables value:

1
2

irb(main):020:0> word
=> "hello, world"

We can also use puts, which stands for put string, which prints out the value to the terminal:

1
2

irb(main):021:0> puts word
hello, world

We can also, format our variable so that we can add something like a exclamation mark:

1
2

irb(main):022:0> puts "#{word}!"
hello, world!

Let’s do the same with integers:

1
2
3
4

irb(main):023:0> num_1 = 10
=> 10
irb(main):024:0> num_2 = 20
=> 20

Now when we calculate the numbers using variables, you will find the expected result of 30:

1
2

irb(main):025:0> num_1 + num_2
=> 30

or:

1
2
3

irb(main):026:0> num_1 + num_2
puts "#{num_1 + num_2}"
30

Variables are Mutable:

Remember that variables are mutable, so they can be changed after they have been set, lets take age for example:

1
2
3
4
5
6
7

irb(main):027:0> age = 20
irb(main):028:0> puts age
20

irb(main):029:0> age = 22
irb(main):030:0> puts age
22

Strings and Integers:

What happens when we add strings and integers together in one line:

1
2
3
4
5
6
7
8
9
10

irb(main):038:0> name = "ruan"
=> "ruan"
irb(main):039:0> id = 120398
=> 120398
irb(main):040:0> puts "#{name + id}"
Traceback (most recent call last):
        3: from /usr/bin/irb:11:in `<main>'
        2: from (irb):40
        1: from (irb):40:in `+'
TypeError (no implicit conversion of Integer into String)

That is because we cant concatenate strings with integers, so we will need to convert the integer to a string, we do that with the to_s method:

1
2

irb(main):041:0> puts "#{name + id.to_s}"
ruan120398

And if we want to define that to a variable:

1
2
3

irb(main):042:0> userid = "#{name + id.to_s}"
irb(main):043:0> userid
=> "ruan120398"

Working with rb files:

We can add this together in a file with a .rb extension and call the file as an argument with ruby, as a script:

Create the file, in my case test.rb

1

$ vim test.rb

1
2
3
4
5

user = "ruan"
idnumber = 23049823
userid = "#{user + idnumber}"

puts "#{userid}"

Running the ruby file:

1
2

$ ruby test.rb
ruan23049823

Resources:

Ruby Programming Tutorial Series

Welcome! This will be a multi post ruby tutorial programming series, as I am on a mission learning ruby.

Outline of the Series:

This may change, but the path will look like this:

  • Setup, The Terminal and Variables
  • Arrays
  • Data Types
  • Objects, Classes and Methods

All posts associated to this tutorial series will be tagged as #ruby-tutorial-series

Resources:

Build a REST API War File for Payara With Java Springboot and Maven Part 1

This is a command line approach to create a java web app for payara that takes war files, which we will be using in conjunction with springboot and apache maven.

Setup Java and Apache Maven:

Setup Java 1.8:

1
2
3
4
5

$ apt update
$ apt install wget openssl vim software-properties-common -y
$ add-apt-repository ppa:webupd8team/java -y
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C2518248EEA14886
$ apt update && apt install oracle-java8-installer -y

Setup Apache Maven:

1
2
3
4
5
6
7

$ cd /opt
$ curl -SL  http://www-eu.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz | tar -xz
$ mv apache-maven-3.5.4 maven
$ echo 'M2_HOME=/opt/maven' > /etc/profile.d/mavenenv.sh
$ echo 'export PATH=${M2_HOME}/bin:${PATH}' >> /etc/profile.d/mavenenv.sh
$ chmod +x /etc/profile.d/mavenenv.sh
$ source /etc/profile.d/mavenenv.sh

Ensure Java and Maven is installed:

1
2
3
4
5

$ java -version
java version "1.8.0_181"

$ mvn -version
Apache Maven 3.5.4

Prepare the directories:

Prepare the directories where we will be working with our application’s source code:

1
2
3
4

$ mkdir -p /root/app
$ cd /root/app
$ mkdir -p src/main/webapp/WEB-INF
$ mkdir -p src/main/java/fish/payara/spring/boot/{controller,domain}

The source code:

The pom.xml file:

1

$ vim pom.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>fish.payara.appserver</groupId>
    <artifactId>payara-micro-with-spring-boot-rest</artifactId>
    <version>1.0</version>
    <packaging>war</packaging>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <compilerArguments>
                        <source>1.8</source>
                        <target>1.8</target>
                    </compilerArguments>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>1.2.6.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-tomcat</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

    <dependency>
        <groupId>org.springframework.batch</groupId>
        <artifactId>spring-batch-test</artifactId>
        <scope>import</scope>
    </dependency>

        <dependency>
            <groupId>org.crsh</groupId>
            <artifactId>crsh.plugins</artifactId>
            <version>1.2.11</version>
            <type>pom</type>
        </dependency>

        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.1.0</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</project>

The web.xml:

1

$ vim src/main/webapp/WEB-INF/web.xml

1
2
3
4
5
6

<web-app
    xmlns="http://xmlns.jcp.org/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaeehttp://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd"
    version="3.1">

</web-app>

The Application.java:

1

$ vim src/main/java/fish/payara/spring/boot/Application.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

package fish.payara.spring.boot;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.context.web.SpringBootServletInitializer;

@SpringBootApplication
public class Application extends SpringBootServletInitializer {

    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
        return application.sources(Application.class);
    }

    public static void main(String[] args) throws Exception {
        SpringApplication.run(Application.class, args);
    }
}

The Person.java:

1

$ vim src/main/java/fish/payara/spring/boot/domain/Person.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

package fish.payara.spring.boot.domain;

public class Person {

    private int id;
    private String name;
    private String lastName;
    private String email;

    public Person() {
    }

    public Person(int id, String name, String lastName, String email) {
        this.id = id;
        this.name = name;
        this.lastName = lastName;
        this.email = email;
    }

    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }
}

The PersonRestController.java:

1

$ src/main/java/fish/payara/spring/boot/controller/PersonRestController.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package fish.payara.spring.boot.controller;

import fish.payara.spring.boot.domain.Person;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.annotation.PostConstruct;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;

@RestController
@RequestMapping("/person")
public class PersonRestController {

    Map<Integer, Person> personMap = new HashMap<>();

    @PostConstruct
    public void init() {
        personMap.put(1, new Person(1, "Ruan", "Bekker", "ruan@gmail.com"));
        personMap.put(2, new Person(2, "Steve", "James", "steve@gmail.com"));
        personMap.put(3, new Person(3, "Frank", "Phillips", "frank@gmail.com"));
    }

    @RequestMapping("/all")
    public Collection<Person> getAll() {
        return personMap.values();
    }
}

Build with Maven:

Build the war file with maven:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$ mvn clean package

[INFO] Packaging webapp
[INFO] Assembling webapp [payara-micro-with-spring-boot-rest] in [/root/app/target/payara-micro-with-spring-boot-rest-1.0]
[INFO] Processing war project
[INFO] Copying webapp resources [/root/app/src/main/webapp]
[INFO] Webapp assembled in [113 msecs]
[INFO] Building war: /root/app/target/payara-micro-with-spring-boot-rest-1.0.war
[INFO] WEB-INF/web.xml already added, skipping
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.662 s
[INFO] Finished at: 2018-08-04T10:46:50Z
[INFO] ------------------------------------------------------------------------

You will find your war file under:

1
2

$ ls target/
classes  maven-archiver  maven-status  payara-micro-with-spring-boot-rest-1.0  payara-micro-with-spring-boot-rest-1.0.war

You can change the name in the pom.xml, but since we already built it, lets rename the file to something shorter:

1

$ mv /root/app/target/payara-micro-with-spring-boot-rest-1.0.war /root/app/target/webapp.war

Deploy your Application with Payara Micro:

Deploy your application with docker:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

$ docker run -it -p 8080:8080 -v /root/app/target:/opt/payara/deployments payara/micro --deploy /opt/payara/deployments/webapp.war


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.2.6.RELEASE)

{
    "Instance Configuration": {
        "Host": "4e90ecf6a1a7",
        "Http Port(s)": "8080",
        "Https Port(s)": "",
        "Instance Name": "Cloudy-Chub",
        "Instance Group": "MicroShoal",
        "Hazelcast Member UUID": "a1af817d-473b-4fa7-9ee9-7d53291a35a2",
        "Deployed": [
            {
                "Name": "webapp",
                "Type": "war",
                "Context Root": "/webapp"
            }
        ]
    }
}
2018-08-04 11:26:39.655  INFO 1 --- [           main] PayaraMicro                              :
Payara Micro URLs:
http://4e90ecf6a1a7:8080/webapp

Testing

Let’s hit our app’s health endpoint to test:

1
2
3
4

$ curl -s http://localhost:8080/webapp/health | jq .
{
  "status": "UP"
}

Now to interact with our API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

$ curl -s http://localhost:8080/webapp/person/all | jq .
[
  {
    "id": 1,
    "name": "Ruan",
    "lastName": "Bekker",
    "email": "ruan@gmail.com"
  },
  {
    "id": 2,
    "name": "Steve",
    "lastName": "James",
    "email": "steve@gmail.com"
  },
  {
    "id": 3,
    "name": "Frank",
    "lastName": "Phillips",
    "email": "frank@gmail.com"
  }
]

Payara also provides a /metrics endpoint:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

$ curl -s http://localhost:8080/webapp/metrics | jq .
{
  "mem": 219648,
  "mem.free": 67104,
  "processors": 4,
  "instance.uptime": 369749,
  "uptime": 390417,
  "systemload.average": 0.14697265625,
  "heap.committed": 219648,
  "heap.init": 32768,
  "heap.used": 152543,
  "heap": 455168,
  "threads.peak": 98,
  "threads.daemon": 37,
  "threads": 72,
  "classes": 16951,
  "classes.loaded": 16951,
  "classes.unloaded": 0,
  "gc.ps_scavenge.count": 42,
  "gc.ps_scavenge.time": 515,
  "gc.ps_marksweep.count": 4,
  "gc.ps_marksweep.time": 634,
  "counter.status.200.health": 1,
  "counter.status.200.mappings": 2,
  "counter.status.200.person.all": 2,
  "counter.status.404.error": 5,
  "gauge.response.error": 6,
  "gauge.response.health": 120,
  "gauge.response.mappings": 3,
  "gauge.response.person.all": 9
}

And to get a mapping of all the endpoints:

1

$ curl -s http://localhost:8080/webapp/mappings | jq .

If you decided to deploy as a jar, you can use the payara-micro jar to deploy the war file:

1

$ java -jar payara-micro-5.182.jar --deploy target/webapp.war

For more info on this, have a look at their website

Hello World Web App With Java Springboot and Maven

In this post we will setup a Java Hello World Web App, using Maven and SpringBoot on Ubuntu 16. I will create all the needed files in this tutorial, but you can head to start.spring.io to generate the zip for you.

Setup Java

Setup Java 1.8:

1
2
3
4
5

$ apt update
$ apt install wget openssl vim software-properties-common -y
$ add-apt-repository ppa:webupd8team/java -y
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C2518248EEA14886
$ apt update && apt install oracle-java8-installer -y

Ensure that Java is installed:

1
2
3
4

$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Install Apache Maven:

Maven is a build automation tool used primarily for Java projects. Let’s setup Maven:

1
2
3
4
5
6
7

$ cd /opt
$ curl -SL  http://www-eu.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz | tar -xz
$ mv apache-maven-3.5.4 maven
$ echo 'M2_HOME=/opt/maven' > /etc/profile.d/mavenenv.sh
$ echo 'export PATH=${M2_HOME}/bin:${PATH}' >> /etc/profile.d/mavenenv.sh
$ chmod +x /etc/profile.d/mavenenv.sh
$ source /etc/profile.d/mavenenv.sh

Verify that Maven is installed:

1
2
3
4
5
6

$ mvn -version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Maven home: /opt/maven
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "4.9.87-linuxkit-aufs", arch: "amd64", family: "unix"

Setup the Application:

Create the home directory:

1

$ mkdir myapp && cd myapp

Create the directory structure:

1

$ mkdir -p src/main/java/hello

Create and Edit the pom.xml:

1

$ vim pom.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>hello</groupId>
    <artifactId>myapp</artifactId>
    <version>1.0</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.9.RELEASE</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
    </dependencies>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Create the Main Application Class:

1

$ vim src/main/java/hello/MainApplicationClass.java

1
2
3
4
5
6
7
8
9
10
11
12

package hello;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class MainApplicationClass {

    public static void main(String[] args) {
        SpringApplication.run(MainApplicationClass.class, args);
    }
}

Create the Route Controller:

1

$ vim src/main/java/hello/HelloController.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

package hello;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class HelloController {

    @RequestMapping("/")
        public String index() {
      return "This is the index!\n";
  }
    @RequestMapping("/hello")
        public String index2() {

      return "Hello, World!\n";
  }

}

Build and Compile:

This will download all the dependencies and build the jar file:

1

$ mvn clean package

Start and Test the Application:

Run the application:

1
2
3
4
5
6
7
8
9
10
11
12

$ java -jar target/myapp-1.0.jar

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.5.9.RELEASE)
...
2018-08-03 12:31:06.967  INFO 5594 --- [           main] hello.MainApplicationClass               : Started MainApplicationClass in 3.656 seconds (JVM running for 4.243)
^

Test the Application:

1
2

$ curl http://localhost:8080/
This is the index!

And for our /hello route:

1
2

$ curl http://localhost:8080/hello
Hello, World!

Hello World Ruby on Rails App Tutorial Using Mac

In this tutorial, we will setup a basic ruby on rails web app, that consists of a /hello_world and a /status controller. The hello_world controller will return Hello, World and our /status controller will return a HTTP 204 no content response code.

Setup Ruby on Rails

Setup Ruby on Rails on your Mac:

1
2
3
4
5
6
7
8
9
10
11

$ brew install rbenv ruby-build

$ echo 'if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi' >> ~/.bash_profile
$ source ~/.bash_profile

$ rbenv install 2.5.0
$ rbenv global 2.5.0
$ ruby -v

$ gem install rails -v 5.1.4
$ benv rehash

Creating the App

Create your ruby on rails application:

1
2
3

$ rails new fist-app
$ cd first-app
$ rails server

Route Config

Our routes config:

1
2
3
4
5
6

$ cat config/routes.rb
Rails.application.routes.draw do
  # For details on the DSL available within this file, see http://guides.rubyonrails.org/routing.html
  get 'hello_world', to: 'hello_world#call'
  get 'status', to: 'status#call'
end

Controllers

Configure the hello_world controller:

1
2
3
4
5
6
7

$ cat app/controllers/hello_world_controller.rb

class HelloWorldController < ApplicationController
  def call
    render body: "Hello, World"
  end
end

Configure the status controller:

1
2
3
4
5
6
7

$ cat app/controllers/status_controller.rb

class StatusController < ApplicationController
  def call
    [204, {}, ['']]
  end
end

Testing

For our hello world controller:

1
2
3
4
5
6
7
8
9
10
11
12

$ curl -i http://localhost:3000/hello_world
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/plain; charset=utf-8
ETag: W/"565339bc4d33d72817b583024112eb7f"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: 62441a6d-faa3-42d5-a5a2-bcf7eff5e917
X-Runtime: 0.001940
Transfer-Encoding: chunked
Hello, World

For our status controller:

1
2
3
4
5
6
7
8

$ curl -i http://localhost:3000/status
HTTP/1.1 204 No Content
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-cache
X-Request-Id: bec91213-ff82-4fc6-8698-3ee7622b1f51
X-Runtime: 0.075504

Resources:

Capture Geo Location Data With Python Flask and PyGeoIP

With the PyGeoIP package you can capture geo location data, which is pretty cool, for example, when you have IOT devices pushing location data to elasticsearch and visualizing the data with Kibana. That will be one example, but the possibilites are endless.

Dependencies:

Get the Maxmind Geo Database:

1
2

$ wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
$ gunzip GeoLiteCity.dat.gz

Install Python Flask and PyGeoIP:

1

$ pip install flask pygeoip

Getting Started with PyGeoIP:

Let’s run through a couple of examples on how to get:

  • Country Name by IP Address and DNS
  • Country Code by IP Address and DNS
  • GeoData by IP Address
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

>>> import pygeoip, json
>>> gi = pygeoip.GeoIP('GeoLiteCity.dat')

>>> gi.country_name_by_addr('8.8.8.8')
'United States'
>>> gi.country_code_by_addr('8.8.8.8')
'US'

>>> gi.country_name_by_name('scaleway.com')
'France'
>>> gi.country_code_by_name('scaleway.com')
'FR'

>>> gi.region_by_name('scaleway.com')
{'region_code': None, 'country_code': 'FR'}

>>> data = gi.record_by_addr('104.244.42.193')
>>> print(json.dumps(data, indent=2))
{
  "city": "San Francisco",
  "region_code": "CA",
  "area_code": 415,
  "time_zone": "America/Los_Angeles",
  "dma_code": 807,
  "metro_code": "San Francisco, CA",
  "country_code3": "USA",
  "latitude": 37.775800000000004,
  "postal_code": "94103",
  "longitude": -122.4128,
  "country_code": "US",
  "country_name": "United States",
  "continent": "NA"
}

>>> data = gi.record_by_name('twitter.com')
>>> print(json.dumps(data, indent=2))
{
  "city": "San Francisco",
  "region_code": "CA",
  "area_code": 415,
  "time_zone": "America/Los_Angeles",
  "dma_code": 807,
  "metro_code": "San Francisco, CA",
  "country_code3": "USA",
  "latitude": 37.775800000000004,
  "postal_code": "94103",
  "longitude": -122.4128,
  "country_code": "US",
  "country_name": "United States",
  "continent": "NA"
}

Python Flask Web App to Capture Data

Let’s create a basic Flask App that will capture the data from the client making the request to the server. In this example we will just return the data, but we can filter the data and ingest it into a database like elasticsearch, etc.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

from flask import Flask, request, jsonify
import pygeoip, json

app = Flask(__name__)

geo = pygeoip.GeoIP('GeoLiteCity.dat', pygeoip.MEMORY_CACHE)

@app.route('/')
def index():
    client_ip = request.remote_addr
    geo_data = geo.record_by_addr(client_ip)
    return json.dumps(geo_data, indent=2) + '\n'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=80, debug=False)

Run the Server:

1

$ python app.py

Make a request from the client over a remote connection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ curl http://remote-endpoint.com
{
  "city": "Cape Town",
  "region_code": "11",
  "area_code": 0,
  "time_zone": "Africa/Johannesburg",
  "dma_code": 0,
  "metro_code": null,
  "country_code3": "ZAF",
  "latitude": -01.12345,
  "postal_code": "8000",
  "longitude": 02.123456789,
  "country_code": "ZA",
  "country_name": "South Africa",
  "continent": "AF"
}

Resources:

Install Java Development Kit 10 on Ubuntu

With the announcement of improved docker container integration with Java 10, the JVM is now aware of resource constraints, as not from prior versions. More information on this post

Differences in Java 8 and Java 10:

As you can see with Java 8:

1
2
3
4
5

$ docker run -it -m512M --entrypoint bash openjdk:latest

$ docker-java-home/bin/java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
    uintx MaxHeapSize                              := 524288000                          {product}
openjdk version "1.8.0_162"

And with Java 10:

1
2
3
4
5

$ docker run -it -m512M --entrypoint bash openjdk:10-jdk

$ docker-java-home/bin/java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
   size_t MaxHeapSize                              = 134217728                                {product} {ergonomic}
openjdk version "10" 2018-03-20

Installing JDK 10 on Ubuntu:

Installing Java Development Kit 10:

1
2
3
4
5

$ apt update && apt upgrade -y
$ add-apt-repository ppa:linuxuprising/java
$ apt update
$ apt install oracle-java10-installer
$ apt install oracle-java10-set-default

Confirming the Java Version:

1
2
3
4

$ java -version
java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)