Ruan Bekker's Blog

From a Curious mind to Posts on Github

Authenticate to Your AWS MySQL RDS Instance via IAM

On Amazon Web Services with RDS for MySQL or Aurora with MySQL compatibility, you can authenticate to your Database instance or cluster using IAM for database authentication. The benefit of using this authentication method is that you don’t need to use a password when you connect to your database, but you use your authentication token instead

Update: - Amazon Supports IAM Authentication for PostgreSQL

  • More info from their docs

What we will be doing today:

We will do the following:

  • Create RDS MySQL Database
  • Create IAM Policy that allows a user to connect via a MySQL User
  • Create IAM User and associate IAM Policy
  • Configure the new user credentials in the awscli credential provider
  • Bash script to generate the auth token and authenticate to RDS via Token instead of password

Create the RDS Database:

In this tutorial I will spin up a db.t2.micro in eu-west-1 with IAMDatabaseAuthentication Enabled:

1
2
3
4
5
6
7
8
9
aws rds create-db-instance \
    --db-instance-identifier rbtest \
    --db-instance-class db.t2.micro \
    --engine MySQL \
    --allocated-storage 20 \
    --master-username dbadmin \
    --master-user-password mysuperpassword \
    --region eu-west-1 \
    --enable-iam-database-authentication

Give it some time to spin up, then get your database endpoint:

1
2
$ aws rds describe-db-instances --db-instance-identifier rbtest | jq -r ".DBInstances[].Endpoint.Address"
rbtest.abcdefgh.eu-west-1.rds.amazonaws.com

If you need to have SSL Enabled, get the bundled certificate as described in the Using SSL with RDS docs.

1
wget -O /tmp/rds.pem https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem

Create the Database Account:

Create the database account on the MySQL RDS instance as described from their docs. IAM handles the authentication via AWSAuthenticationPlugin, therefore we do not need to set passwords on the database.

Connect to the database:

1
$ mysql -u dbadmin -h rbtest.abcdefgh.eu-west-1.rds.amazonaws.com -p

Create the database:

1
2
mysql> CREATE USER mydbaccount IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
mysql> FLUSH PRIVILEGES;

Creating the Databases and Granting Permissions

While you are on the database, create 2 databases (db1 and db2) with some tables, which we will use for our user to have read only access to, and create one database (db3) which the user will not have access to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
mysql> create database db1;
mysql> create database db2;

mysql> use db1;
mysql> create table foo (name VARCHAR(20), age INT);
mysql> insert into foo values('ruan', 31);
mysql> insert into foo values('james', 32);

mysql> use db2;
mysql> create table foo (location VARCHAR(255));
mysql> insert into foo values('south africa');
mysql> insert into foo values('new zealand');
mysql> insert into foo values('australia');

mysql> grant select on db1.* to 'mydbuser';
mysql> grant select on db2.* to 'mydbuser';

mysql> create database db3;
mysql> use db3;
mysql> create table foo (passwords VARCHAR(255));
mysql> insert into foo values('superpassword');
mysql> insert into foo values('sekret');

mysql> flush privileges;

IAM Permissions to allow our user to authenticate to our RDS.

First to create the user and configure awscli tools. My default profile has administrative access, so we will create our db user in its own profile and configure our awscli tools with its new access key and secret key:

1
2
3
4
5
$ aws configure --profile dbuser
AWS Access Key ID [None]: xxxxxxxxxxxxx
AWS Secret Access Key [None]: xxxxxxxxxxxxxxxxxx
Default region name [None]: eu-west-1
Default output format [None]: json

Now we need to create a IAM policy to allow our user to authenticate to our RDS Instance via IAM, which we will associate with our Users account.

We need the AWS Account ID, the Database Identifier Resource ID, and the User Account that we created on MySQL.

To get the DB ResourceId:

1
2
$ aws rds describe-db-instances --db-instance-identifier rbtest | jq -r ".DBInstances[].DbiResourceId
db-123456789ABCDEFGH

Create the IAM Policy and associate it with the new user account:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
   "Version": "2012-10-17",
   "Statement": [
      {
           "Sid": "RDSIAMAUTH",
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:eu-west-1:123456789012:dbuser:db-123456789ABCDEFGH/mydbaccount"
         ]
      }
   ]
}

The bash script will get the authentication token which will be used as the password. Note that the authentication token will expire after 15 minutes after creation. The docs

1
2
3
4
5
#!/bin/bash
db_endpoint="rbtest.abcdefgh.eu-west-1.rds.amazonaws.com"
local_mysql_user="mydbaccount"
auth_token="$(aws --profile dbuser rds generate-db-auth-token --hostname ${RDSHOST} --port 3306 --username ${local_mysql_user} )"
mysql --host=${db_endpoint} --port=3306 --enable-cleartext-plugin --user=${local_mysql_user} --password=${auth_token}

Testing it out:

Now that our policies are in place, credentials from the credential provider has been set and our bash script is setup, lets connect to our database:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
./conn-mysql.sh

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| db1                |
| db2                |
+--------------------+
3 rows in set (0.16 sec)

mysql> select * from db2.foo;
+--------------+
| location     |
+--------------+
| south africa |
| new zealand  |
| australia    |
+--------------+

mysql> select * from db3.foo;
ERROR 1044 (42000): Access denied for user 'mydbaccount'@'*' to database 'db3'

mysql> create database test123;
ERROR 1044 (42000): Access denied for user 'mydbaccount'@'%' to database 'test123'

Changing the IAM Policy to revoke access:

1
2
3
./conn-mysql.sh
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'mydbaccount'@'10.0.0.10' (using password: YES)

Creating a MySQL Client Wrapper Script:

Using bash we can create a wrapper script so we can connect to our database like the following:

1
2
$ mysql-iam prod rbtest.eu-west-1.amazonaws.com mydbaccount
mysql>

Here is the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#!/usr/bin/env bash

# Wrapper MySQL Client for IAM Based Authentication for MySQL and Amazon Aurora on RDS
# Read: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html
# Usage: [app] [aws_profile] [rds_endpoint] [rds_mysql_username]

command_exists() {
  type "$1" &> /dev/null ;
}

check_required_parameters() {
  aws_profile="$1"
  rds_hostname="$2"
  rds_username="$3"
  if ! [[ -n "$aws_profile" && -n "$rds_username" && -n "$rds_username" ]]
    then
      echo "Error: Missing Parameters"
      echo "Expected: $0 aws_profile_name rds_endpoint_name rds_db_username"
      echo "Usage: $0 prod dbname.eu-west-1.amazonaws.com dba"
      exit 1
  fi
}

get_auth_token() {
  aws_bin=$(which aws | head -1)
  auth_token="$($aws_bin --profile $aws_profile rds generate-db-auth-token --hostname $rds_hostname --port 3306 --username $rds_username )"
}

connect_to_rds() {
  mysql_bin=$(which mysql | head -1)
  ${mysql_bin} --host=${rds_hostname} --port=3306 --enable-cleartext-plugin --user=${rds_username} --password=${auth_token}
}

if [ "$1" == "help" ]
  then
    echo "Help"
    echo "Expected: $0 aws_profile_name rds_endpoint_name rds_db_username"
    echo "Usage: $0 prod dbname.eu-west-1.amazonaws.com dba_user"
    exit 0
fi

if command_exists aws && command_exists mysql
then
  check_required_parameters $1 $2 $3
  get_auth_token
  connect_to_rds
else
  echo "Error: Make sure aws-cli and mysql client is installed"
fi

For more information on this, have a look at the docs

Using Hive for Small Datasets on My Mac Using Docker

I wanted to process a small subset of data, and not wanting to spin up a cluster, so I used nagasuga/docker-hive docker image to run Hive on my Mac.

Running Hive

1
2
$ docker run -it -v /home/me/resource-data.csv:/resource-data.csv nagasuga/docker-hive /bin/bash -c 'cd /usr/local/hive && ./bin/hive'
hive>

Once I was entered into my hive shell, I created a table for my CSV data:

Creating the Table

1
2
3
4
5
6
7
8
9
10
hive> create table resources (ResourceType STRING, Owner STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ;

hive> hive> show tables;
OK
resources

hive> describe resources;
OK
resourcetype           string
owner                  string

Loading the Data

My csv data is located at /resource-data.csv on the container, which I will load into my table:

1
2
hive> load data local inpath '/resource-data.csv' into table resources;
Loading data to table default.resources

Query the Data

Just two simple queries for demonstration:

1
2
3
4
5
6
7
8
9
10
11
hive> select * from resources limit 3;
OK
EC2     Engineering
EC2     Finance
EC2     Product

hive> hive> select count(resourcetype) as num, owner from resources group by owner order by num desc limit 3;
K
50     Engineering
20     Product
10     Finance

Resource:

Thanks to https://github.com/nagasuga/docker-hive

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


Ad space:

Thanks for reading!

Using Getpass in Python to Accept Passwords From Stdin Without Echoing It Back

Using raw_input in python expects standard input, which echo’s it back after enter is executed, below is an example:

1
2
3
4
5
>>> word = raw_input("What is the word? \n")
What is the word?
football
>>> print(word)
football

Using getpass, the standard input gets masked, like you would expect when entering a password, like below:

1
2
3
4
5
>>> from getpass import getpass
>>> word = getpass()
Password:
>>> print(word)
rugby

Changing the default prompt:

1
2
3
4
>>> word = getpass(prompt='What is your name? ')
What is your name?
>>> print(word)
Ruan

Creating a Simple Insecure Password Verification App:

1
2
3
4
5
6
7
from getpass import getpass

password = getpass()
if password.lower() == 'simplepass':
    print 'Password Correct'
else:
    print 'Password Failed'

Testing it, by first entering a incorrect string, then the correct one:

1
2
3
4
5
6
7
$ python auth-check.py
Password:
Password Failed

$ python auth-check.py
Password:
Password Correct

You definitely don’t want to hard code the credentials in your app, but you get the idea.

Setup a Basic Hello World Pipeline on Concourse

We will setup a basic pipeline that pulls down content from github, then executes a task that prints hello world.

Content on Github

The config can be found on my Github Branch but I will display each file in this post.

Running our Pipeline

Our pipeline.yml that we need to have for concourse to know what to do:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
---
resources:
- name: my-git-repo
  type: git
  source:
    uri: https://github.com/ruanbekker/concourse-test
    branch: basic-helloworld

jobs:
- name: hello-world-job
  public: true
  plan:
  - get: my-git-repo
  - task: task_print-hello-world
    file: my-git-repo/ci/task-hello-world.yml

We can see from our pipeline.yml file, it points to a task-hello-world.yml, which I will preview below, but can be found in the repo:

1
2
3
4
5
6
7
8
9
10
11
---
platform: linux

image_resource:
  type: docker-image
  source:
    repository: busybox

run:
  path: echo
  args: ["hello world"]

Set Pipeline:

1
$ fly -t tutorial sp -c pipeline.yml -p pipeline-01

Unpause Pipeline:

1
$ fly -t tutorial up -p pipeline-01

Trigger Job:

1
2
3
4
5
6
7
8
9
10
$ fly -t tutorial tj -j pipeline-01/hello-world-job --watch
started pipeline-01/hello-world-job #2

Cloning into '/tmp/build/get'...
Fetching HEAD
292c84b change task name
initializing
running echo hello world
hello world
succeeded

This was all done through the command line, but you can also accessed it from the web ui

Change Your Relayhost on Postfix Using Sed

a Quick post on how to change your relayhost on Postfix to a External SMTP Provider and aswell how to revert back the changes so the Relay server sends out mail directly.

Checking your current relayhost configuration:

We will assume your /etc/postfix/main.cf has a relayhost entry of #relayhost =, in my example it will look like this:

1
2
$ cat /etc/postfix/main.cf
#relayhost =

If not, you can just adjust your sed command accordingly.

Changing your relayhost configuration to a External SMTP Provider:

We will use sed to change the relayhost to za-smtp-outbound-1.mimecast.co.za for example:

1
$ sed -i 's/#relayhost\ =/relayhost\ =\ \[za-smtp-outbound-1.mimecast.co.za\]/g' /etc/postfix/main.cf

to verify that we have set the config, look at the config:

1
2
$ cat /etc/postfix/main.cf | grep relayhost
relayhost = [za-smtp-outbound-1.mimecast.co.za]

Once you see the changes looks as expected, you can restart postfix:

1
$ /etc/init.d/postfix restart

Then you can tail the logs to see if the mail gets delivered:

1
$ tail -f /var/log/maillog

Revert your changes so that postfix sends out directly:

To revert your changes, let’s change the config back to what it was:

1
$ sed -i 's/relayhost\ =\ \[za-smtp-outbound-1.mimecast.co.za\]/#relayhost\ =/g' /etc/postfix/main.cf

To verify your changes:

1
2
$ cat /etc/postfix/main.cf | grep relayhost
#relayhost =

As you can see the relayhost is commented out, meaning that the relayhost property will not be active, go ahead and restart the service for the changes to take effect:

1
$ /etc/init.d/postfix restart

Same as before, look at the logs to confirm mailflow is as expected:

1
$ tail -f /var/log/maillog

Graphing Pretty Charts With Python Flask and Chartjs

I am a big sucker for Charts and Graphs, and today I found one awesome library called Chart.js, which we will use with Python Flask Web Framework, to graph our data.

As Bitcoin is doing so well, I decided to graph the monthly Bitcoin price from January up until now.

Dependencies:

Install Flask:

1
$ pip install flask

Create the files and directories:

1
2
$ touch app.py
$ mkdir templates

We need the Chart.js library, but I will use the CDN version, in my html.

Creating the Flask App:

Our data that we want to graph will be hard-coded in our application, but there are many ways to make this more dynamic, in your app.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from flask import Flask, Markup, render_template

app = Flask(__name__)

labels = [
    'JAN', 'FEB', 'MAR', 'APR',
    'MAY', 'JUN', 'JUL', 'AUG',
    'SEP', 'OCT', 'NOV', 'DEC'
]

values = [
    967.67, 1190.89, 1079.75, 1349.19,
    2328.91, 2504.28, 2873.83, 4764.87,
    4349.29, 6458.30, 9907, 16297
]

colors = [
    "#F7464A", "#46BFBD", "#FDB45C", "#FEDCBA",
    "#ABCDEF", "#DDDDDD", "#ABCABC", "#4169E1",
    "#C71585", "#FF4500", "#FEDCBA", "#46BFBD"]

@app.route('/bar')
def bar():
    bar_labels=labels
    bar_values=values
    return render_template('bar_chart.html', title='Bitcoin Monthly Price in USD', max=17000, labels=bar_labels, values=bar_values)

@app.route('/line')
def line():
    line_labels=labels
    line_values=values
    return render_template('line_chart.html', title='Bitcoin Monthly Price in USD', max=17000, labels=line_labels, values=line_values)

@app.route('/pie')
def pie():
    pie_labels = labels
    pie_values = values
    return render_template('pie_chart.html', title='Bitcoin Monthly Price in USD', max=17000, set=zip(values, labels, colors))

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Populating the HTML Static Content:

As we are using render_template we need to populate our html files in our templates/ directory. As you can see we have 3 different html files:

  • templates/bar_chart.html :
  • templates/line_chart.html:
  • templates/pie_chart.html:

Running our Application:

As you can see, we have 3 endpoints, each representing a different chart style:

  • /line
  • /bar
  • /pie

Let’s start our flask application:

1
$ python app.py

When we access our /line endpoint:

When we access our /bar endpoint:

When we access our /pie endpoint:

Resources:

Create a Chatbot With Chatterbot on Python

So I’ve been wanting to take a stab at chatbots for some time, and recently discovered Chatterbot, so in this tutorial I will go through some examples on setting up a very basic chatbot.

Getting the Dependencies:

I will be using Alpine on Docker to run all the the examples, I am using Alpine so that we have a basic container with nothing special pre-installed.

Chatterbot is written in Python, so let’s install Python and Chatterbot:

1
2
3
4
$ docker run -it --name chatbot alpine:edge sh
$ apk update && apk add python py2-pip
$ pip install pip --upgrade --user
$ pip install chatterbot

Setup the Basic Chatbot:

Now that our dependencies is installed, enter the Python interpreter where we will instantiate our Chatbot, and get a response from our Chatbot. By default the library will create a sqlite database to build up statements that is passed to and from the bot.

At this point, the bot is still pretty useless:

1
2
3
4
5
6
7
$ python
>>> from chatterbot import ChatBot
>>> chatbot = ChatBot('Ben')
>>> chatbot.get_response('What is your name?')
<Statement text:What is your name?>
>>> chatbot.get_response('My name is Ruan, what is your name?')
<Statement text:What is your name?>

Training your Bot:

To enable your bot to have some knowledge, we can train the bot with training data. The training data is populated in a list, which will represent the conversation.

Exit the python interpreter and delete the sqlite database:

1
$ rm -rf db.sqlite3

Now our Bot wont have any history of what we said. Start the interpreter again and add some data to train our bot. In this example, we want our Chatbot to respond when we ask it, what his name is:

1
2
3
4
5
6
>>> from chatterbot import ChatBot
>>> from chatterbot.trainers import ListTrainer
>>> chatbot = ChatBot('Ben')
>>> chatbot.set_trainer(ListTrainer)
>>> chatbot.train(['What is your name?', 'My name is Ben'])
List Trainer: [####################] 100%

Now that we have trained our bot, let’s try to chat to our bot:

1
2
3
4
>>> chatbot.get_response('What is your name?')
<Statement text:My name is Ben>
>>> chatbot.get_response('Who is Ben?')
<Statement text:My name is Ben>

We can also enable our bot to respond on multiple statements:

1
2
3
4
5
6
7
>>> chatbot.train(['Do you know someone with the name of Sarah?', 'Yes, my sisters name is Sarah', 'Is your sisters name, Sarah?', 'Faw shizzle!'])
List Trainer: [####################] 100%

>>> chatbot.get_response('do you know someone with the name of Sarah?')
<Statement text:Yes, my sisters name is Sarah>
>>> chatbot.get_response('is your sisters name Sarah?')
<Statement text:Faw shizzle!>

With that said, we can define our list of statements in our code:

1
2
3
4
5
6
7
8
9
10
11
>>> conversations = [
...     'Are you an athlete?', 'No, are you mad? I am a bot',
...     'Do you like big bang theory?', 'Bazinga!',
...     'What is my name?', 'Ruan',
...     'What color is the sky?', 'Blue, stop asking me stupid questions'
... ]

>>> chatbot.train(conversations)
List Trainer: [####################] 100%
>>> chatbot.get_response('What color is the sky?')
<Statement text:Blue, stop asking me stupid questions>

So we can see it works as expected, but let’s state one of the answers from our statements, to see what happens:

1
2
3
4
>>> chatbot.get_response('Bazinga')
<Statement text:What is my name?>
>>> chatbot.get_response('Your name is Ben')
<Statement text:Yes, my name is Ben>

So we can see it uses natural language processing to learn from the data that we provide our bot. Just to check another question:

1
2
>>> chatbot.get_response('Do you like big bang theory?')
<Statement text:Bazinga!>

If we have quite a large subset of learning data, we can add all the data in a file, seperated by new lines then we can use python to read the data from disk, and split up the data in the expected format.

The training file will reside in our working directory, let’s name it training-data.txt and the content will look like this:

1
2
3
4
What is Bitcoin?
Bitcoin is a Crypto Currency
Where is this blog hosted?
Github

A visual example of how we will process this data will look like this:

1
2
3
>>> data = open('training-data.txt').read()
>>> data.strip().split('\n')
['What is Bitcoin?', 'Bitcoin is a Crypto Currency', 'Where is this blog hosted?', 'Github']

And in action, it will look like this:

1
2
3
4
5
6
7
>>> data = open('training-data.txt').read()
>>> conversations = data.strip().split('\n')
>>> chatbot.train(conversations)
List Trainer: [####################] 100%

>>> chatbot.get_response('Where is this blog hosted?')
<Statement text:Github>

There is also pre-populated data that you can use to train your bot, on the documentation is a couple of examples, but for demonstration, we will use the CorpusTrainer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
>>> from chatterbot.trainers import ChatterBotCorpusTrainer
>>> chatterbot.set_trainer(ChatterBotCorpusTrainer)
>>> chatbot.train("chatterbot.corpus.english")
ai.yml Training: [####################] 100%
botprofile.yml Training: [####################] 100%
computers.yml Training: [####################] 100%
conversations.yml Training: [####################] 100%
emotion.yml Training: [####################] 100%
food.yml Training: [####################] 100%
gossip.yml Training: [####################] 100%
greetings.yml Training: [####################] 100%
history.yml Training: [####################] 100%
humor.yml Training: [####################] 100%
literature.yml Training: [####################] 100%
money.yml Training: [####################] 100%
movies.yml Training: [####################] 100%
politics.yml Training: [####################] 100%
psychology.yml Training: [####################] 100%
science.yml Training: [####################] 100%
sports.yml Training: [####################] 100%
trivia.yml Training: [####################] 100%

>>> chatbot.get_response('Do you like peace?')
<Statement text:not especially. i am not into violence.>
>>> chatbot.get_response('Are you emotional?')
<Statement text:Sort of.>
>>> chatbot.get_response('What language do you speak?')
<Statement text:Python.>
>>> chatbot.get_response('What is your name?')
<Statement text:My name is Ben>
>>> chatbot.get_response('Who is the President of America?')
<Statement text:Richard Nixon> #data seems outdated :D
>>> chatbot.get_response('I like cheese')
<Statement text:What kind of movies do you like?>

Using an External Database like MongoDB

Instead of using sqlite on the same host, we can use a NoSQL Database like MongoDB that resides outside our application.

For the sake of this tutorial, I will use Docker to spin up a MongoDB Container:

1
$ docker run -d --name mongodb -p 27017:27017 -p 28017:28017 -e AUTH=no -e OPLOG_SIZE=50 tutum/mongodb

Below is my code of a terminal application that uses Chatterbot, MongoDB as a Storage Adapter, and we are using a while loop, so that we can chat with our bot, and in our except statement, we can stop our application by using our keyboard to exit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer

chatbot = ChatBot(
    "Chatbot Backed by MongoDB",
    storage_adapter="chatterbot.storage.MongoDatabaseAdapter",
    database="chatterbot_db",
    database_uri="mongodb://172.17.0.3:27017/",
    logic_adapters=[
        'chatterbot.logic.BestMatch'
    ],
    trainer='chatterbot.trainers.ChatterBotCorpusTrainer',
    filters=[
        'chatterbot.filters.RepetitiveResponseFilter'
    ],
    input_adapter='chatterbot.input.TerminalAdapter',
    output_adapter='chatterbot.output.TerminalAdapter'
)

chatbot.set_trainer(ChatterBotCorpusTrainer)
chatbot.train("chatterbot.corpus.english")

print('Chatbot Started:')

while True:
    try:
        print(" -> You:")
        botInput = chatbot.get_response(None)
    except (KeyboardInterrupt, EOFError, SystemExit):
        break

Running the example:

1
2
3
4
5
6
7
$ python bot.py
 -> You:
How are you?
I am doing well.
 -> You:
Tell me a joke
A 3-legged dog walks into an old west saloon, slides up to the bar and announces "I'm looking for the man who shot my paw."

And from mongodb, we can see some data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ mongo
> show dbs
admin          0.078GB
chatterbot_db  0.078GB
local          0.078GB

> use chatterbot_db
switched to db chatterbot_db

> show collections;
conversations
statements
system.indexes

> db.conversations.find().count()
4
> db.statements.find().count()
1240
> db.system.indexes.find().count()
3

That was a basic tutorial on Chatterbot, next I will be looking into mining data from Twitter’s API and see how clever our bot can become.

Resources:

SSH Host Key Warnings With Strict Checking Enabled

When you format / reload a server and the host gets the same IP, when you try to SSH to that host, you might get a warning like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ ssh 192.168.1.104
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
a1:a2:a3:a4:a5:a6:a7:a8:a9:b0:b1:b2:b3:b4:b5:b6.
Please contact your system administrator.
Add correct host key in /home/pi/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/pi/.ssh/known_hosts:10
ECDSA host key for 192.168.1.104 has changed and you have requested strict checking.
Host key verification failed.

This is because we have StrictMode enabled in our SSH Configuration:

1
2
$ cat /etc/ssh/sshd_config | grep -i stric
StrictModes yes

To remove the offending key from your known_hosts file, without opening it, you can use ssh-keygen to remove it:

1
2
3
4
$ ssh-keygen -f .ssh/known_hosts -R 192.168.1.104
# Host 192.168.1.104 found: line 10 type ECDSA
.ssh/known_hosts updated.
Original contents retained as .ssh/known_hosts.old

Now when you SSH the warning should be removed.

Setup a 3 Node Kubernetes Cluster on Ubuntu

Setup a 3 Node Kubernetes Cluster on Ubuntu 16.04

What is Kubernetes?

As referenced from their website:

  • “Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.”

Our Setup:

For this setup I will be using 3 AWS EC2 Instances with Ubuntu 16.04. One node will act as the master node, and the other 2 nodes, will act as nodes, previously named minions.

We will deploy Kubernetes on all 3 nodes, the master will be the node where we will initialize our cluster, deploy our weave network, applications and we will execute the join command on the worker nodes to join the master to form the cluster.

Deploy Kubernetes: Master

The following commands will be used to install Kubernetes, it will be executed with root permissions:

1
2
3
4
5
6
7
$ apt update && sudo apt upgrade -y
$ sudo apt install docker.io apt-transport-https -qy
$ sudo apt update
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo su -c 'echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/app' root
$ apt update
$ sudo apt install kubelet kubeadm kubernetes-cni -y

Now we would like to set up the master by initializing the cluster:

1
$ sudo kubeadm init --kubernetes-version stable-1.8

The output will provide you with instructions to setup the configurations for the master node, and provide you with a join token for your worker nodes, remember to make not of this token string, as we will need it later for our worker nodes. As your normal user, run the following to setup the config:

Remember to not run this as root, and as the normal user:

1
2
3
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Now we need to deploy a network for our pods:

1
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Lets confirm if all our resources are in its desired state, a small snippet of the output will look like the one below:

1
2
3
4
5
6
$ kubectl get all -n kube-system

...
NAME                                          READY     STATUS    RESTARTS   AGE
po/etcd-ip-172-31-40-211                      1/1       Running   0          6h
po/kube-apiserver-ip-172-31-40-211            1/1       Running   0          6h

Once all of the resources are in its desired state, we can head along to our worker nodes, to join them to the cluster

Deploy Kubernetes: Worker Nodes

As I have 2 worker nodes, we will need to deploy the following on both of our worker nodes, first to deploy Kubernetes on our nodes with root permission:

1
2
3
4
5
6
7
$ apt update && sudo apt upgrade -y
$ sudo apt install docker.io apt-transport-https -qy
$ sudo apt update
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo su -c 'echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/app' root
$ apt update
$ sudo apt install kubelet kubeadm kubernetes-cni -y

Once Kubernetes is installed, join the Master node by executing the join command:

1
$ sudo kubeadm join --token 49abf7.247d663db97f8504 172.31.40.211:6443 --discovery-token-ca-cert-hash sha256:3a3b301cfbac0995c69a0115989ea384230470d6836ae0e13e71dbdf15ffbb48

Do the 2 steps on the other node, then head back to the master node.

Verifying if All Nodes are Checked In

To verify if all nodes are available and reachable in the cluster:

1
2
3
4
5
$ kubectl get nodes
NAME               STATUS    ROLES     AGE       VERSION
ip-172-31-36-68    Ready     <none>    6h        v1.8.5
ip-172-31-40-211   Ready     master    6h        v1.8.5
ip-172-31-44-80    Ready     <none>    6h        v1.8.5

Deploy Services to Kubernetes:

Kubernetes has Awesome Examples on their Github Repository.

Since the awesomeness of OpenFaas, I will deploy OpenFaas on Kubernetes:

1
2
3
$ git clone https://github.com/openfaas/faas-netes
$ cd faas-netes
$ kubectl apply -f faas.yml,monitoring.yml,rbac.yml

Give it about a minute or so, then you should see the pods running in their desired state:

1
2
3
4
5
6
7
$ kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
alertmanager-77b4b476b-zxtcz   1/1       Running   0          4h
crypto-7d8b7f999c-7l85k        1/1       Running   0          1h
faas-netesd-64fb9b4dfb-hc8gh   1/1       Running   0          4h
gateway-69c9d949f-q57zh        1/1       Running   0          4h
prometheus-7fbfd8bfb8-d4cft    1/1       Running   0          4h

When we have the desired state, head over to the OpenFaas Gateway WebUI: http://master-public-ip:31112/ui/, select “Deploy New Function”, you can use your own function or select one from the store.

I am going to use Figlet from the store, once the pod has been deployed, select the function, enter any text into the request body and select invoke. I have used my name and surname, and turns out into:

1
2
3
4
5
6
 ____                      ____       _    _
|  _ \ _   _  __ _ _ __   | __ )  ___| | _| | _____ _ __
| |_) | | | |/ _` | '_ \  |  _ \ / _ \ |/ / |/ / _ \ '__|
|  _ <| |_| | (_| | | | | | |_) |  __/   <|   <  __/ |
|_| \_\\__,_|\__,_|_| |_| |____/ \___|_|\_\_|\_\___|_|

Resources:

Rejoining or Bootstrapping MySQL Galera Cluster Nodes After Shutdown

I have a 3 Node MySQL Galera Cluster that faced a shutdown on all 3 nodes at the same time, luckily this is only a testing environment, but at that time it was down and did not want to start up.

Issues Faced

When trying to start MySQL the only error visible was:

1
2
3
4
5
$ /etc/init.d/mysql restart
 * MySQL server PID file could not be found!
Starting MySQL
........ * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
 * Failed to restart server.

At that time I can see that the galera port is started, but not mysql:

1
2
3
4
5
6
7
8
$ ps aux | grep mysql
root     23580  0.0  0.0   4508  1800 pts/0    S    00:37   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/run/mysqld/mysqld.pid
mysql    24144  0.7 22.2 1185116 455660 pts/0  Sl   00:38   0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=long:string

$ netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:4567            0.0.0.0:*               LISTEN      25507/mysqld

Why?

More in detail is explained on a SeveralNines Blog Post, but due to the fact that all the nodes left the cluster, one of the nodes needs to be started as a referencing point, before the other nodes can rejoin or bootstrapped to the cluster.

Rejoining the Cluster

Consult the blog for more information, but from my end, I had a look at the node with the highest seqno and then updated safe_to_bootstrap to 1:

1
2
3
4
5
6
$ cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid:    e9f9cf6a-87a1-11e7-9fb4-52612b906897
seqno:   123512
safe_to_bootstrap: 1

Then made sure that no mysql processes are running, then did a bootstrap:

1
2
3
$ /etc/init.d/mysql bootstrap
Bootstrapping the cluster
Starting MySQL

Then restarted mysql on the other nodes.

Verifying

To verify that all your nodes has checked in, I have 3 nodes:

1
2
3
4
5
6
7
8
9
10
11
12
mysql> SHOW STATUS LIKE 'wsrep_%';
+------------------------------+---------------------------------------------------+
| Variable_name                | Value                                             |
+------------------------------+---------------------------------------------------+
| wsrep_local_recv_queue_avg   | 0.000000                                          |
| wsrep_local_state_comment    | Synced                                            |
| wsrep_incoming_addresses     | 10.3.132.91:3306,10.4.1.201:3306,10.4.113.21:3306 |
| wsrep_evs_state              | OPERATIONAL                                       |
| wsrep_cluster_size           | 3                                                 |
| wsrep_cluster_status         | Primary                                           |
| wsrep_connected              | ON                                                |
+------------------------------+---------------------------------------------------+

or a shorter version:

1
2
3
4
5
6
mysql> SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';
+------------------------------+---------------------------------------------------+
| Variable_name                | Value                                             |
+------------------------------+---------------------------------------------------+
| wsrep_cluster_size           | 3                                                 |
+------------------------------+---------------------------------------------------+