Ruan Bekker's Blog

From a Curious mind to Posts on Github

Building Python Serverless Slack Apps on OpenFaas

If you are not familliar with OpenFaas, it’s definitely time that you should have a look at it, plus, they are doing some pretty awesome work!

From their documentation: “OpenFaaS (Functions as a Service) is a framework for building serverless functions with Docker and Kubernetes which has first class support for metrics. Any process can be packaged as a function enabling you to consume a range of web events without repetitive boiler-plate coding.”

Make sure to give them a visit at openfaas.com and while you are there, in the world of serverless, have a look at how Alex outlines architecture and patterns he applies in a real-world example, absolutely great read!

What are we doing today?

Today we will build a slack app using python which we will deploy as a function on OpenFaas!

Our slash command will make a request to our slack-request function, which will respond with a json string, which will then be parsed in a slack attachment message, then based on your button decision, it will then invoke our slack-interaction function, which will then respond with another message that will allow you to follow the embedded link.

The slack messages are really basic, but you can create a awesome workflow using slack apps. And the best of all, its running on OpenFaas!

Deploying OpenFaas

Docker Swarm and Kubernetes are supported, but since I am using Docker Swarm at the moment of writing, this tutorial will show how to deploy OpenFaas to your cluster. Have a look at OpenFaas Documentation for more detailed information.

Installing OpenFaas CLI for Mac:

1
$ brew install faas-cli

Deploy the OpenFaas Stack:

1
2
3
$ git clone https://github.com/openfaas/faas
$ cd faas
$ ./deploy_stack.sh

Credentials: The default configuration will create credentials for you and returns instructions on how to authorize faas-cli, for demonstration it will look more or less like the following:

1
$ echo -n <some_hash_secret> | faas-cli login --username=admin --password-stdin

The UI will be available at: http://127.0.0.1:8080. For this demonstration we will only use the cli.

Create the Functions

I will create 2 python functions:

  • The slack-request function, which will be associated to the slash command
  • The slack-interactive function, which will be used for interactivity

Create a home directory for your functions and create 2 functions:

1
2
3
$ mkdir -p ~/functions && cd ~/functions
$ faas-cli new --lang python slack-request
$ faas-cli new --lang python slack-interactive

Read the documentation if you’d like to learn more.

Configure the first function:

1
$ vim slack-request/handler.py

And our function code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
import json

def handle(req):
    data = {
        "text": "Serverless Message",
        "attachments": [{
            "title": "The Awesome world of Serverless introduces: OpenFaas!",
            "fields": [{
                "title": "Amazing Level",
                "value": "10",
                "short": True
            },
      {
                "title": "Github Stars",
                "value": "15k +",
                "short": True
            }],
            "author_name": "OpenFaas",
            "author_icon": "",
            "image_url": "https://blog.alexellis.io/content/images/2017/08/small.png"
        },
        {
            "title": "About OpenFaas",
            "text": "OpenFaaS is a framework for packaging code, binaries or containers as Serverless functions on any platform."
        },
        {
            "fallback": "Would you recommend OpenFaas to your friends?",
            "title": "Would you recommend OpenFaas to your friends?",
            "callback_id": "response123",
            "color": "#3AA3E3",
            "attachment_type": "default",
            "actions": [
                {
                    "name": "recommend",
                    "text": "Ofcourse!",
                    "type": "button",
                    "value": "recommend"
                },
                {
                    "name": "definitely",
                    "text": "Most Definitely!",
                    "type": "button",
                    "value": "definitely"
                }
            ]
        }]
    }
    return json.dumps(data)

Since our response needs to be parsed as json, we need to set the content type for our environment in our yaml configuration. Read more on it here. Edit the slack-request.yml :

1
2
3
4
5
6
7
8
9
10
provider:
  name: faas
  gateway: http://<your.gw.address>:8080
functions:
  slack-request:
    lang: python
    handler: ./slack-request
    image: <your-repo>/slack-request:latest
    environment:
      content_type: application/json

Now we need to build our image, push it to our repository like dockerhub, then deploy to openfaas:

1
2
3
4
5
6
7
$ faas-cli build -f ./slack-request.yml
$ faas-cli push -f ./slack-request.yml
$ faas-cli deploy -f ./slack-request.yml
Deploying: slack-request.

Deployed. 202 Accepted.
URL: http://your.gw.address:8080/function/slack-interactive

Configure the slack-interactive function:

1
$ vim slack-interactive/handler.py

Note that whenever your interact with the first message, a post request will be made against the interactivity request url, you will notice that I decoded the payload (but not doing anything with it), where you will find the callback_id, request_url etc. But for simplicity, I am just using a static json message to respond. Our function code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import json
import urllib

def handle(req):
    urlstring = urllib.unquote(req).decode('utf8').strip('payload=')
    response = json.loads(urlstring)
    data = {
        "attachments": [
            {
                "replace_original": True,
                "response_type": "ephemeral",
                "fallback": "Required plain-text summary of the attachment.",
                "color": "#36a64f",
                "pretext": "Ahh yeah! Great choice, OpenFaas is absolutely brilliant!",
                "author_name": "",
                "author_link": "https://github.com/openfaas/faas",
                "author_icon": "http://flickr.com/icons/bobby.jpg",
                "title": "OpenFaas",
                "title_link": "https://github.com/openfaas/faas",
                "text": "Head over to OpenFaas",
                "image_url": "https://avatars2.githubusercontent.com/u/27013154?s=400&v=4",
                "thumb_url": "https://github.com/openfaas/faas",
                "footer": "Slack Apps built on OpenFaas",
                "footer_icon": "https://a.slack-edge.com/45901/marketing/img/_rebrand/meta/slack_hash_256.png",
                "ts": 123456789
            }
        ]
    }
    return json.dumps(data)

We also need to set the content type to json:

1
2
3
4
5
6
7
8
9
10
provider:
  name: faas
  gateway: http://<your.gw.address>:8080
functions:
  slack-interactive:
    lang: python
    handler: ./slack-interactive
    image: <repo>/slack-interactive:latest
    environment:
      content_type: application/json

Build, deploy and ship:

1
2
3
4
5
6
7
8
$ faas-cli build -f ./slack-interactive.yml
$ faas-cli push -f ./slack-interactive.yml
$ faas-cli deploy -f ./slack-interactive.yml

Deploying: slack-interactive.

Deployed. 202 Accepted.
URL: http://<your.gw.address>:8080/function/slack-interactive

When your functions are deployed, go ahead and create the slack app.

Create the Slack App

  • Head over to https://api.slack.com/apps and create a new app
  • Create a incoming webhook
  • Head over to slash commands and create a new command, in my case it was /supersam, set the request url to the public endpoint of your function: http://pub-ip:8080/function/slack-request
  • Head over to interactive components, set the request url for the interactivity: http://pub-ip:8080/function/slack-interactive
  • If you dont have a public routable address, have a look at ngrok

Once you are set, you should be able to see the slash command integration in your slack workspace, head over to slacks documentation if you run into any trouble.

Test your Slack App

Now that everything is good to go, its time to test your slack app running on OpenFaas!

Head over to slack and run your command /<your-slack-slash-command>. You should see this output:

When you select one of the buttons, you will get a new message:

This is a real basic example of slack apps, but slack apps are really powerful. You can for example create a slack app that deploys ephemeral environments on swarm, or create change management approval workflows etc.

I hope this was informative, I am really enjoying OpenFaas at the moment and if your have not tested it, I encourage you to try it out, its really, really amazing!

Parallel Processing on AWS Lambda With Python Using Multiprocessing

If you are trying to use multiprocessing.Queue or multiprocessing.Pool on AWS Lambda, you are probably getting the exception:

1
2
3
4
[Errno 38] Function not implemented: OSError

    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

The reason for that is due to the Lambda execution environment not having support on shared memory for processes, therefore you can’t use multiprocessing.Queue or multiprocessing.Pool.

As a workaround, Lambda does support the usage of multiprocessing.Pipe instead of Queue.

Parallel Processing on Lambda Example

Below is a very basic example on how you would achieve the task of executing parallel processing on AWS Lambda for Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import time
import multiprocessing

region_maps = {
        "eu-west-1": {
            "dynamodb":"dynamodb.eu-west-1.amazonaws.com"
        },
        "us-east-1": {
            "dynamodb":"dynamodb.us-east-1.amazonaws.com"
        },
        "us-east-2": {
            "dynamodb": "dynamodb.us-east-2.amazonaws.com"
        }
    }

def multiprocessing_func(region):
    time.sleep(1)
    endpoint = region_maps[region]['dynamodb']
    print('endpoint for {} is {}'.format(region, endpoint))

def lambda_handler(event, context):
    starttime = time.time()
    processes = []
    regions = ['us-east-1', 'us-east-2', 'eu-west-1']
    for region in regions:
        p = multiprocessing.Process(target=multiprocessing_func, args=(region,))
        processes.append(p)
        p.start()

    for process in processes:
        process.join()

    output = 'That took {} seconds'.format(time.time() - starttime)
    print(output)
    return output

The output when the function gets invoked:

1
2
3
4
pid: 30913 - endpoint for us-east-1 is dynamodb.us-east-1.amazonaws.com
pid: 30914 - endpoint for us-east-2 is dynamodb.us-east-2.amazonaws.com
pid: 30915 - endpoint for eu-west-1 is dynamodb.eu-west-1.amazonaws.com
That took 1.014902114868164 seconds

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


Sharing Global Variables in Python Using Multiprocessing

While I was using multiprocessing, I found out that global variables are not shared between processes.

Example of the Issue

Let me first provide an example of the issue that I was facing.

I have 2 input lists, which 2 processes wil read from and append them to the final list and print the aggregated list to stdout

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import multiprocessing
final_list = []

input_list_one = ['one', 'two', 'three', 'four', 'five']
input_list_two = ['six', 'seven', 'eight', 'nine', 'ten']

def worker(data):
    for item in data:
        final_list.append(item)

process1 = multiprocessing.Process(target=worker, args=[final_list_one])
process2 = multiprocessing.Process(target=worker, args=[final_list_two])

process1.start()
process2.start()
process1.join()
process2.join()

print(final_list)

When running the example:

1
2
$ python3 mp_list_issue.py
[]

As you can see the response from the list is still empty.

Resolution

We need to use multiprocessing.Manager.List.

From Python’s Documentation:

The multiprocessing.Manager returns a started SyncManager object which can be used for sharing objects between processes. The returned manager object corresponds to a spawned child process and has methods which will create shared objects and return corresponding proxies.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import multiprocessing
manager = multiprocessing.Manager()
final_list = manager.list()

input_list_one = ['one', 'two', 'three', 'four', 'five']
input_list_two = ['six', 'seven', 'eight', 'nine', 'ten']

def worker(data):
    for item in data:
        final_list.append(item)

process1 = multiprocessing.Process(target=worker, args=[final_list_one])
process2 = multiprocessing.Process(target=worker, args=[final_list_two])

process1.start()
process2.start()
process1.join()
process2.join()

print(final_list)

Now when we run our script, we can see that our processes are aware of our defined list:

1
2
$ python3 mp_list.py
['one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten']

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


Parallel Processing With Python and Multiprocessing Using Queue

Today I had the requirement to achieve a task by using parallel processing in order to save time.

The task to be achieved

For this demonstration, I have a list of people and each task needs to lookup its pet name and return to stdout. I want to spawn a task for each persons pet name lookup and run the tasks in parallel so that all the results can be returned back at once, instead of sequential.

This is a basic task, but you could have a CPU intensive job, where it will shine better.

Multiprocesing Queues

When using multiple processes, one generally uses message passing for communication between processes and avoids having to use any synchronization primitives like locks.

The Queue type is a multi producer, multi consumer FIFO queues modelled on the queue.Queue class in the standard library. You can read more up on it here

Our Workflow

Our multiprocessing workflow will look like this:

  • We will define our data, which will be a dictionary of people and their pet names
  • We will define an output queue
  • Create a example function that will produce each task to the queue
  • Then we will setup a lost of processes that we want to run
  • From the list of processes that we defined, we will run each process, then wait and exit the completed processes
  • We will then consume from the queue. For each process in our processes list

Note that I also added a delay of 2 seconds, so that you can see that the tasks are run in parallel, so the delay will only be 2 seconds.

Our code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import multiprocessing as mp
import random
import string
import time

pet_maps = {
        "adam": {"pet_name": "max"},
        "steve": {"pet_name": "sylvester"},
        "michelle": {"pet_name": "fuzzy"},
        "frank": {"pet_name": "pete"},
        "will": {"pet_name": "cat"},
        "natasha": {"pet_name": "tweety"},
        "samantha": {"pet_name": "bob"},
        "peter": {"pet_name": "garfield"},
        "susan": {"pet_name": "zazu"},
        "josh": {"pet_name": "tom"},
    }

pet_owners = pet_maps.keys()

output = mp.Queue()

def get_pet_name(data, output):
    time.sleep(2)
    print('adding to queue')
    response = 'pet name: {}'.format(data)
    output.put(response)

processes = [mp.Process(target=get_pet_name, args=(pet_maps[name]['pet_name'], output)) for name in pet_owners]

for p in processes:
    p.start()

for p in processes:
    p.join()

print('consuming from queue:')
results = [output.get() for p in processes]
print(results)

Running the example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ python3 mp.py
adding to queue
adding to queue
adding to queue
adding to queue
adding to queue
adding to queue
adding to queue
adding to queue
adding to queue
adding to queue

consuming from queue:
['pet name: max', 'pet name: sylvester', 'pet name: fuzzy', 'pet name: pete', 'pet name: cat', 'pet name: tweety', 'pet name: garfield', 'pet name: bob', 'pet name: zazu', 'pet name: tom']

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


Concourse Pipeline With Resources Tutorial

In Concourse, Resources refer to external resource types such as s3, github etc.

So for example, we can run a pipeline which pulls data from github, such as cloning a repository, so in other words the data that was cloned from the github repository is within the container where your tasks will be executed.

Concourse Github Resourse Example

In this tutorial we will use the github resource type, in conjunction with a task that will execute a script, where the script will be inside the github repository.

Our pipeline as pipeline.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
resources:
- name: concourse-tutorial
  type: git
  source:
    uri: https://github.com/ruanbekker/concourse-tutorial.git
    branch: master

jobs:
- name: job-hello-world
  public: true
  plan:
  - get: concourse-tutorial
  - task: hello-world
    file: concourse-tutorial/00-basic-tasks/task_hello_world.yml

You can head over to hello-world task on github to see the task, but all it does is running a uname -a

So our job has a task that will call the action defined in our task_hello_world.yml which retrieves it from the get step, as you can see it’s the concourse-tutorial resource, which is defined under the resources section as a git resource type.

Set the pipeline:

1
2
3
4
$ fly -t ci sp -c pipeline.yml -p 04-hello-world

apply configuration? [yN]: y
pipeline created!

Unpause the pipeline:

1
2
$ fly -t ci up -p 04-hello-world
unpaused '04-hello-world'

Trigger the job (trigger is off; default)

1
2
3
4
5
6
7
$ fly -t ci tj -j 04-hello-world/job-hello-world --watch
started 04-hello-world/job-hello-world #4

initializing
running uname -a
Linux 6a91b808-c488-4e3c-7b51-404f73405c31 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 GNU/Linux
succeeded

So this job cloned the github repository, called the task file which calls the bash script from th github repository to run uname -a

For my other content on concourse, have a look at the concourse category.

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


How to Cache Data With Python Flask

If you depending on a external source to return static data you can implement cachetools to cache data from preventing the overhead to make the request everytime you make a request to Flask.

This is useful when your upstream data does not change often. This is configurable with maxsize and ttl so whenever the first one’s threshold is met, the application will fetch new data whenever the request has been made to your application.

Example

Let’s build a basic flask application that will return the data from our data.txt file to the client:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from flask import Flask
from cachetools import cached, TTLCache

app = Flask(__name__)
cache = TTLCache(maxsize=100, ttl=60)

@cached(cache)
def read_data():
    data = open('data.txt', 'r').read()
    return data

@app.route('/')
def main():
    get_data = read_data()
    return get_data

if __name__ == '__main__':
    app.run()

Create the local file with some data:

1
2
$ touch data.txt
$ echo "version1" > data.txt

Start the server:

1
$ python app.py

Make the request:

1
2
$ curl http://localhost:5000/
version1

Change the data inside the file:

1
$ echo "version2" > data.txt

Make the request again:

1
2
$ curl http://localhost:5000/
version1

As the ttl is set to 60, wait for 60 seconds so that the item kan expire from the cache and try again:

1
2
$ curl http://localhost:5000/
version2

As you can see the cache expired and a new request has been made to read the file again and load it in cache, and then return to the client.

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


Concourse Tasks and Inputs Tutorial

In this tutorial I will show you how to execute task scripts and using task inputs to have the ability to pass data to concourse for processing.

For my other content on concourse, have a look at the concourse category.

Task Inputs

First, let’s run a task on concourse that does not rely on any inputs.

no_inputs.yml
1
2
3
4
5
6
7
8
9
platform: linux

image_resource:
  type: docker-image
  source: {repository: busybox}

run:
  path: uname
args: ["-a"]

Running execute with the configuration from above:

1
2
3
4
5
6
7
$ fly -t ci e -c no_inputs.yml

executing build 37 at http://10.20.30.40/builds/37
initializing
running uname -a
Linux 2fd4e261-a708-4e15-4a4a-2bc50221a664 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 GNU/Linux
succeeded

As you can see we have executed the command uname -a on one of the containers in Concourse.

Tasks Inputs: Specify Path

Now lets say, we have data that needs to be transferred to the container where we are running our tasks. For that we are using inputs.

In this example, we will set the input parameter in our task definition, and override the path with the cli. We will create a couple of files in a folder, then list them in the container where the task is running.

Creating the data:

1
2
$ mkdir my-data-folder
$ touch my-data-folder/test1.txt my-data-folder/test2.txt

Our task definition:

inputs_required.yml
1
2
3
4
5
6
7
8
9
10
11
12
platform: linux

image_resource:
  type: docker-image
  source: {repository: busybox}

inputs:
- name: my-input

run:
  path: ls
  args: ['-alR']

As you can see our input name is called my-input and we will use the cli to map the local folder my-data-folder to the parameter name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ fly -t ci e -c inputs_required.yml -i my-input=./my-data-folder/

executing build 32 at http://10.20.30.40/builds/32
initializing
my-input: 262.13 KiB/s 0s
running ls -alR
.:
total 0
drwxr-xr-x    1 root     root            16 Feb 10 08:53 .
drwxr-xr-x    1 root     root            16 Feb 10 08:53 ..
drwxr-xr-x    1 root     root            36 Feb 10 08:53 my-input

./my-input:
total 0
drwxr-xr-x    1 root     root            36 Feb 10 08:53 .
drwxr-xr-x    1 root     root            16 Feb 10 08:53 ..
-rw-r--r--    1 501      staff            0 Feb 10 08:52 test1.txt
-rw-r--r--    1 501      staff            0 Feb 10 08:52 test2.txt
succeeded

As you can see from the above output, the folder was uploaded and placed inside the container where we ran our task.

Task Inputs: Parent Directory

Then we can use parent directories. Running a task that relies on the input path which will be our current working directory. Note: the input name should be the same as the current working directory

The input name will be the only thing that differs, which will look like:

1
2
inputs:
- name: my-data-folder

Running the task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ cd my-data-folder
$ fly -t ci e -c ../input_parent_dir.yml

executing build 35 at http://10.20.30.40/builds/35
initializing
my-data-folder: 395.85 KiB/s 0s
running ls -alR
.:
total 0
drwxr-xr-x    1 root     root            38 Feb 10 09:17 .
drwxr-xr-x    1 root     root            16 Feb 10 09:17 ..
drwxr-xr-x    1 root     root            18 Feb 10 09:17 my-data-folder

./my-data-folder:
total 0
drwxr-xr-x    1 root     root            18 Feb 10 09:17 .
drwxr-xr-x    1 root     root            38 Feb 10 09:17 ..
-rw-r--r--    1 501      staff            0 Feb 10 09:15 test1.txt
-rw-r--r--    1 501      staff            0 Feb 10 09:15 test2.txt
succeeded

The source code for this can be found at https://github.com/ruanbekker/concourse-tutorial/tree/master/02-task-inputs

Task Scripts:

In conjunction with inputs, we can let our task configuration reference a script that we want to execute, and using inputs, we can upload the current working directory to concourse, so then the container has context about the data that it needs.

Our task configuration task_show_hostname.yml

1
2
3
4
5
6
7
8
9
10
11
platform: linux

image_resource:
  type: docker-image
  source: {repository: busybox}

inputs:
- name: 03-task-scripts

run:
  path: ./03-task-scripts/task_show_hostname.sh

Our executable script 03-task-scripts/task_show_hostname.sh

1
2
3
4
#!/bin/sh

get_hostname=$(hostname)
echo "Hostname is: ${get_hostname}"

Make sure to apply the executable permissions to the script:

1
$ chmod +x 03-task-scripts/task_show_hostname.sh

With this configuration, it uploads the current working directory to concourse, and the data inside the directory gets placed on the container’s working directory: 03-task-scripts, which is the name of the input.

1
2
3
4
5
6
7
8
$ fly -t ci e -c 03-task-scripts/task_show_hostname.yml

executing build 39 at http://10.20.30.40/builds/39
initializing
03-task-scripts: 347.15 KiB/s 0s
running ./03-task-scripts/task_show_hostname.sh
Hostname is: 3ccb3c28-d452-4068-5ea1-101153803d93
succeeded

The source code for this example can be found at https://github.com/ruanbekker/concourse-tutorial/tree/master/03-task-scripts

That’s it for Task Inputs and Task Scripts on Concourse, please feel free to have a look at my other content about Concourse

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


Sysadmin Linux Troubleshooting Cheatsheet

This is a one pager of all the commands I use when I have to troubleshoot problems. This post will be updated as time goes by.

Curl / Web Response Times

Template file:

1
2
3
4
5
6
7
8
9
$ cat curl-format.txt
time_namelookup:  %{time_namelookup}\n
time_connect:  %{time_connect}\n
time_appconnect:  %{time_appconnect}\n
time_pretransfer:  %{time_pretransfer}\n
time_redirect:  %{time_redirect}\n
time_starttransfer:  %{time_starttransfer}\n
----------\n
time_total:  %{time_total}\n

The host header, source addres, destination address:

1
2
3
4
5
6
7
8
9
10
$ curl -sk -w "@curl-format.txt" -o /dev/null -H "Host: remote-host.mydomain.com" 10.0.2.10 https://10.244.0.240:443 -L

time_namelookup:  0.012178
time_connect:  0.012225
time_appconnect:  0.062149
time_pretransfer:  0.062175
time_redirect:  0.000172
time_starttransfer:  0.125631
----------
time_total:  0.125849

MTR / Network Latencies / Packetloss

No dns, TCP, counts, port, source address, destination address:

1
2
3
4
5
6
$ mtr -n -T -c 10 --port 443 10.2.0.2 10.244.10.5 --report
Start: Sun Feb 10 19:04:50 2019
HOST: my-internet-gatewewy         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 172.18.110.22              0.0%    10    0.3   0.3   0.3   0.3   0.0
  2.|-- 172.18.110.22              0.0%    10    0.3   0.3   0.3   0.3   0.0
  3.|-- 172.18.110.22              0.0%    10    0.3   0.3   0.3   0.3   0.0

TCPTraceroute

No dns, TCP, port, source address, destination address:

1
2
3
4
$ traceroute -T -n -p 443 -s 10.80.4.7 10.2.129.4; done
traceroute to 10.2.129.4 (10.2.129.4), 30 hops max, 60 byte packets
 1  10.80.4.1   0.322 ms  0.291 ms  0.224 ms
 2  10.2.129.4  179.090 ms  179.022 ms  179.023 ms

Connection Related:

Connection flow: Thanks to

1
2
3
4
Consider two programs attempting a socket connection (call them a and b). Both set up sockets and transition to the LISTEN state. Then one program (say a) tries to connect to the other (b). a sends a request and enters the SYN_SENT state, and b receives the request and enters the SYN_RECV state. When b acknowledges the request, they enter the ESTABLISHED state, and do their business. Now a couple of things can happen:

    a wishes to close the connection, and enters FIN_WAIT1. b receives the FIN request, sends an ACK (then a enters FIN_WAIT2), enters CLOSE_WAIT, tells a it is closing down and the enters LAST_ACK. Once a acknowledges this (and enters TIME_WAIT), b enters CLOSE. a waits a bit to see if anythings is left, then enters CLOSE.
    a and b have finished their business and decide to close the connection (simultaneous closing). When a is in FIN_WAIT, and instead of receiving an ACK from b, it receives a FIN (as b wishes to close it as well), a enters CLOSING. But there are still some messages to send (the ACK that a is supposed to get for its original FIN), and once this ACK arrives, a enters TIME_WAIT as usual.

Active Connections:

1
$ netstat -n -A  inet | grep -v "127.0.0.1"

Established Connections:

1
2
$ netstat -nputw | grep ESTABLISHED
$ netstat -antp | grep :3306 | grep ESTABLISHED

Time Wait Connections:

1
$ netstat -antp | grep TIME_WAIT

How many connections:

1
$ wc -l /proc/net/tcp

Listing Open files per Port:

1
$ lsof -i:3306

Listing Open files per User:

1
$ lsof -u glassfish

Resources

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


Port Status Checker Script in C Language

This is a simple script in the C Programming Language to test the port status of a remote address.

Requirements:

You will need the gcc package to compile the program:

For RHEL based distro’s:

1
$ yum install gcc -y

For Debian based distro’s:

1
$ apt install gcc -y

Check TCP Port Status in C Language:

Our file: test.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>


int main(int argc, char *argv[]) {

    int portno     = 443;
    char *hostname = "google.com";

    int sockfd;
    struct sockaddr_in serv_addr;
    struct hostent *server;

    sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd < 0) {
        error("Error opening socket\n");
    }

    server = gethostbyname(hostname);

    if (server == NULL) {
        fprintf(stderr,"ERROR, no such host\n");
        exit(0);
    }

    bzero((char *) &serv_addr, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    bcopy((char *)server->h_addr,
         (char *)&serv_addr.sin_addr.s_addr,
         server->h_length);

    serv_addr.sin_port = htons(portno);
    if (connect(sockfd,(struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0) {
        printf("Port is Closed\n");
    } else {
        printf("Port is Open\n");
    }

    close(sockfd);
    return 0;
}

Compile:

Compile using gcc:

1
$ gcc -o test test.c

Execute:

Execute the script:

1
2
$ ./test
Port is Open

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.


The AWS CLI Cheatsheet for Bash

This is a post for all the AWS CLI oneliners that I stumble upon. Note that they will be updated over time.

RDS

Describe All RDS DB Instances:

1
$ aws --profile prod rds describe-db-instances --query 'DBInstances[*].[DBInstanceArn,DBInstanceIdentifier,DBInstanceClass,Endpoint]'

Describe a RDS DB Instance with a dbname:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ aws --profile prod rds describe-db-instances --query 'DBInstances[?DBInstanceIdentifier==`db-staging`].[DBInstanceArn,DBInstanceIdentifier,DBInstanceClass,Endpoint]'
[
    [
        "arn:aws:rds:eu-west-1:<customer_id>:db:db-staging",
        "db-staging",
        "db.t2.micro",
        {
            "HostedZoneId": "ASKDJSAKDJBA",
            "Port": 5432,
            "Address": "db-staging.asdkjahsd.eu-west-1.rds.amazonaws.com"
        }
    ]
]

List all RDS DB Instances and limit output:

1
2
3
4
5
6
7
8
9
10
11
12
$ aws --profile prod rds describe-db-instances --query 'DBInstances[*].[DBInstanceArn,DBInstanceIdentifier,DBInstanceClass,Endpoint]'
[
    [
        "arn:aws:rds:eu-west-1:<customer_id>:db:db-name",
        "db-name",
        "db.t2.micro",
        {
            "HostedZoneId": "ABCDEFGHILKL",
            "Port": 5432,
            "Address": "db-name.abcdefg.eu-west-1.rds.amazonaws.com"
        }
    ],

List all RDS DB Instances that has backups enabled, and limit output:

1
2
3
4
5
6
7
8
9
10
11
12
$ aws --profile prod rds describe-db-instances --query 'DBInstances[?BackupRetentionPeriod>`0`].[DBInstanceArn,DBInstanceIdentifier,DBInstanceClass,Endpoint]'
[
    [
        "arn:aws:rds:eu-west-1:<customer_id>:db:db-name",
        "db-name",
        "db.t2.micro",
        {
            "HostedZoneId": "ABCDEFGHILKL",
            "Port": 5432,
            "Address": "db-name.abcdefg.eu-west-1.rds.amazonaws.com"
        }
    ],

Describe DB Snapshots for DB Instance Name:

1
2
3
4
5
6
7
8
$ aws --profile prod rds describe-db-snapshots --db-instance-identifier db --query 'DBSnapshots[?DBInstanceIdentifier==`db`].[DBInstanceIdentifier,DBSnapshotIdentifier,SnapshotCreateTime,Status]'
[
    [
        "db",
        "rds:db-2018-05-16-04-08",
        "2018-05-16T04:08:53.696Z",
        "available"
    ],

Events for the last 24 Hours:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ aws --profile prod rds describe-events --source-identifier "rds:db-2018-05-16-04-08" --source-type db-snapshot --duration 1440 --query 'Events[*]'
[
    {
        "EventCategories": [
            "creation"
        ],
        "SourceType": "db-snapshot",
        "SourceArn": "arn:aws:rds:eu-west-1:<customer_id>:snapshot:rds:db-2018-05-16-04-08",
        "Date": "2018-05-16T04:08:40.264Z",
        "Message": "Creating automated snapshot",
        "SourceIdentifier": "rds:db-2018-05-16-04-08"
    },
    {
        "EventCategories": [
            "creation"
        ],
        "SourceType": "db-snapshot",
        "SourceArn": "arn:aws:rds:eu-west-1:<customer_id>:snapshot:rds:db-2018-05-16-04-08",
        "Date": "2018-05-16T04:32:04.047Z",
        "Message": "Automated snapshot created",
        "SourceIdentifier": "rds:db-2018-05-16-04-08"
    }
]

List Public RDS Instances:

1
2
3
4
5
6
7
8
$ aws --profile prod rds describe-db-instances --query 'DBInstances[?PubliclyAccessible==`true`].[DBInstanceIdentifier,Endpoint.Address]'

[
  [
    "name",
    "name.abcdef.eu-west-1.rds.amazonaws.com"
  ]
]

SSM Parameter Store:

List all parameters by path:

1
2
3
4
$ aws --profile prod ssm get-parameters-by-path --path '/service-a/team-a/my-app-name/' | jq '.Parameters[]' | jq -r '.Name'
/service-a/team-a/my-app-name/db_hostname
/service-a/team-a/my-app-name/db_username
/service-a/team-a/my-app-name/db_password

Get a value from a parameter:

1
2
$ aws --profile prod ssm get-parameters --names '/service-a/team-a/my-app-name/db_username' --with-decryption | jq '.Parameters[]' | jq -r '.Value'
my_db_user

Thank You

Please feel free to show support by, sharing this post, making a donation, subscribing or reach out to me if you want me to demo and write up on any specific tech topic.