As the curious person that I am, I like to play around with new stuff that I stumble upon, and one of them was having a docker swarm cluster running on 3 Raspberry Pi’s on my LAN.
The idea is to have 3 Raspberry Pi’s (Model 3 B), a Manager Node, and 2 Worker Nodes, each with a 32 GB SanDisk SD Card, which I will also be part of a 3x Replicated GlusterFS Volume that will come in handy later for some data that needs persistent data.
If you have an internal DNS Server, set an A Record for each node, or for simplicity, set your hosts file on each node so that your hostname for each node responds to it’s provisioned IP Address:
Time to set up our swarm. As we have more than one network interface, we will need to setup our swarm by specifying the IP Address of our network interface that is accessible from our LAN:
Now that we have our IP Address, initialize the swarm on the manager node:
12345678910
pi@rpi-01:~ $ docker swarm init --advertise-addr 192.168.0.2
Swarm initialized: current node (siqyf3yricsvjkzvej00a9b8h) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \ --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 \ 192.168.0.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
Then from rpi-02 join the manager node of the swarm:
12
pi@rpi-02:~ $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 192.168.0.2:2377
This node joined a swarm as a worker.
Then from rpi-03 join the manager node of the swarm:
12
pi@rpi-03:~ $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 192.168.0.2:2377
This node joined a swarm as a worker.
Then from the manager node: rpi-01, ensure that the nodes are checked in:
12345
pi@rpi-01:~ $ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
62s7gx1xdm2e3gp5qoca2ru0d rpi-03 Ready Active
6fhyfy9yt761ar9pl84dkxck3 * rpi-01 Ready Active Leader
pg0nyy9l27mtfc13qnv9kywe7 rpi-02 Ready Active
Setting Up a Replicated GlusterFS Volume
I have decided to setup a replicated glusterfs volume to have data replicated throughout the cluster if I would like to have some persistent data. From each node, install the GlusterFS Client and Server:
Then your GlusterFS Volume will be mounted on all the nodes, and when a file is written to the /mnt/ partition, data will be replicated to all the nodes in the Cluster:
Docker Swarm’s Routing mesh takes care of the internal routing, so requests will respond even if the container is not running on the node that you are making the request against.
With that said, verifying on which node our service is running:
123
pi@rpi-01:~ $ docker service ps web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
sd67cd18s5m0 web.1 hypriot/rpi-busybox-httpd:latest rpi-02 Running Running 2 minutes ago
When we make a HTTP Request to one of these Nodes IP Addresses, our request will be responded with this awesome static page:
We can see we only have one container in our swarm, let’s scale that up to 3 containers:
12
pi@rpi-01:~ $ docker service scale web01=3
web01 scaled to 3
Now that the service is scaled to 3 containers, requests will be handled using the round-robin algorithm. To ensured that the service scaled, we will see that we will have 3 replicas:
123
pi@rpi-01:~ $ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
vsvyanuw6q6y web replicated 3/3 hypriot/rpi-busybox-httpd:latest *:891->80/tcp
Verifying where these containers are running on:
12345
pi@rpi-01:~ $ docker service ps web01
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
sd67cd18s5m0 web.1 hypriot/rpi-busybox-httpd:latest rpi-02 Running Running 2 minutes ago
ope3ya7hh9j4 web.2 hypriot/rpi-busybox-httpd:latest rpi-03 Running Running 30 seconds ago
07m1ww7ptxro web.3 hypriot/rpi-busybox-httpd:latest rpi-01 Running Running 28 seconds ago
Lastly, removing the service from our swarm:
12
pi@rpi-01:~ $ docker service rm web01
web01
Massive Thanks:
a Massive thanks to Alex Ellis for mentioning me on one of his blogposts:
The last couple of days I picked up on my ELK Stack a couple thousands of SSH Brute Force Attacks, so I decided I will just revisit my SSH Server configuration, and change my SSH Port to something else for the interim. The dashboard that showed me the results at that point in time:
Then I decided I actually would like to setup a SSH Honeypot to listen on Port 22 and change my SSH Server to listen on 222 and capture their IP Addresses, Usernames and Passwords that they are trying to use and dump it all in a file so that I can build up my own password dictionary :D
SSH Configuration:
Changing the SSH Port:
1
$ sudo vim /etc/ssh/sshd_config
Change the port to 222:
1
Port 222
Restart the SSH Server:
1
$ sudo /etc/init.d/ssh restart
Verify that the SSH Server is running on the new port:
I’m working on capturing some data that I want to use for analytics, and a big part of that is capturing the query string parameters that is in the request URL.
So essentially I would like to break the data up into key value pairs, using Python and the urllib module, which will then pushed into a database like MongoDB or DynamoDB.
Our URL:
So the URL’s that we will have, will more or less look like the following:
So we have documents ingested into Elasticsearch, and one of the fields has a IP Address, but at this moment it’s just an IP Address, the goal is to have more information from this IP Address, so that we can use Kibana’s Coordinate Maps to map our data on a Geographical Map.
In order to do this we need to make use of the GeoIP Ingest Processor Plugin, which adds information about the grographical location of the IP Address that it receives. This information is retrieved from the Maxmind Datases.
So when we pass our IP Address through the processor, for example one of Github’s IP Addresses: 192.30.253.113 we will in return get:
While testing DynamoDB for a specific use case I picked up at times where a GetItem will incur about 150ms in RequestLatency on the Max Statistic. This made me want to understand the behavior that I’m observing.
I will go through my steps drilling down on pointers where latency can be reduced.
DynamoDB Performance Testing Overview
Tests:
Create 2 Tables with 10 WCU / 10 RCU, one encrypted, one non-encrypted
Your database is getting hammered with requests and building up some load over time and we would like to place a caching layer in front of our database that will return data from the caching layer, to reduce some traffic to our database and also improve our performance for our application.
The Scenario:
Our scenario will be very simple for this demonstration:
Database will be using SQLite with product information (product_name, product_description)
Caching Layer will be Memcached
Our Client will be written in Python, which checks if the product name is in cache, if not a GET_MISS will be returned, then the data will be fetched from the database, returns it to the client and save it to the cache
Next time the item will be read, a GET_HIT will be received, then the item will be delivered to the client directly from the cache
SQL Database:
As mentioned we will be using sqlite for demonstration.
Create the table, populate some very basic data:
12345678
$sqlite3db.sql-header-columnimportsqlite3assqlSQLiteversion3.16.02016-11-0419:09:39Enter".help"forusagehints.sqlite>createtableproducts(product_nameSTRING(32),product_descriptionSTRING(32));sqlite>insertintoproductsvalues('apple','fruit called apple');sqlite>insertintoproductsvalues('guitar','musical instrument');
importsqlite3assqlfrompymemcache.clientimportbaseproduct_name='guitar'client=base.Client(('localhost',11211))result=client.get(product_name)defquery_db(product_name):db_connection=sql.connect('db.sql')c=db_connection.cursor()try:c.execute('select product_description from products where product_name = "{k}"'.format(k=product_name))data=c.fetchone()[0]db_connection.close()except:data='invalid'returndataifresultisNone:print("got a miss, need to get the data from db")result=query_db(product_name)ifresult=='invalid':print("requested data does not exist in db")else:print("returning data to client from db")print("=> Product: {p}, Description: {d}".format(p=product_name,d=result))print("setting the data to memcache")client.set(product_name,result)else:print("got the data directly from memcache")print("=> Product: {p}, Description: {d}".format(p=product_name,d=result))
Explanation:
We have a function that takes a argument of the product name, that makes the call to the database and returns the description of that product
We will make a get operation to memcached, if nothing is returned, then we know the item does not exists in our cache,
Then we will call our function to get the data from the database and return it directly to our client, and
Save it to the cache in memcached so the next time the same product is queried, it will be delivered directly from the cache
The Demo:
Our Product Name is guitar, lets call the product, which will be the first time so memcached wont have the item in its cache:
12345
$ python app.py
got a miss, need to get the data from db
returning data to client from db=> Product: guitar, Description: musical instrument
setting the data to memcache
Now from the output, we can see that the item was delivered from the database and saved to the cache, lets call that same product and observe the behavior:
123
$ python app.py
got the data directly from memcache=> Product: guitar, Description: musical instrument
When our cache instance gets rebooted we will lose our data that is in the cache, but since the source of truth will be in our database, data will be re-added to the cache as they are requested. That is one good reason not to rely on a cache service to be your primary data source.
What if the product we request is not in our cache or database, let’s say the product tree
123
$ python app.py
got a miss, need to get the data from db
requested data does not exist in db
This was a really simple scenario, but when working with masses amount of data, you can benefit from a lot of performance using caching.
This post I will demostrate how to dockerize a memcached server on Alpine and how to create a boot script that allows you to pass environment variables through to the application.
What is Memcached
Memcached is a multi-threaded, in-memory key/value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, etc. More on Memcached
The Dockerfile:
Our Dockerfile will consist of a simple install of memcached and add a boot script that we will start it from:
1234567
FROM alpine:3.7
COPY boot.sh /boot.sh
RUN apk --no-cache add memcached && chmod +x /boot.sh
USER memcached
CMD["/boot.sh"]
The Boot Script:
As you can see we have set defaults so when the user does not specify any environment variables, that it will inherit the default values
$ echo -e "stats"| nc localhost 11211STAT pid 8
STAT uptime 2
STAT time 1535833177
STAT version 1.5.6
STAT libevent 2.1.8-stable
STAT pointer_size 64
STAT rusage_user 0.030000
STAT rusage_system 0.000000
STAT max_connections 1024
STAT curr_connections 1
STAT total_connections 2
STAT rejected_connections 0
STAT connection_structures 2
STAT reserved_fds 20
STAT cmd_get 0
STAT cmd_set 0
STAT cmd_flush 0
STAT cmd_touch 0
STAT get_hits 0
STAT get_misses 0
STAT get_expired 0
STAT get_flushed 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 6
STAT bytes_written 0
STAT limit_maxbytes 33554432
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT time_in_listen_disabled_us 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT slab_reassign_rescues 0
STAT slab_reassign_chunk_rescues 0
STAT slab_reassign_evictions_nomem 0
STAT slab_reassign_inline_reclaim 0
STAT slab_reassign_busy_items 0
STAT slab_reassign_busy_deletes 0
STAT slab_reassign_running 0
STAT slabs_moved 0
STAT lru_crawler_running 0
STAT lru_crawler_starts 255
STAT lru_maintainer_juggles 155
STAT malloc_fails 0
STAT log_worker_dropped 0
STAT log_worker_written 0
STAT log_watcher_skipped 0
STAT log_watcher_sent 0
STAT bytes 0
STAT curr_items 0
STAT total_items 0
STAT slab_global_page_pool 0
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0
STAT reclaimed 0
STAT crawler_reclaimed 0
STAT crawler_items_checked 0
STAT lrutail_reflocked 0
STAT moves_to_cold 0
STAT moves_to_warm 0
STAT moves_within_lru 0
STAT direct_reclaims 0
STAT lru_bumps_dropped 0
END
Some descriptions:
evictions - when items are evicted from the cache
total_items - the number of items the server has stored since it was started
current_items - the number of items in the cache
bytes - the current number of bytes used to store items
limit_maxbytes - the number of bytes the server is allowed to use for storage
get_misses - the number of times a item has been requested, but not found
get_hits - the number of times a item has been served from the cache
When you see evictions value increases, this essentially means that memcache had to remove the oldest items from memory for new or more frequent used items. If this number remains high, consider increasing your memory allocated to memcache.
Slab Stats: returns information about each of the slabs created by memcached during runtime:
This post is a bit different from my other posts, but I feel it’s a important one: Facebook Security.
Facebook, everyone loves it right? Yeah, but what happens when you get locked out of your account, or an attacker gains access to your account and start doing things that you dont want to, and especially all the photos / messages that needs to remain private, can potentially end up in the wrong hands.
Facebook usually detects strange behavior, but being able to be pro-active on security on this can help a lot.
There’s a couple of ways how attackers can gain access to your account, but I won’t go into that, google will be your friend if you are curious how they do it.
Scenario: Something suspicious is up / weird behavior / getting unusual messages from groups etc
Usually Facebook will detect this, but if not you can and should do the following:
Reset your password
Enable Two-Factor Authentication
Terminate or Logout all sessions from your account, if you find unknown sessions, report it to facebook and log them out.
Review your account’s activity
Review Group Activity, if you are subscribed to groups, unsubscribe
Reach out to facebook support
Head over to your Facebook Accounts Settings Page:
Head over to https://www.facebook.com/settings , this will be the main view where you are able to configure/review your account. It should look like this:
The list of your devices that is currently logged on:
Hit the See More dropdown to review all your devices, which is currently logged onto Facebook, if you are not aware of the sessions, hit the Log Out button to terminate that session, or select Not You if you are not aware of that session, then continue to report the activity to Facebook, so that they can look into it.
This is actually the first thing that I would do, is to change your password. If someone did manage to gain access to your password and you are still logged on, change it immediately. If they reset your password before you do, game over. Well kind of..
From the same page, change your password:
Enable "Two-Factor Authentication", when you are logged out, or trying to logon from a new device, a notification will be sent to your device where Facebook is installed, or alternatively, you will receive a code sent to you which you will need to enter after you have logged on, just to provide you with a extra layer of security.
Enable "Get alerts about unrecognized logins", which allows you to set up to 5 friends that can help you unlock your account, if your account has been locked out.
Select "Activity Log", to review your recent activity:
Below Comments, select more, then Security and Login Information:
Then we will be presented with the Active Sessions, Login and Logouts and Recognized Devices.
First look at Active Sessions:
Then Logins and Logouts:
From this same page you can review other activity like Search History, Groups. etc.
If someone had to access/subscribed to groups, you will be able to review the activity, within 3 different views:
Groups: any interaction with groups, such as likes, comments etc.
Membership Activity: Any group memberships
Posts and Comments: Self explanatory.
Final Note:
People try to access accounts all the time, watch out for the following:
Friend Requests: people have a lot of private information on facebook, keep it private
Watch out for strange applications that wants your permission, review the permission levels closely
Reset your password time to time, use unique passwords, and not the same password as the password that your main email account is associated with
Watch out for links, some of them can end you up in a bad spot.
When you see weird activity from your friends account, report it, so that facebook can investigate it. It happened to a friend and Facebook sorted it out within 20 minutes.
In situations where a group of participants join together to split up a secret in a form of secret sharing, where the secret is devided into parts, giving each participant their own unique part. Together contributing to reconstruct the initial secret. We can achieve this with Shamir’s Secret Sharing which is an algorithm in cryptography created by Adi Shamir.
“Secret sharing (also called secret splitting) refers to methods for distributing a secret amongst a group of participants, each of whom is allocated a share of the secret. The secret can be reconstructed only when a sufficient number, of possibly different types, of shares are combined together; individual shares are of no use on their own.”
Installing ssss
On Mac OSX:
1
$ brew install ssss
On Debian:
1
$ apt install ssss -y
Creating a Secret:
Generate a Secret where we will distribute 5 shares, where each participant will have their own unique share, and to reconstruct the secret, we will need 3 participants to rebuild the secret. In our case our shares will be distributed to the following example users:
12345
- Share 1: James
- Share 2: John
- Share 3: Frank
- Share 4: Paul
- Share 5: Ryan
For this demonstration our secret’s value will be SuperSecret@123!, which we will split into 5 shares, but to reconstruct, we need 3 parts / shares:
12345678
$ ssss-split -t 3 -n 5
Generating shares using a (3,5) scheme with dynamic security level.
Enter the secret, at most 128 ASCII characters: Using a 128 bit security level.
1-41ac84013bf568d1cc88b751539f1ff5
2-7d9ca3ca26442bfcca35e0ad205e5659
3-519038837bbf1b7ceefde331ad1ae40f
4-6d4f4e0f086af5be033f516bb3e227d2
5-4143d5465591c53e27f752f73ea69596
In this case, each share will be distributed to each user to save in a secure location.
Reconstructing the Secret:
Let’s reconstruct the secret, and as we need 3 participants, we will ask John, Paul and Ryan for their shares, so that we can reconstruct the secret:
As you can see the secret is verified the same as the initial secret.
Using ssss and qrencode for MFA Codes
This can be useful for Multi Factor Authentication as well. Setup a Virtual MFA with a Identity that supports MFA Authentication, copy or make note of the “Secret Key / Secret Configuration Key”, go ahead and setup the MFA Device on your MFA Device to complete the setup.
Once verified and able to logon, logout and delete the MFA Account from your Device.
Generate the same share scheme for the MFA Secret Key, for this example: ABCDEXAMPLE1029384756:
12345678
$ ssss-split -t 3 -n 5
Generating shares using a (3,5) scheme with dynamic security level.
Enter the secret, at most 128 ASCII characters: Using a 168 bit security level.
1-8d2cf979fb346297cab47ff691bddc1c5a5f34af37
2-4d0f2cdcfff653cc60a4f293c15805f7e84b0a956d
3-dadb6d2cbe42772c9a9042273f0b71dd71422f19cb
4-546bcef428151ceb01fdc6007ac2e5e4f1516670ca
5-c3bf8f0469a1380bfbc976b4849191ce685843fc7e
Distribute the Shares, and when the MFA Device needs to be restored on a Device, reconstruct the secret to get the Secret Key for the MFA Device:
Now that we have the Secret Key for our MFA Device, let’s Generate a QRCode that we can scan in from our device, which will save us from typing a lot of characters. We will need qrencode for this:
For Mac OSX:
1
$ brew install qrencode
for Debian:
1
$ apt install qrencode -y
To generate the QRCode, we will pass the filename: myqrcode.png, the name that will appear on our device: MyNewMFADevice, and the Secret: ABCDEXAMPLE1029384756:
