Ruan Bekker's Blog

From a Curious mind to Posts on Github

Using the GeoIP Processor Plugin With Elasticsearch to Enrich Your Location Based Data

So we have documents ingested into Elasticsearch, and one of the fields has a IP Address, but at this moment it’s just an IP Address, the goal is to have more information from this IP Address, so that we can use Kibana’s Coordinate Maps to map our data on a Geographical Map.

In order to do this we need to make use of the GeoIP Ingest Processor Plugin, which adds information about the grographical location of the IP Address that it receives. This information is retrieved from the Maxmind Datases.

So when we pass our IP Address through the processor, for example one of Github’s IP Addresses: 192.30.253.113 we will in return get:

1
2
3
4
5
6
7
8
9
10
11
12
13
"_source" : {
  "geoip" : {
    "continent_name" : "North America",
    "city_name" : "San Francisco",
    "country_iso_code" : "US",
    "region_name" : "California",
    "location" : {
      "lon" : -122.3933,
      "lat" : 37.7697
    }
  },
  "ip" : "192.30.253.113",
}

Installation

First we need to install the ingest-geoip plugin. Change to your elasticsearch home path:

1
2
$ cd /usr/share/elasticsearch/
$ sudo bin/elasticsearch-plugin install ingest-geoip

Setting up the Pipeline

Now that we’ve installed the plugin, lets setup our Pipeline where we will reference our GeoIP Processor:

1
2
3
4
5
6
7
8
9
10
11
12
$ curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_ingest/pipeline/geoip' -d '
{
  "description" : "Add GeoIP Info",
  "processors" : [
    {
      "geoip" : {
        "field" : "ip"
      }
    }
  ]
}
'

Ingest and Test

Let’s create the Index and apply the mapping:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/my_index' -d '
{
  "mappings": {
    "doc": {
      "properties": {
        "geoip": {
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}'

Create the Document and specify the pipeline name:

1
2
3
4
5
6
7
8
$ curl -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/my_index/metrics/?pipeline=geoip' -d '
{
  "identifier": "github", 
  "service": "test", 
  "os": "linux", 
  "ip": "192.30.253.113"
}
'

Once the document is ingested, have a look at the document:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
$ curl -XGET 'http://localhost:9200/my_index/_search?q=identifier:github&pretty'
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.6931472,
    "hits" : [
      {
        "_index" : "my_index",
        "_type" : "doc",
        "_id" : "2QVXzmUBZLvWjZA0DvLO",
        "_score" : 0.6931472,
        "_source" : {
          "identifier" : "github",
          "geoip" : {
            "continent_name" : "North America",
            "city_name" : "San Francisco",
            "country_iso_code" : "US",
            "region_name" : "California",
            "location" : {
              "lon" : -122.3933,
              "lat" : 37.7697
            }
          },
          "service" : "test",
          "ip" : "192.30.253.113",
          "os" : "linux"
        }
      }
    ]
  }
}

Kibana

Let’s plot our data on Kibana:

  • From Management: Select Index Patterns, Create index pattern, set: my_index
  • From Visualize: Select Geo Coordinates, select your index: my_index
  • From Buckets select Geo Corrdinates, Aggregation by GeoHash, then field, select geoip.location then hit run and you should see something like this:

Resources:

Investigating High Request Latencies on Amazon DynamoDB

While testing DynamoDB for a specific use case I picked up at times where a GetItem will incur about 150ms in RequestLatency on the Max Statistic. This made me want to understand the behavior that I’m observing.

I will go through my steps drilling down on pointers where latency can be reduced.

DynamoDB Performance Testing Overview

Tests:

  • Create 2 Tables with 10 WCU / 10 RCU, one encrypted, one non-encrypted
  • Seed both tables with 10 items, 18KB per item
  • Do 4 tests:
    • Encrypted: Consistent Reads
    • Encrypted: Eventual Consistent Reads
    • Non-Encrypted: Consistent Reads
    • Non-Encrypted: Eventual Consistent Reads

Seed the Table(s):

Seed the Table with 10 items, 18KB per item:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
from boto3 import Session as boto3_session
from time import sleep, strftime
from random import sample

# session ids that will be fetched in a random.choice order
session_ids = [
    '77c81e29-c86a-411e-a5b3-9a8fb3b2595f',
    'b9a2b8ee-17ab-423c-8dbc-91020cd66097',
    'cbe01734-c506-4998-8727-45f1aa0de7e3',
    'e789f69b-420b-4e6d-9095-cd4482820454',
    'c808a4e6-311e-48d2-b3fd-e9b0602a16ac',
    '2ddf0416-6206-4c95-b6e5-d88b5325a7b1',
    'e8157439-95f4-49a9-91e3-d1afc60a812f',
    'f032115b-b04f-423c-9dfe-e004445b771b',
    'dd6904c5-b65b-4da4-b0b2-f9e1c5895086',
    '075e59be-9114-447b-8187-a0acf1b2f127'
]

generated_string = 'DHUijK8WK03hU0OF3UusjI0hhNd0hLRg03Vh430hJdUh9JsFi2hwO0s00FwO3wwI90Ju28UH94UUHU90fj9S99IfhVu89fU3o0OF04D8ljKNLsK83HJsLK8hUfsffURuuJfgR984h98jjF3sjsL0I3W0g039FJ8I0gh0IPI0Wisd8hhUg3308W3iVOJ34uO9OfhFJ900uL30oFu9JkwjfFUR4008OFkiO8H49DD4ONkKu0hUuiL0hR3NLNjh9uu0J0h09hH33gKf9980JhVL9483Ngh0h0fII4IOUOhHuIWodJlF90wuuK8uF84J4uRL00i30IO0d8g0UhIHo0I90D8U0Jwh0u08f9R9r9j00ujsOKJ3HU0hI90hrR8Sjs9jwhj0WFWrfo9g3j09IUu0ufjU83uHWK89i3r4Of8N00KN04O3I9DF09hfUOSdjHhs8I30hPkV0iRUhk9UN0J9983i9UIWioNFSUSWP0903U04RS84f0UI80j3U9I0wUFd90u9JJ9kJk9Hff43ul99hhhU0wdwIHusuh0HoV9Jh4I3U3hj4KH40NLH8L008N9FU8hjsWUjwUIO9N8FUs9fhkfV8NjF8g0OKU9UuS8I44009J8UJu99NVDhjH9S9wDO00fKh3uJgu3JosJD899jPR4L8839Duj9J0SjjLI90Uhhhh4hIsN8uIrjhJjIoi80084wf9KH0LiuNwPhOFOFhIO0o80UF0s308shwIj9iOJhkRUOI3J8fPUJhfN8uJhj8LU0804Ji803OwOHWgU90NU8N3hFF0fJ8008hh98jU9JJ0OFwfJ92JR909909Uh8sFf9P9lUgIH9IOI93rsIJlHKNH0hf099h9Nl98IskhV0hIN8oUo0JhfDs4JlhFl0I99jRhO80J9JoPjN008F03H94U9PHrPuOsPV0FIogD40Js94LIw3JwRP8S3LFk09uj8WFWUJ0iRJjhLh9UoOIjP403kji9hsswRfwNh940jwHwKLIsRL8suouIl9IiH99h0j04h0H989O0uLKhJSjN0D39jffOO88K8hKu8R3V9VRU3Hug09WF0ssJhofU4fW8Fj08KKs2IjVI0i3NjfD0uPFLsO2IIFJdH90KJNR8uR0h90H3J9UFR0890OJI88jfs4Kgf99w800JUHIlh9rs98k3u8249ghju9UFuO8Iks0OOfKO00w94KL980Fw88UiO9IfFh903OIF8dIO3IN99hJh4ufFO3hNUS2i9P9rKlIVl00WH3I3hKP9Rs0u09HNhNHUVhd98IL9fhu0DUhgH0OrJIPsJ0d8fIj9uJOF090R9O8JokI00U9jOLH9h8Us9ifiIhw00g4HKiK09oK8Ij09IUh9US3h8F9FhN39Kd09r99R0dI30huhfR0jL0Hs8u898ji8WjNORuiJKf39kJ099V99h300gJFwl0gUs0j0h0I0wI4Uh9o0l0hNIKjNO02JIj0jJj90iIR0uLhJH09L9H9h4SVsF89hiJV8hh8fhu394uIshIkLH8JKI9JFhsiOJ4s9gk0hu9kfOh8HP9jo4hN0k8JfogI48hr0FwfRjU0SijwNh8VF9J9jUk80J92H93F8h0S29j392RiSuNoh9i00HhFL8I9jN08OHhj0IIh0RKIJ8N0j03uHd0fDUhJ090033FH9luhjugfRhUL8JwH0if8f0hH9h8hfOjiVL8giJSsWgkh9UhOwhjOLhiUL8IjF3JsUJ3HFDh9KKF9Fuh8FDju3uwNFJhPH29shJjjJUOr0f9P94Ff8NIDHOR09WKhUI9hUURhN8FwJP990of9dVsH8W8h889j9g800I8wJjKPR9fu04uINkujF94f9w0O939f0I0ORL94f00O0ujDrIUI0D8rUK9fu4I9lFwogio9WLj0flf8SV0w8F0OUJNVhjWO0KI2RIjh49h3I9NUiIhHUsgDkhhhsJJj9iH8KjI0h0fkUfUsIujOVrLjUi8098Rjih943380D9wss9Kugdh3SH400N888h0j9IPg0Fu8joJ9Lf4U2oJJN833JJ8Fu94kfwR3Oh0iuIH2SFFIh3J93Jj03J39hKI9D8Ouhh0ih8IJsI000I8NRhkI80hkfgjJN0N9hhr09wULF3is0uJ9jV0POIdIO0gu808KH3FF8O2P39r9003dPOIJ9iK3lNI30h0wJN0II4803KNjf9uW9490j080I009H0R9uJ33ggVhlk9j0Hs09uIfdJ009oU9hiiRLi0s8fhFN83d303hhF0N9l0h9F3SF9998gddu0iK9RuiS4ow8Ssg9hi0JNKffOj33Iuh98IHO8sjNRusK0ushWN0JsKO43fd98F490NJg0iU8hhUUFPOJNjg4HO3hh00P0ihUI9wgw9JO0hr0NhOr0wF890U8FK44hhOVN89Kh99IFS90UOI0UIJ8UwjKj9i48jj0sU00ho39RrUIfIuOK8h0O0h0RP8dJ0SOI9sON9l0wl9jo38LOFIURuk0KJiu9L9uI9J3LfdhODujHoW0DIOD3RLUI8Ih0VIFhu80LOO9FJ0hJI89jhfF00HIJuh9gKR8UNfhV8wiI0sIN40iN0fOKIRukDO48JRIIFO0g3IR9fHFhi4uFIN9393JH89g0OVhi8lONuJ8UkW0K098J8INUh9uOjKuJu88gjsWDFJjOLf0OFww93FjiJSH90F0ujPOhwO2oPUNUOkj0uH9j409jh3hJhf0rV8JfwI3PhIIOukffuhDhU2ujh80IJLU0IIuU0wF3g8iu9ik09jF09fhO90449H8uIFLOjFR9RiI84f88oug4jJ3gIhHfKHi2S0u9JFUNS0U4OsOlUIdI3JrikU80h93iRL8fu3ffuIH0U8ulo29Owh39us9I9900DjJwg90rR8JhHhw4ODuRrL8Lgg34jsN00OJhRuJJJuHjSFS0Nh8hjIfsi90494i0Low0u3w83D4j0RRuNI8JL2hKNRIRJ0dNhL83uwL9hfFIdFJO0wJF90OIP0O2Ig94Ww8NKH09w9DO3090OJ89DOo0h0L8s0f888fUdhfPWr090oI3UwNjNKfNUUW3hHifDS4Ji9IHK000hh88u09OufIKI3l0098hO0030O0I8W980lRI0J94JW9j9lRJOh4uNiJP0fhUf9h02uFoFjIOU88i4hj48W99fUjko0L8FSJ8fjf99I2oi4J0d0jPjJj9Lo8fjh0K8u9JsU3f8FUh8F3k8HFU900wU8U9Rhwfsw0U90oF84008Uf98wfRJ003w2uUhhlW0L0Vh9H9OuoIUj08L898KU9388UUh0093g3fFJU8kF3j0WFoFhL90N9KI9838IKfr9djgK08ghVFW098N89hJJ0JU3fhh0jU0F83dhgV8lVLh020L3L2VWOD3ldL080hh8JiuhUjFN9uIjwIFk2FhP8K99Joi9s093034hU3r4r0FO0j3IIK903ff08kU988ohru4whO4uRh8Iuj9oIuhFJIJwhugLF8KW0j9hwsKf3wH0J83RjN3LhKINJK9DSUFJNU083sF03KfRhI830lj9h90i4FiJW9NRhi0suh990jJOR9woIUU99HK29R8H0J9VJh9LRf40Fshukj990IFgf902uiH009FKFJKNOU0o4fI9WhVV0ulVh3ROfiHf9FJLwI30d2ii0OOWhDwR00NN2juf8U90IPWfJ0JOFh00wsh8gg99l083lNJNuF039JFssPh09DO0O2UV9FO0guK0oLKrjklj909u880gN9iWDKl40uOhu8o0r9hwIoKN0rOk0h3S3SUhuw0F0Hrh9fIjVFjVVUNJ8VjkhU0Us0f009U8URhj4Iw89O9sod39NFhw9hHUUhfd3uR0hfhhhP9f00JVLI3O90JjORH0FFs2J8if288jd0h8hFf0j89KRNh9kN080lhURiJ0jFPjW498LuhJh9OgK0wNJ803f9hfV0O989Jhk8wf0UrKFuU9hf9LF0DdOuHI4I800j4wD0w899jF9kJ8f0u0P4iPWf8u8I9hF9Is8gHJ9hIDfI00Jd99OKsIO30UdI88800N993JsJF089OswgFiwhu9sj880l9RkwHJw0K9ULLw80h080J49o00U0Osks89N2lwIL08u0kOg800h9NWHjjgF8Lj3kKlOfRh9h0L990N8lSfLu92hjhR9hkIf093sl9uJf0s9I0IPF3Sh9IK08U9dO3S80dJI94LlsVJ0hNVuh8R909h3I8fuh408R3o89h0uW9J0sHw0RF8iD0I9OU2hH9fPS9fjPP4fUgPhi0J9jJOOu8h9Du0L0998U98UsRs99jsID408URP09jk9fiJLiiL89IhV9Ks94U2fULHr00D809JjVi94u4FI2N0INhJHu903988U98hUsRg08F08UjfJF0Uk0fhHI89oRhl9N0ufoO3Is9hhju3rIgRJ0OkPFLOh9i0JfJhOhhuuNIhI8003uL9ljH39L9h2L4SJ0Hi8uKh398K9gh9H0SI9WJK98hUjj09J00DL9URNJNihI39H0sKu00fj880HIUwFI44iNL9UshH99jwI8lH39H9NWRO9LUURK8uoS80J893lr8P399OJjJ24wV409IjN9u0I4s8IJfhfh49udF9hlj00uIsUhfNdFIkK9h999UJi8Hh0lKL04hh308UVl0J4O9g48fHh4Of909Il09higwR9I9WUR28V8OIk9K088hiih204I9o9K89dIijL000HN0RKrS94IOwWdSW8s0wiJ0JlRjVIId988Wu93Kk3u0fw8930808HsisF3LUPdNrL9LP9IJPOL083Id8wJ9h0oHJUoW0JJ38f8u3ws43JRr9Oj9OKf8oJ0ohiF898UjDuLhj0LW9U8hN2j0I2iJO038u49Phs9RUj883HOfWIO480wRUh000fj0HNh4VL9Jjh0W2FJdJJfJOR099g3gk0oI0R0Og9jU0jgiOOfOIjdULJIfI94HN9gJo0I9909O83UOJS0rJ8J0L09d9og8w090DUhfKhPshRKIIIf90Vf8Sf8I0K40us8LFj88W0Fh23922O80JNsJoKwLSwhKhW4Ir8Pi8JNhi9KUIifs88UN3j0HiH0L9NjDPsIL00s8jJh9890dUWNj9NsIhh9J9o8g00dk4s8hJH8IjhFg348W9uOgO9h9RlhHFi0WJhwdiPfKgJuuuh0hOi3Jd8hF008hjJ9j90WJhhiUWJhuOjDwHPsUw4L8wUKU0Sh8wh9h02h8KiU09j800PNu08dhlSuIN0oi9uJD38w9IR0hhdLHhJo2wJWI0RhHIhi0J4dFi3ugV0F498hhLsI0Ui0Od8jKSRghU2wP9H99J30Wj0hj8IJF8hU09FSrF0FWrjhf0Luo0I9j8V3JfuKR880j0Ofi9Su8uUj98hIuOJid3h0FwJJFhoUI909SWjoF9I90H90LowN00sU929JsjFdij0gh0NrJHHhRs9hh880FIL93h9J0038PoIh83N00J8PjUJf0899Fhi309l98IJiSISPRFL0HK8ji0k0LUjwgJ00hu48FoIPW0hHu0HDJjK40uhUOLIU8H8hN9IUf2gSNU8NNOlg8LwOw0uRjJHhSPO8NFf84Hh893hD932LV90300FUuJKh8F08uOhwK3LI9V00hI9hlRs8WL8J9IhJrDNWwHVKifUh0fNLU8h99Rhh3hs9whgJh2Jf8UhIuJLksodj0UlIuJwJ904jDjW9s0i9S90i8NK03i299w3dw3fFjHI0ssN4009KiUSKNf9hNUhKDi80i9OL0fKK8hIJgJO2ULJOgfN8h0iIIhwJJ0uf99uKIO9j80f9RuRIHIoU3PhIW3K40Jud8h902w03h09UhIO3OwJIIgHujf488HgFOhdhiis8OLf0kIJJWHJojV4Rh08ODOO8FLg9fUf98jJJ9484L8hji9J3f9ROWFiO409hP00090WU9ww9Ih3H4KDjHF8UuwWOOw00OOig3IIJfI0k0Oi098UifRPJJ4D99us9FDIgRijI9UfR84wU0F9V0hIhI8rs9KrRh0jRF9uUw8K4HhSh8UIUPsWJ0of8SsVuL900H8o0g884ukJLOfu00hl080RuhUOu83OPKH909DDf8iuurLL94h9989lOJhr8JUJL0hhjIshf09U090IFUj0880f0UJP83ojNS08880dw32040sifJl39r82w8P9h40r8OUO0WhIFljwh8KuNJf94KhI8800u9j0j40jhiRN0f9493h8hIJ9D8RhOKVHUj99ONIHROFihIRwHUhi0F9903Nf0fN0O4ROo3udf8i3hIi0F9UI8L88993hwI0UI9i8i9f89IIdhR8ffF9hIh93hhjFf8DKD0UI0j9LPIu8Nr04008990jflO3ldsFo0skhK0JjVOw8O8409sFuFJf08h0KIujrJ928i2RNHfjO88008089HON2dJu9N90hfFh9hK9UP0h088hO8f09S8Uhjo0U8g88JgFI8VIJ0UJi00F3guP9L9l8uVH0V4iih8k0fJ8OKjIO9F3JjR9ro0h9Iij09irVj0If8jU9HIuS89khJ9RUUjrh9oF8L0oNu9094FN8kh08O88jh8sFIf4I4F8FKliVh0DN09FhJuhwiu9uU9Fi8HNhjff9h80F9VK0h00NLjWdj9RJ8hJ80rKhU8Jf08ur00l888u4JOHKPOui98hUJ8PhNOh0990iu9K03IIOHROPUJwS00wuO8iuIOU90D49hS88Ld39KiNI48SRw09j9FV4u0wi0Wfj4shR3HolP9f0Jlu0IRNi8Dr308f8jhudhh49hF0II38RR80rRh9hUuh0K9IuW0jOHNkIhs88fHgu00su8Jf4UHIj9I400F09fUJo3r3SNJ038JjDLhjfUN9JlUj9UVKS0J80893ugjOUhUJ9wuDsNVi9huDRhfJlVK0KR9I998ihfsWOfULOFUhOFU3H0O0I2P4rFu9D9F9gkh8Ogj80K8s938HhKJg08HJRhKPI48hu9ODUH93VD0uk9FH0V04h89jig0fOIHOjhh0d4FIh9UIUH9UfUhi98Kl9UugwUs4JuOI402IL3VJWgO0hf00g9sHH329KsSuJr902O0Phh9S0U0j0RiKhfLhs038LNOR8sh0k8lhRj0h4FN39oJ8NWlhsWJ00hf090Ihj9RNuI0J0o9uKO89h040KjJS9osjs9KJIuKr0wi3UuD2U9r990Dih99j09fj9k9jl3Su0JfkJi3JJhJh9KOUhi0WVSlUiKJSju0fLFDh99OFuJi0uHj0uUKj0fOHHfu0f00LO0082K90hR04J90hOLIhd8uh9O3LJH8s89hiL9O09jL9999f3h4NhR88U0ri9R08sKKLh0U8Lw9hUHIfi0000R90hsKHJN0wUlfgFjfhHIDU0P320NFO0ihOrO9JIVUU8000rr00UriNus84hj0OgU0dF3OHJuFD8hIuL00sJ880Vf938j09J00Hj9dR8L88I0ji9WOs99FIuh980083IVsUfdIS9rO8hhLhOsL080hO8fwjOI8gF3Ff0Lhu0JJhF8fS0DwOwKs9NoNJ3P0hhO834g4NS0h98w3i9uhj9990080LfrUo808h0R0DFhFIjw3RjfgU98i894r8R9uF9f8i90g3ifjL8uIw0JWiO09O04u8Nh0iOOV8j0I0I099909VwhJJNUFKrjdh0KhJ99HuJLo3W0N8UUI4Ishj3N9jIfS9S9FJs04fIrsl9uw99fhjRo8jO09U8JwjjjROFf0Ni8u0lU0o0u9O0V0N0w9j84W94jFh09Fhh4Oi9UHF40jU0furFsIJJfI8hO9LFUoLF4j08889V3N9FhUifjuUNRu38jJHj83huNLH80whhIKsF0JFU8I0fIij9Ph0R8fhUhN2h03hu4rKV9L0wOh3fhjU0O09R0gWJu4F0hRuVsgINsDHjWuW89098f90ljHUwUON3Kwo9fWU000fFN89RJuNJKhI003F9DSNOP09hu3oIJJNg2u880J0wh0PIsPH09P843rgIhFIIOk9i04U3hIgIIuIIs8ir892HFO2djLH9899hI93I0NO0i0JljH2RgU8h3D0ui9OfHgRRW3r0J888WINhLSUh83DKN3L3099wR9HHFUIuK9394Kl0VUJh0NJUOg0JwH98Ih8duK00iuhhUhkwoJfiL0hONf9U9uI0HU8U89NPF9HIhOf02fgoNh00R9r8ff38IswL40Hi08h98h8uJuug8Sl4IhrwfOHsi9HI8Idh8I4FNo9KIL902oUfHgVl0HI9N909SgkFDhh0JgK0lgf9F9J3009du03jfRiIw0i998O8lJ0gJ0wkRjSsJwg9lI8088FrR9wFFi3jsFNh9w0OjFJ298k04Uhu9NF9DJ08h9Us8g0hsRFWO9909hUjj9w98UNONOou9028809hIPo8fDL38sJhw8V0IwUJ4w9Hk9Hj9JoiFo89ijiL90guu3J2K0KNsNUfrOPJKuh0KH098SO0PUUs0hRNLs0u3O0j3O8uIuIsI94O0hHjVfu0oilFiJSN0Jf9890ShFJr2uds0H43D88O9IP0h98IJJOPh3ru0300L009hRhLdsOVs0hH0SO0fFiDF890o8hfg4JOj4089U0RfV3f9HjdsD808VOF0O8LwfF8ruJ80i8O0P4h8RhIOh9wjK9Nf3F298N0HIdUNNr9dkO29DOw8NhoH0SR9H2kPfhR0jP9su00L9OFkOs4jIh839Fh9I9JuOwR9iJ09SIIjI09fj0390h2U9O8UkNjdjF890hkJ908808K4UoP9uFRWL02LfHw0R4ffNIPJ0909i8u9fhkJshjh98JJN00hOjhj0Fu43VI8suH8UkwhOd243UN9J00h0hKwhfFFO8P9dN09W9hRI2IIjjJf942LRhD2UOg2fDsfuOUFF2OgluuW49j09L3gUN80h2hIjs9s8UuR0hohj99hgK0LVwO9FhNH000jLhV98sINhf89HiwK8h99KL04uUIjJuh9903sJK9Kk99D00jhIIugJ4JF8W0f999I9iFD9k0N0OI9F00OL9P03gJ8wI3L3FKW0F0I98hhsdoDS08jJ3I9LR4kLFkHhLFF9s8I09j8JDRLNhUfg0UfJsKV00J9f9KLR3soP8jRodJ9gO0h00hUu3Ojf8Dh83LHN90ws00u4u8usu8S990oLj0I3O80NFRJ0h08hjOfJu3l993Nh09hU3w98h00P9HJ9hIihouh32r98Jhh9JuOok0900Uh9fo8Rh889809wUh8Ls0i98VhHu0lU90V9JghLJW8FJUJUhiujdokhPfu8OJJ82ffrhfi9Nhijw9uwUhh92uN48FKUjNg0VRjF08IOiJUr08Fr08UuIIwDKuNP0j98SuJLdfIuOUJd80D94LlRUlN9KHRihFwI0ud900h80L8lIj9RJ0DNlwflhw9hJlK9hh9N8diDhOU0osO30VRh0L00dS98LS8KfH9NhdH0888uUIWhK80O8d8hR000sJJPNHhUfsu0ish3fU2w9UF200UJOiPJr9O0KhWSjj4RIjl0jVI9030O0hjFl99uh89W04UfO89U0iu9jODho03s9809R0Ph9loh9808V9R09fUKJj0k9R0UH9J2hjLUflf9uojLRNVOuKO0FhjIL9I93h9fUfuFJihfiFNuFdJRuU0shJh003w0ugo0Hh0RI2iD4IW000gs99h08I9PFS990S4w0HKuIW8K9909sI9lHUUL9UiiN9SOjFfkoLI89gIHiHj8hIJ8OJhij9h9OJIf0IrL89djJ0SIo9339H0iFR909II09lFK9uFIU99jfsI08ffR08s8jjIHfhfPu9909HI80889OUIO804uDW034O9d803I8Fl2ssFR0wi999S9U0k9I049douUIdIPu030FN8KS9RhIuUhV90uJl8uUIHfP9Rshj08HuISF3kj0Uh89fHuhuh89jHUfOf0w0Sk23J089s8080JkWPuuR98wUhU003k0IhwDKLh99ghL4I0VRj4ijOg8fJrLJJ9iIfH82F009UHh39h0Fjj0Ijh08498k9s3swFUF94hIO409Ni08Lwi3jh88F2hs9s48Rg03u9I9sUhuIkiL8wFudUOffS9ShJN83JjJ0ih9l80UJh0ROwH9kHhPPsr0i90fIJNN0sDISw0PshOjdFujo930RSIU04u988LhJwrU0H089Sjw9083088Oho99980FLiOIUduuuj04hJN0h4hJru9J0FDO89IuWUJ0FFghIhorJVPIgJh89WuV8iOIIhwj00U9w2I0Pr9IFj9S9UgLF9JHJ8H0KHj09jh8ir90hUi8J30PgFoiwhJK9j4HOVJf3fH39ho33IJPJj0i94L9j8NJOiD4w2R3i9LFLN9RO4u88f0s9O0LjO0Jl9Pol0Uih8wDijkJHHhN308dw99h9U3H38IK0LO0L8iFOFI0394i0hw9ruF0989h0090w8j9Fu0h9hSVUVH8hw33900UJJI0fhJFKi49u9so09NuFI0fIwDs0uJ8j0hu0ODUhhURKuFLjUN8D49j0rWsh0DkSjfuU90kJsjV90dO9Jf3K8gwF34JOj8hiF0HJ0hU0II0092hjUI40IJIK9I8UPIoI80sLUHRif4jJ0JI9SIhiHjV8hFFsfUIjOJL3jUI04rd04JS0O9FJI8hfFIJj09kSui9w843l88h398OfJKFwoj0JlhO8iii93hhk8KU0VifUhh8KKIOhUINiURHHFOS9RKk8K0f00i8h9NOJk0kjJF00V09hwuI4JKR2R80s0K0Lf809huu9o9UNh0oshh9hP9hJOI9K0NI004jww9D3uu9wrfwuijfJU0urN038sh49JOwOi90IIUlhfSJU3i099h0R3IsjU3OuKKH9IhI0ufKhJ98HFFh00hI999JRIFwO998hDif9KUU0RfIRjwu0L8IJIJj8i009H90Iw9wsjjdOsohFO0FIJi49943KoRJ9W9rO0uUlguwH4jLFJH9kuk8sL43PjF8989K8wdWHs0U0Idwh09N0h2IFhJR0ud0D90r8jOj9OFf92UDkLH4IK88ho2iw0hkwjN89IHPiFUNRjfHhJ0NRH998h90Rf3hjwWu9jr08U92i30FfNUPk8iJfj9L9K0990O4UhfIjFJgV989LgH8V989sDd8i9u00jU0hhUIDuI8h8U9fd3hUhI9J9kFw800RkLu39OO9F30uhWF8IfhUFLR99ohrhh8h9OwUi9V3ih90L0JKDVJhOR89Ju0KhduUgNII9RN98h0OF0HfhuI9I3KI0k9WI4K8OsV9IhhIFDjhPK0u099hJKuLu84K8hfR0FOUNl0W4I93I9l9VNfDJulj90JUjJ9JL9hR9LI2JVVK9H99hJF89R0kuj3u9090iIF99NIfh0F4UFiwkFHU0S0h80OFioPJ3Ks04HhO9hjsh0oNPuR8LUU08gHu8K20PR9u0Ihj8F9fL3J98L9Jh8320SUI39889uLu8iF0sfIH0i0098l0i9u39IP3020h0FHFO8LIji38oL228842hR4shOU8duu9jdKOIUJ4i09Ohh04uH0sUjI0JH90S8S00w4JDLh0o04O9I30w0k0jj0k00sIh90kId88U0iHlkO80sj9I08LN9Jgj83R4J0h00P9wJ9Ri90fOih9h4kufiiRhFL38Dwh3sO090KOJ802RIhhUh00VK9f8f2Dl00fiF9KPRwwr9hK988KL0Ff98J9Nuu8Nrihf9LFOKu0jLj09JdhsSKHr0URofjUJUKw0JNiJ0ROiJJO9839O4WFhfhoDfIU0j0Ndui008939fIV2SN00J00FJV80VF20LwJNd38jUD0I00hL8F0jf9Ils84KswFUHO0998hu409Jd0Ljjl00Ok8O08jO3hUHKPi89I3J9989998jsrfLu8IhJ0N98hR909OO9UIW3PfU8fK9hhIssI90IINRHhOUh0iJ8j9isJJf0hh89HLfWiNj9h3W3s3urU3oi39hshw08fSL0ih839L4h22oPjUkUl9F03h900FhjViV8u8IH0R9F9kI8iJO9Okw9W0k8329r8JjOh0fKOjOjIU8sKL88h8urjIgrSF9O9hI8340KfI9wJkH0uFhdO9O0u9OU0Jr0IhR3hFJRR8uJ9hhfK98OuJJFS8sNJ90308l3gi8N9uOh9SOUfdU38FIg9393P9VwLN4lhiuFKHRUrsFNfII0oO0OJLDK09wSI3JOrUJ08hfk3UU8u9OiP2Us9P08idOFFODWOH8uikfs80O993j9uL9SNVL9J0F9hsOIuV40R80j0L9d04hh04N8gf0I9R2K3I0ws400uN98089FK8jFW2hLuOf9I999f4RUhi3L0D90JfS0lu8w0800huhSN9VFjHkIuHPFUj0hKhOgiUwSNFN0ouiI80F49h403HRUulLD02J9Wu883302UFjV39Wr9NihJ3FUN8h40K9fLJJjL989wNIH394NF8fhNjsJJfoNLr8udIj8h29UuRLHHhHJUIVJ0Ns943D8Ps8hOhjr00OVjFsj08RFkUj49099UhfKu9sHuLN0oFIhV8i99JJSfH0fs03uwwwS0f90JL0rJ39J9KWjJ9F0I0KNOjhhj4H0IKFFo8r0O0sFN0sOF4uLd9WHih8OuO0o00gffNsU39880NfI09H898U8NUW8ls0ISJ0UD00i0NL990fJuJJP9h0ghIUKIgwIIwh8I8j8FK0w003Hff00F999u98Ou9UI3JhdH9LO2V8dfJso089OHu24W3IFUihViUJOjhiwS98ijhVJIhjL0u30K98FJu0N3sh9uk0RVi23iI0Jjih8fHVl098i8hU899R99Pd8I0S8IDJSIJ8IuurfiW28833040huLj4jiI008huI00SlJ0fUhklF99Hrk2LRIKR2RfhOf8K8fK40Hh98H0wRO8O4L0UPLK4F9wO99OjNL43jf0hD9suuj9uUN9uUuUwWUj009ksUf9L00S930fUU8kR4NUdh38J0H89sf9fLr0LFgJfu0UWJRUjUUH033h0IwK8U8lR0Ul09hLOHOuwg8Rw8Ji9HgKUuHjwuRhIf0h9hKi9f0LHI8IwWNhJOOS9RhOLk9uf29U4KUIRw9JR998wRU43u0i89LifFP8L3P9h338h8Jg48FUu9diUO0PjusJiRIFkhUOV9PF9R3u0g99j9ODFiKK0jO30IROu4O9I3sjku0838W0FgFNDUJ9shu9rW8RgRi00Of9O8hg0P3j8gSK9RS0f9Ui9UshH9uJI89FofIFrjfRhIR90I0fVUh0J00I0FgufJHOdJ9ufhfk9u88IOKKOuLS9kRLfuWl0LIhRI90JJPjh00DsUhlfwUs0iJi8RFRJ9i0g999O09J09KUoJ4whh8i4u9sFu8IO8909Kfgs9j0Oj9O0k8F0Rji9u9o9Lj9j3i98iDlHjIl098IU9N0JkWUh9f0dKdfI40FjjF0Uiwu09JjjUNiFLhssJHVOsws89RS3LI0RROIUIJLwI0U44jUsrJlHjLH043j90hlLlwh0H9lLjUjL98f8L0dfJ9U2FHURH0RJ9KOw3UjF33J0LIJLH8r9h090fOROOwPHHjiiLI0sOR98K0UuUUIKJ0wh00g0JSJW0RiR898Wf8LsWUJ9Luu98Kg40fjUU00899FIJfIUh0880JUOIFWs98Uwu0iu09wFsDORgS0J0uH2I0JF099IIFKFuwdds9U88IUjsUju4hkf8N8N9R9OSuIiuOH9r9lu090jL000VI39fiF9I89f89luD0F980DSO08ihDwj9u0WN9hjSiWHUOorFL034jO8ilOK8kll4L900fRhh9h90hu9O9fuIVN8jw8shsiru9hgf99Vi08jSUW9U088w03FwRu334gO00uPu9gf0f3UlWg090jwSUVHoFO9jNKr8FH0Rsifk8J9djhO9009hIH3f483DHKJIKiHs2sU0JOj9K00hju034NjiIOhsN9sNJjuhhwOHu899lfHNKVgSKO3f8IsfgUkfjr04uh080L3HIKVfIg80H00N0huj9R9OHVRK0LIdjRULu80hU4UJ300OsUIfhUN9IUR8sF0k0fuOP8LhHhLj0W8FUUH0U0u0sh0i0lINfFIUoR3rIw8UJjwJ8Jl3RIl8938j099800WKluuJ9Nh0DI00o480R98JhH89dhFu0IILhsU3i0Jiou0UORjh9OfOHh8r9gLhsU9or3FUksI89FFO0NOwhUKKh3fi8f84RI09jFFhFjO4R0UFuIKg8D03i403Ooi30j40IJ0NU0OLN8wD00u0w3I9O02hhjjirH0j8JJIVN0f29O3HUDjuh3K0ujku0020ikhD8J3W9O9KhUh83R09gLJ9uD99DFhL90rK0FsI0h89w0JDUuflUU40wIo30F8HOgsFRhu80IJh0FdI0U3O883l0wR9IhRV4d99HhO9FhS03dH9902wwD8slV4iNfHHshJN09LIf9f9UF9D8KJ3Hfi3w9K80ffu2FJIKK3u9OiJI90hLLuj3JhiJ0fiJhIhON0foRUO8hhU03RU9LjfFujDN4UVOJ3IKHfo8r0uN9sw3Iuj8D8230VUF82NI39JJ9ofKOu9hjPo4USUOds8joUf8OI8UV0FI9F88Ndrl48f024lLJ3FfjL0KUNfIfONS8j0Rr30H8f2HIoKs8fw94fh8UljhD8Jhr08FRfihh8WVOLIU34sFk9dkN8NV4HWF2h9uhFu03sgKV0j08FiFhI0jrUFf9OJ0huRFU9o99RHhR8Nj0O08h0099Uj0J88oOjwfhUwNijw84ONhhIOsNj99hWh9jPNOK989fOI8oUK2933j38NshRIldUPh9i9S0KOs9OJh4UI400jiiflFu0hiDJ80lwgfdjWkoR9lRhh3su90IK9P0lh0IS3L900dod9ro8099J8sKhrU8UJOs8I03kgHOJIIjhr9HLDOd9999uOh0Ig0fsHk0J8F89J8whh9DJ9NOf3JJILh2kj090P00UKIIuw9P040rUJUU9JihfJHhJ4RU93F02FH9V0Jr933j94JLs9jk9I2ih9RJ4Ig9W0jjIFhhFhkU2LoKH40J8hu0809wwShSK9juI048Nh3rihw8039wFKiPI880usoj84wuL889fJI9uhOFK8l0huhju4Fh9JhLVk38KjJfu0L0h3FNKius90Ff0oji8sUffN9L9sR8RPfIiFPu93jLr889FVHWw0HuuhsdJuJ04U38u80FluILh43UikhUwgh3JKI9R9N0908hHLIJsHSUOI9h8O09ikFIwsKDufW4OIhJ0JNNwj89fIihwO09f3h9U30FK8WHJw9R8fr9IIf0iFN890IU8gh8UNo88u0LI2UHN0U0fUf00V88wiKIFi0U9343WdI0388g02hFI0hwJfHfwsrOHHFIh9sUhhrNK9hjhl0Pj0NIhfjF9J9hihfl9H8DHJuuhIWf0HFLoUN8F3w0IRh3w8N2f03JUN3KRH99849hh34H3hIIw9iD3Hk38ksRRO0ufj3d9wjPh0h9Ui0JkOSJRKjW0FI9i29s8hFIk8SK99H8dWw89UiU0iUR0NK0uk4H9iO98L9ORhR8FPOVfFiIjU8d8VhD9w8sU20LW9NI8jWjUjhwDrhNJF8k3ihjFlU039KLJrN098i8I0UFgIJ09gO9oUNhI0l9iwIlO9UJJ43h4r0OS83uFr03hIjdiP0HJw9hU9dulKO0SFIOJKKOJJ0F98988Ifu8jo9P08S0D0OIl9JJ94Hkh900I9juouUJHNhsjjJK9SWS8FR08rUDK9hPU4OJIFhd3LURhfjufhw9I8LId9JihhV9u00J00R9ki9Dh0899ho9i90HO8rFw0UhJfu8ofoo9lOO0JN0IN0WID2i2R43LI3IfFNj09389hRU0VKLu880uog8Hh98O002sUFN9Uh99j9or8r02OS809909NWroRjKhl9f9IL8490I9kHF0f9u9JO83LJ0h8008j4S0wf9SJ99KhjF0W9DOk0goKhu90HLiJF90kUKgO99JLJ0hh90h0Dfou29kiwVI0h0uHR0H0s98Oh0H9lr9s490JHf9N4089Lih9O0Hhw20khSKKw900F3H0h3iJINU3w9WN08wRd0jRN9UWfO909J00O0ju9FIs99RJ9L300O09g99k8H9UswuWis3H094j9Nd8N8kdoh8rfKL8Io800OSiWIRrH9lR9s8Is40RFI9uF00ur8oou4flN3wd0iw0u0h0L0N9sVw0FULiui9IjI98W4kjlPjo0O0h0OK8Hi33iIju9Ruf309do8SL4dr9FIui0L3Jh88JS3F0JJg999oShNI0fouIN0r9h8HN82Fdu48sNN8449w04hIV0Ur09IJg9RhRUU9rO9080JIP048hwfJsIJ0Hs09l8hh2ff9WifR9luNi4Jj3KdfNR0F4wDuK33u8i2hh3R480H009UsiuUL4N0grf4HJsdU0LP0I0fw9O00O00kO8OIFJ3DhN9u98RH8wh9RPghOoOu9V28JiFdWhF3s0UuJd089fH289L93u8000UoIWKigVhI0g383JU9rl9N43hdJ9NkO8j0lu0hVV90LHSjUF0jj0uh0Ku808Oj20g9NF20w08u4ND0F8D8IVPshHh99932Jo9J880J0SjLwuJHDI9Ju8INRr0H9jK9OU39hsJh3ifh8hUfh0Ihh0FWoHd0OVwHh9gR800f90sf0jVh0I98lFL0OLKhRPhI08jPh40swJiU04j0F3o8JK08ND9048oU9IwOI9h9j8s90f3RPdRi0OdhsUJ3900hFi4owikK8hRUhK8Dh0JlJUILu89f8jO90jDINNW3h9g9UIFLh0IFK4PJ0808hd0l309uDsPgUuHjhlkF00F39WjL02S840Ou3KO3ihNJKoiOOlIWJ9Oo08ufs0s9jSw83L0LVwr9VK0I8wH8F9hjIOJ3V0Fik34w28IFVJwF03W988RNfF9sf0IU0H8Uwhw00F3Ihujhd0UF3I9090s809RrhdfRJNwUrOU0hfR9909ShJ3JhVsUhwJRH0jh3JF2883s80O80Oo3dhj8IJIHFWh00Ff900fWhLhHJj93J8j09J9O0S080uhK80uIN003IfK8RI00fUu8h8jF9s0N80883hH9k89O8j8IFHhSIrVILIOuwgJ0f0h0iou9H089U8290hh8i9I9o900wjsf89O0KU9w3J9020KJiJ0jVI398fJHuJJ4O4O0S9U0hkfJr9FU9f8I0Hhw990Ps4KgO3jPs8sjhF99PJuI08FIWu8fh0jPs03HjfSIK9dL0I8l0fr0N8Fh93u3F9I9skufuh80098gS008u90hiL0uUuII4Rd88JO8gLdsUKw9F9uwiljruLJ0jhiJi4j4F8U88irFjFRJUUhhFhsf0WFu0r40FLhUu9IFWhKHU0P9ji093s0Hj9HiUo00UhfO39N0jShUP900uUdj8i08g0IhIIFijN9DUH3NUHIRLiS9L3ufuU9r0i90VDoL03ODi2lu9hfNl34jFRLKk8JDw83LIh0l902h90LOf80gfK88D9I9fuJLJ9Uu09H3sN0wOhF8kh3IL8hujw9U9F8JdhP9f0j03KUNiVU8IUR8wfhwNd0RIs8J8hrJ0lh8uS0IoOH0ffjR8SUu8H2U00829J0FhRu90h8Olf889Okhr9g9030Nh00grWho93JwuFSoJ9iJRfu8D9hK2JJukURPK29NfhDIHOu28hFjJOSN4NLHgV0Hsu0W0K9jLhfHfFOi0I49u3Ps9d00Kujh8N2hNhLlJNwLwSH098hu0PIFf0j2FLs0I9sNF0HiRi8oUjFjiI8fKNWsUuJO0hIOiuKwhNNj9VOr9P2U8UuuFJ0HwKHjr90PihHWjrRh982fhLR0KFwOj092FdDNiNKhi0j8'

# instantiating dynamodb client
session = boto3_session(region_name='eu-west-1', profile_name='perf')
dynamodb = session.client('dynamodb')

timestamp = strftime("%Y-%m-%dT-%H:%M")
results = open('dynamodb-put-results_{}.txt'.format(timestamp), 'a')
count = 0

for sid in session_ids:
    count += 1
    gen_data = ''.join(sample(generated_string, len(generated_string)))
    sleep(1)

    response = dynamodb.put_item(
        TableName='ddb-perf-testing',
        Item={
            'session_id': {'S': sid },
            'data': {'S': gen_data },
            'item_num': {'S': str(count) }
        }
    )

    results.write('Call Number: {call_num} \n'.format(call_num=count))
    results.write('Call ResponseMetadata: {metadata} \n\n'.format(metadata=response['ResponseMetadata']))

results.close()

Read from the Table(s):

  • Read 18KB per second for 3 Hours:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
from boto3 import Session as boto3_session
from time import sleep, strftime
from random import choice

# delay between each iteration
iteration_delay = 1

# iterations number - 3 hours
iterations = 10800

# session ids that will be fetched in a random.choice order
session_ids = [
    '77c81e29-c86a-411e-a5b3-9a8fb3b2595f',
    'b9a2b8ee-17ab-423c-8dbc-91020cd66097',
    'cbe01734-c506-4998-8727-45f1aa0de7e3',
    'e789f69b-420b-4e6d-9095-cd4482820454',
    'c808a4e6-311e-48d2-b3fd-e9b0602a16ac',
    '2ddf0416-6206-4c95-b6e5-d88b5325a7b1',
    'e8157439-95f4-49a9-91e3-d1afc60a812f',
    'f032115b-b04f-423c-9dfe-e004445b771b',
    'dd6904c5-b65b-4da4-b0b2-f9e1c5895086',
    '075e59be-9114-447b-8187-a0acf1b2f127'
]

# instantiating dynamodb client
session = boto3_session(region_name='eu-west-1', profile_name='perf')
dynamodb = session.client('dynamodb')
dynamodb-table = 'ddb-perf-testing'

timestamp = strftime("%Y-%m-%dT-%H:%M")
results = open('dynamodb-results_{}.txt'.format(timestamp), 'a')

for iteration in range(iterations):
    count = iteration + 1
    print(count)
    sleep(iteration_delay)

    response = dynamodb.get_item(
        TableName=dynamodb-table,
        Key={'session_id': {'S': choice(session_ids)}},
        ConsistentRead=False
    )

    results.write('Call Number: {cur_iter}/{max_iter} \n'.format(cur_iter=count, max_iter=iterations))
    results.write('Call Item Response => Key: {attr_id}, Key Number:{attr_num} \n'.format(attr_id=response['Item']['session_id']['S'], attr_num=response['Item']['item_num']['S']))
    results.write('Call ResponseMetadata: {metadata} \n\n'.format(metadata=response['ResponseMetadata']))

results.close()

Results

Notes from AWS Support:

Reasons for High Latencies:

  • RequestLatency is a Server Side Metric
  • Long requests could relate to metadata lookups
  • Executing Relative Low Amount of Requests there is Frequent Metadata Lookups; This may cause a spike in latency
  • Consistent Requests can have higher average latency then Eventual Consistent Reads
  • Requests in general can encounter higher then normal latency at times, due to network issue, storage node issue, metadata issue.
  • The p90 should still be single digit
  • Using Encryption has to interact with KMS API as well (mechanisms in place to deal with KMS integration though to still offer p90 under 10 ms)
  • DAX: Strongly consistent reads will be passed on to DynamoDB and not handled by the cache
  • 1 RCU reading in Eventual Consistent manner can read 8 kb
  • Consistent read costs double an eventual consistent read
  • DDB not 100% of requests will be under 10 ms

Resources: - https://aws.amazon.com/blogs/developer/tuning-the-aws-sdk-for-java-to-improve-resiliency/ - https://aws.amazon.com/blogs/developer/enabling-metrics-with-the-aws-sdk-for-java/ - https://en.wikipedia.org/wiki/Eventual_consistency

Give Your Database a Break and Use Memcached to Return Frequently Accessed Data

So let’s take this scenario:

Your database is getting hammered with requests and building up some load over time and we would like to place a caching layer in front of our database that will return data from the caching layer, to reduce some traffic to our database and also improve our performance for our application.

The Scenario:

Our scenario will be very simple for this demonstration:

  • Database will be using SQLite with product information (product_name, product_description)
  • Caching Layer will be Memcached
  • Our Client will be written in Python, which checks if the product name is in cache, if not a GET_MISS will be returned, then the data will be fetched from the database, returns it to the client and save it to the cache
  • Next time the item will be read, a GET_HIT will be received, then the item will be delivered to the client directly from the cache

SQL Database:

As mentioned we will be using sqlite for demonstration.

Create the table, populate some very basic data:

1
2
3
4
5
6
7
8
$ sqlite3 db.sql -header -column
import sqlite3 as sql
SQLite version 3.16.0 2016-11-04 19:09:39
Enter ".help" for usage hints.

sqlite> create table products (product_name STRING(32), product_description STRING(32));
sqlite> insert into products values('apple', 'fruit called apple');
sqlite> insert into products values('guitar', 'musical instrument');

Read all the data from the table:

1
2
3
4
5
6
sqlite> select * from products;
product_name  product_description
------------  -------------------
apple         fruit called apple
guitar        musical instrument
sqlite> .exit

Run a Memcached Container:

We will use docker to run a memcached container on our workstation:

1
$ docker run -itd --name memcached -p 11211:11211 rbekker87/memcached:alpine

Our Application Code:

I will use pymemcache as our client library. Install:

1
2
$ virtualenv .venv && source .venv/bin/activate
$ pip install pymemcache

Our Application Code which will be in Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import sqlite3 as sql
from pymemcache.client import base

product_name = 'guitar'

client = base.Client(('localhost', 11211))
result = client.get(product_name)

def query_db(product_name):
    db_connection = sql.connect('db.sql')
    c = db_connection.cursor()
    try:
        c.execute('select product_description from products where product_name = "{k}"'.format(k=product_name))
        data = c.fetchone()[0]
        db_connection.close()
    except:
        data = 'invalid'
    return data

if result is None:
    print("got a miss, need to get the data from db")
    result = query_db(product_name)
    if result == 'invalid':
        print("requested data does not exist in db")
    else:
        print("returning data to client from db")
        print("=> Product: {p}, Description: {d}".format(p=product_name, d=result))
        print("setting the data to memcache")
        client.set(product_name, result)

else:
    print("got the data directly from memcache")
    print("=> Product: {p}, Description: {d}".format(p=product_name, d=result))

Explanation:

  • We have a function that takes a argument of the product name, that makes the call to the database and returns the description of that product
  • We will make a get operation to memcached, if nothing is returned, then we know the item does not exists in our cache,
  • Then we will call our function to get the data from the database and return it directly to our client, and
  • Save it to the cache in memcached so the next time the same product is queried, it will be delivered directly from the cache

The Demo:

Our Product Name is guitar, lets call the product, which will be the first time so memcached wont have the item in its cache:

1
2
3
4
5
$ python app.py
got a miss, need to get the data from db
returning data to client from db
=> Product: guitar, Description: musical instrument
setting the data to memcache

Now from the output, we can see that the item was delivered from the database and saved to the cache, lets call that same product and observe the behavior:

1
2
3
$ python app.py
got the data directly from memcache
=> Product: guitar, Description: musical instrument

When our cache instance gets rebooted we will lose our data that is in the cache, but since the source of truth will be in our database, data will be re-added to the cache as they are requested. That is one good reason not to rely on a cache service to be your primary data source.

What if the product we request is not in our cache or database, let’s say the product tree

1
2
3
$ python app.py
got a miss, need to get the data from db
requested data does not exist in db

This was a really simple scenario, but when working with masses amount of data, you can benefit from a lot of performance using caching.

Resources:

Dockerizing a Memcached Server for Docker on Alpine

This post I will demostrate how to dockerize a memcached server on Alpine and how to create a boot script that allows you to pass environment variables through to the application.

What is Memcached

Memcached is a multi-threaded, in-memory key/value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, etc. More on Memcached

The Dockerfile:

Our Dockerfile will consist of a simple install of memcached and add a boot script that we will start it from:

1
2
3
4
5
6
7
FROM alpine:3.7

COPY boot.sh /boot.sh
RUN apk --no-cache add memcached && chmod +x /boot.sh

USER memcached
CMD ["/boot.sh"]

The Boot Script:

As you can see we have set defaults so when the user does not specify any environment variables, that it will inherit the default values

1
2
3
4
5
6
7
8
9
10
11
#!/bin/sh

/usr/bin/memcached \
  --user=${MEMCACHED_USER:-memcached} \
  --listen=${MEMCACHED_HOST:-0.0.0.0} \
  --port=${MEMCACHED_PORT:-11211} \
  --memory-limit=${MEMCACHED_MEMUSAGE:-64} \
  --conn-limit=${MEMCACHED_MAXCONN:-1024} \
  --threads=${MEMCACHED_THREADS:-4} \
  --max-reqs-per-event=${MEMCACHED_REQUESTS_PER_EVENT:-20} \
  --verbose

Build and Deploy:

Build the image, if you just want to run the container you can use my public image in the next step:

1
$ docker build -t local/memcached:0.1 .

Run the Memcached Container:

1
$ docker run -itd --name memcached -p 11211:11211 -e MEMCACHED_MEMUSAGE=32 local/memcached:0.1

Or my Public Image from Docker Hub:

1
$ docker run -itd --name memcached -p 11211:11211 -e MEMCACHED_MEMUSAGE=32 rbekker87/memcached:alpine

Check out the Stats:

Pass the command stats through the exposed port:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
$ echo -e "stats" | nc localhost 11211
STAT pid 8
STAT uptime 2
STAT time 1535833177
STAT version 1.5.6
STAT libevent 2.1.8-stable
STAT pointer_size 64
STAT rusage_user 0.030000
STAT rusage_system 0.000000
STAT max_connections 1024
STAT curr_connections 1
STAT total_connections 2
STAT rejected_connections 0
STAT connection_structures 2
STAT reserved_fds 20
STAT cmd_get 0
STAT cmd_set 0
STAT cmd_flush 0
STAT cmd_touch 0
STAT get_hits 0
STAT get_misses 0
STAT get_expired 0
STAT get_flushed 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 6
STAT bytes_written 0
STAT limit_maxbytes 33554432
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT time_in_listen_disabled_us 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT slab_reassign_rescues 0
STAT slab_reassign_chunk_rescues 0
STAT slab_reassign_evictions_nomem 0
STAT slab_reassign_inline_reclaim 0
STAT slab_reassign_busy_items 0
STAT slab_reassign_busy_deletes 0
STAT slab_reassign_running 0
STAT slabs_moved 0
STAT lru_crawler_running 0
STAT lru_crawler_starts 255
STAT lru_maintainer_juggles 155
STAT malloc_fails 0
STAT log_worker_dropped 0
STAT log_worker_written 0
STAT log_watcher_skipped 0
STAT log_watcher_sent 0
STAT bytes 0
STAT curr_items 0
STAT total_items 0
STAT slab_global_page_pool 0
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0
STAT reclaimed 0
STAT crawler_reclaimed 0
STAT crawler_items_checked 0
STAT lrutail_reflocked 0
STAT moves_to_cold 0
STAT moves_to_warm 0
STAT moves_within_lru 0
STAT direct_reclaims 0
STAT lru_bumps_dropped 0
END

Some descriptions:

evictions - when items are evicted from the cache total_items - the number of items the server has stored since it was started current_items - the number of items in the cache bytes - the current number of bytes used to store items limit_maxbytes - the number of bytes the server is allowed to use for storage get_misses - the number of times a item has been requested, but not found get_hits - the number of times a item has been served from the cache

To get specific stats, like evictions:

1
2
$ echo -e "stats" | nc localhost 11211 | grep -w evictions
STAT evictions 0

When you see evictions value increases, this essentially means that memcache had to remove the oldest items from memory for new or more frequent used items. If this number remains high, consider increasing your memory allocated to memcache.

Slab Stats: returns information about each of the slabs created by memcached during runtime:

1
2
3
$ echo -e "stats slabs" | nc localhost 11211
STAT active_slabs 0
STAT total_malloced 0

active_slabs - Total number of slab classes allocated. total_malloced - Total amount of memory allocated to slab pages.

For detailed description about statistics, have a look at their github resource: - https://github.com/memcached/memcached/blob/master/doc/protocol.txt

Resources:

Review and Secure Your Facebook Account

This post is a bit different from my other posts, but I feel it’s a important one: Facebook Security.

Facebook, everyone loves it right? Yeah, but what happens when you get locked out of your account, or an attacker gains access to your account and start doing things that you dont want to, and especially all the photos / messages that needs to remain private, can potentially end up in the wrong hands.

Facebook usually detects strange behavior, but being able to be pro-active on security on this can help a lot.

There’s a couple of ways how attackers can gain access to your account, but I won’t go into that, google will be your friend if you are curious how they do it.

Scenario: Something suspicious is up / weird behavior / getting unusual messages from groups etc

Usually Facebook will detect this, but if not you can and should do the following:

  • Reset your password
  • Enable Two-Factor Authentication
  • Terminate or Logout all sessions from your account, if you find unknown sessions, report it to facebook and log them out.
  • Review your account’s activity
  • Review Group Activity, if you are subscribed to groups, unsubscribe
  • Reach out to facebook support

Head over to your Facebook Accounts Settings Page:

Head over to https://www.facebook.com/settings , this will be the main view where you are able to configure/review your account. It should look like this:

When you select the “Security and Login” tab: https://www.facebook.com/settings?tab=security , you will be presented with a couple of login options:

Security and Login Info:

The list of your devices that is currently logged on:

Hit the See More dropdown to review all your devices, which is currently logged onto Facebook, if you are not aware of the sessions, hit the Log Out button to terminate that session, or select Not You if you are not aware of that session, then continue to report the activity to Facebook, so that they can look into it.

You can follow up on your incident via https://www.facebook.com/support/ .

Password and Two Factor Authentication:

This is actually the first thing that I would do, is to change your password. If someone did manage to gain access to your password and you are still logged on, change it immediately. If they reset your password before you do, game over. Well kind of..

From the same page, change your password:

Enable "Two-Factor Authentication", when you are logged out, or trying to logon from a new device, a notification will be sent to your device where Facebook is installed, or alternatively, you will receive a code sent to you which you will need to enter after you have logged on, just to provide you with a extra layer of security.

Enable "Get alerts about unrecognized logins", which allows you to set up to 5 friends that can help you unlock your account, if your account has been locked out.

Review your Activity Log

From https://www.facebook.com/settings?tab=your_facebook_information , head over to "Activity Log":

Select "Activity Log", to review your recent activity:

Below Comments, select more, then Security and Login Information:

Then we will be presented with the Active Sessions, Login and Logouts and Recognized Devices.

First look at Active Sessions:

Then Logins and Logouts:

From this same page you can review other activity like Search History, Groups. etc.

If someone had to access/subscribed to groups, you will be able to review the activity, within 3 different views:

  • Groups: any interaction with groups, such as likes, comments etc.
  • Membership Activity: Any group memberships
  • Posts and Comments: Self explanatory.

Final Note:

People try to access accounts all the time, watch out for the following:

  • Friend Requests: people have a lot of private information on facebook, keep it private
  • Watch out for strange applications that wants your permission, review the permission levels closely
  • Reset your password time to time, use unique passwords, and not the same password as the password that your main email account is associated with
  • Watch out for links, some of them can end you up in a bad spot.
  • When you see weird activity from your friends account, report it, so that facebook can investigate it. It happened to a friend and Facebook sorted it out within 20 minutes.

Resources:

Distributing a Shared Secret Amongst a Group of Participants Using Shamirs Secret Sharing Scheme Aka Ssss

In situations where a group of participants join together to split up a secret in a form of secret sharing, where the secret is devided into parts, giving each participant their own unique part. Together contributing to reconstruct the initial secret. We can achieve this with Shamir’s Secret Sharing which is an algorithm in cryptography created by Adi Shamir.

More info on Secret Sharing

Referenced from Wikipedia: Secret Sharing:

“Secret sharing (also called secret splitting) refers to methods for distributing a secret amongst a group of participants, each of whom is allocated a share of the secret. The secret can be reconstructed only when a sufficient number, of possibly different types, of shares are combined together; individual shares are of no use on their own.”

Installing ssss

On Mac OSX:

1
$ brew install ssss

On Debian:

1
$ apt install ssss -y

Creating a Secret:

Generate a Secret where we will distribute 5 shares, where each participant will have their own unique share, and to reconstruct the secret, we will need 3 participants to rebuild the secret. In our case our shares will be distributed to the following example users:

1
2
3
4
5
- Share 1: James
- Share 2: John
- Share 3: Frank
- Share 4: Paul
- Share 5: Ryan

For this demonstration our secret’s value will be SuperSecret@123!, which we will split into 5 shares, but to reconstruct, we need 3 parts / shares:

1
2
3
4
5
6
7
8
$ ssss-split -t 3 -n 5
Generating shares using a (3,5) scheme with dynamic security level.
Enter the secret, at most 128 ASCII characters: Using a 128 bit security level.
1-41ac84013bf568d1cc88b751539f1ff5
2-7d9ca3ca26442bfcca35e0ad205e5659
3-519038837bbf1b7ceefde331ad1ae40f
4-6d4f4e0f086af5be033f516bb3e227d2
5-4143d5465591c53e27f752f73ea69596

In this case, each share will be distributed to each user to save in a secure location.

Reconstructing the Secret:

Let’s reconstruct the secret, and as we need 3 participants, we will ask John, Paul and Ryan for their shares, so that we can reconstruct the secret:

1
2
3
4
5
6
$ ssss-combine -t 3
Enter 3 shares separated by newlines:
Share [1/3]: 2-7d9ca3ca26442bfcca35e0ad205e5659
Share [2/3]: 4-6d4f4e0f086af5be033f516bb3e227d2
Share [3/3]: 5-4143d5465591c53e27f752f73ea69596
Resulting secret: SuperSecret@123!

As you can see the secret is verified the same as the initial secret.

Using ssss and qrencode for MFA Codes

This can be useful for Multi Factor Authentication as well. Setup a Virtual MFA with a Identity that supports MFA Authentication, copy or make note of the “Secret Key / Secret Configuration Key”, go ahead and setup the MFA Device on your MFA Device to complete the setup.

Once verified and able to logon, logout and delete the MFA Account from your Device.

Generate the same share scheme for the MFA Secret Key, for this example: ABCDEXAMPLE1029384756:

1
2
3
4
5
6
7
8
$ ssss-split -t 3 -n 5
Generating shares using a (3,5) scheme with dynamic security level.
Enter the secret, at most 128 ASCII characters: Using a 168 bit security level.
1-8d2cf979fb346297cab47ff691bddc1c5a5f34af37
2-4d0f2cdcfff653cc60a4f293c15805f7e84b0a956d
3-dadb6d2cbe42772c9a9042273f0b71dd71422f19cb
4-546bcef428151ceb01fdc6007ac2e5e4f1516670ca
5-c3bf8f0469a1380bfbc976b4849191ce685843fc7e

Distribute the Shares, and when the MFA Device needs to be restored on a Device, reconstruct the secret to get the Secret Key for the MFA Device:

1
2
3
4
5
6
$ ssss-combine -t 3
Enter 3 shares separated by newlines:
Share [1/3]: 1-8d2cf979fb346297cab47ff691bddc1c5a5f34af37
Share [2/3]: 2-4d0f2cdcfff653cc60a4f293c15805f7e84b0a956d
Share [3/3]: 3-dadb6d2cbe42772c9a9042273f0b71dd71422f19cb
Resulting secret: ABCDEXAMPLE1029384756

Now that we have the Secret Key for our MFA Device, let’s Generate a QRCode that we can scan in from our device, which will save us from typing a lot of characters. We will need qrencode for this:

For Mac OSX:

1
$ brew install qrencode

for Debian:

1
$ apt install qrencode -y

To generate the QRCode, we will pass the filename: myqrcode.png, the name that will appear on our device: MyNewMFADevice, and the Secret: ABCDEXAMPLE1029384756:

1
$ qrencode -o myqrcode.png -d 300 -s 10 "otpauth://totp/MyNewMFADevice?secret=ABCDEXAMPLE1029384756"

You will find the myqrcode.png in your current working directory, open the file scan the barcode and your MFA device will be setup and enabled to use.

Resources:

Improving Performance From Your Lambda Function From the Use of Global Variables

When using Lambda and DynamoDB, you can use global variables to gain performance when your data from DynamoDB does not get updated that often, and you would like to use caching to prevent a API call to DynamoDB everytime your Lambda Function gets invoked.

You can use external services like Redis or Memcached when you would like to verify that each invocation is as true as your source of truth which will be DynamoDB. Then your application logic can work with caching.

But in this case we just want a simple piece of code that can keep the state for the remaining time that the function is running on that underlying container. I am not 100% sure, but I have seen that the data can be cached for up to 60 minutes. This can be a total mess when your data gets updated regularly, then I would set all my calls in functions, as the global variables keeps their state for some time.

Example Function:

This function gets data from DynamoDB, iterates through a small dataset (10 Items), and appends each group name to my list which is the value of my groups key inside my dictionary.

Due to my global variable mydata, you will see that the first invocation will result in a API call to DynamoDB as the length of my mydata["groups"] being 0, the second invocation, the data will exist inside my global variable, therefore I am returning the data directly from my variable.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import boto3, json

client = boto3.resource('dynamodb', region_name='eu-west-1')
tbl = client.Table('my-dynamo-table')

mydata = {}
mydata["groups"] = []

def lambda_handler(event, context):
    if len(mydata["groups"]) == 0:
        # data is not cached, make call to dynamo
        data = tbl.scan()
        group_data = data['Items']

        for group in group_data:
            mydata["groups"].append(group['name'])
        return mydata

    else:
        # return cached content
        return mydata

Results of my Invocations:

The first call that I made:

The second call that I made:

If you need a small layer of caching that can improve your latency, this can be used. But if you need your data to be accurate from every call, rather looking into a different approach and external caching services.

Resources:

Take advantage of Execution Context reuse to improve the performance of your function.:

“Make sure any externalized configuration or dependencies that your code retrieves are stored and referenced locally after initial execution. Limit the re-initialization of variables/objects on every invocation. Instead use static initialization/constructor, global/static variables and singletons. Keep alive and reuse connections (HTTP, database, etc.) that were established during a previous invocation.”

Send Emails Using Python and Sendgrid Using SMTPlib

Quick tutorial on how to send emails using Python and smtplib.

Sendgrid

Sendgrid offers 100 free outbound emails per day, sign up with them via sendgrid.com/free, create a API Key and save your credentials in a safe place.

You first need to verify your account by sending a mail using their API, but it’s step by step so won’t take more than 2 minutes to complete.

Python Code

Once the verification is completed, our Python Code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText

mail_from = 'Ruan Bekker <ruan@ruanbekker.com>'
mail_to = 'Ruan Bekker <xxxx@gmail.com>'

msg = MIMEMultipart()
msg['From'] = mail_from
msg['To'] = mail_to
msg['Subject'] = 'Sending mails with Python'
mail_body = """
Hey,

This is a test.

Regards,\nRuan

"""
msg.attach(MIMEText(mail_body))

try:
    server = smtplib.SMTP_SSL('smtp.sendgrid.net', 465)
    server.ehlo()
    server.login('apikey', 'your-api-key')
    server.sendmail(mail_from, mail_to, msg.as_string())
    server.close()
    print("mail sent")
except:
    print("issue")

When I ran the code, I received the mail, and when you inspect the headers you can see that the mail came via sendgrid:

1
Received: from xx.xx.s2shared.sendgrid.net (xx.xx.s2shared.sendgrid.net. [xx.xx.xx.xx])

Resources:

Great post on SSL / TLS: - https://stackabuse.com/how-to-send-emails-with-gmail-using-python/

Enjoy :D

Using IAM Authentication With Amazon Elasticsearch Service

Today I will demonstrate how to allow access to Amazons Elasticsearch Service using IAM Authenticationi using AWS Signature Version4.

Elasticsearch Service Authentication Support:

When it comes to security, Amazons Elasticsearch Service supports three types of access policies:

  • Resource Based
  • Identity Based
  • IP Access Based

More information on this can be found below: - https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html

Securing your Amazon Elasticsearch Search Domain:

To secure your domain with IAM Based Authentication, the following steps will be neeed:

  • Create IAM Policy to be associated with a IAM User or Role
  • On Elasticsearch Access Policy, associate the ARN to the Resource
  • Use the AWS4Auth package to sign the requests as AWS supports Signature Version 4
1
2
3
4
5
6
7
8
9
10
11
12
13
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "es:*"
            ],
            "Resource": "arn:aws:es:eu-west-1:<ACCOUNT-ID>:domain/<ES-DOMAIN>"
        }
    ]
}

Create the IAM Role with EC2 Identity Provider as a Trusted Relationship eg. es-role and associate the IAM Policy es-policy to the role.

Create/Moodify the Elasticsearch Access Policy, in this example we will be using a combination of IAM Role, IAM User and IP Based access:

  • IAM Role for EC2 Role Based Services
  • IAM User for User/System Account
  • IP Based for cients that needs to be whitelisted via IP (ip-based just for demonstration, as the tests will be used only for IAM)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT-ID>:role/<IAM-ROLE-NAME>",
          "arn:aws:iam::<ACCOUNT-ID>:user/<IAM-USER-NAME>"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:<ACCOUNT-ID>:domain/<ES-DOMAIN>/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:<ACCOUNT-ID>:domain/<ES-DOMAIN>/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "x.x.x.x",
            "x.x.x.x"
          ]
        }
      }
    }
  ]
}

After the Access Policy has been updated, the Elasticsearch Domain Status will show Active

Testing from EC2 using IAM Instance Profile:

Launch a EC2 Instance with the IAM Role eg. es-role, then using Python, we will make a request to our Elasticsearch Domain using boto3, aws4auth and the native elasticsearch client for python via our IAM Role, which we will get the temporary credentials from boto3.Session.

Installing the dependencies:

1
2
3
4
$ pip install virtualenv
$ virtualenv .venv
$ source .venv/bin/activate
$ pip install boto3 elasticsearch requests_aws4auth

Our code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import boto3, json
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

my_region = 'eu-west-1'
my_service = 'es'
my_eshost = 'search-replaceme.eu-west-1.es.amazonaws.com'

session = boto3.Session(region_name=my_region) # thanks Leon
credentials = session.get_credentials()
credentials = credentials.get_frozen_credentials()
access_key = credentials.access_key
secret_key = credentials.secret_key
token = credentials.token

aws_auth = AWS4Auth(
    access_key,
    secret_key,
    my_region,
    my_service,
    session_token=token
)

es = Elasticsearch(
    hosts = [{'host': my_eshost, 'port': 443}],
    http_auth=aws_auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(json.dumps(es.info(), indent=2))

Running our piece of code, will result in this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ python get-info-from-role.py
{
  "cluster_name": "<ACCOUNT-ID>:<ES-DOMAIN>",
  "cluster_uuid": "sLUnqFSsQdCMlBLrn7BTUA",
  "version": {
    "lucene_version": "6.6.0",
    "build_hash": "Unknown",
    "build_snapshot": false,
    "number": "5.5.2",
    "build_date": "2017-10-18T04:35:01.381Z"
  },
  "name": "KXSwBvT",
  "tagline": "You Know, for Search"
}

Testing using IAM Credentials from Credentials Provider:

Configure your credentials provider:

1
2
3
4
5
6
$ pip install awscli
$ aws configure --profile ruan
AWS Access Key ID [None]: xxxxxxxxx
AWS Secret Access Key [None]: xxxxxx
Default region name [None]: eu-west-1
Default output format [None]: json

Using Python, we will get the credentials from the Credential Provider, using our profile name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import boto3, json
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

my_service = 'es'
my_region = 'eu-west-1'
my_eshost = 'search-replaceme.eu-west-1.es.amazonaws.com'

session = boto3.Session(
    region_name=my_region,
    profile_name='ruan'
)

credentials = session.get_credentials()
access_key = credentials.access_key
secret_key = credentials.secret_key

aws_auth = AWS4Auth(
    access_key,
    secret_key,
    my_region,
    my_service
)

es = Elasticsearch(
    hosts = [{'host': my_eshost, 'port': 443}],
    http_auth=aws_auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(json.dumps(es.info(), indent=2))

Running it will result in:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ python get-info-from-user.py
{
  "cluster_name": "<ACCOUNT-ID>:<ES-DOMAIN>",
  "cluster_uuid": "sLUnqFSsQdCMlBLrn7BTUA",
  "version": {
    "lucene_version": "6.6.0",
    "build_hash": "Unknown",
    "build_snapshot": false,
    "number": "5.5.2",
    "build_date": "2017-10-18T04:37:21.381Z"
  },
  "name": "KXSwBvT",
  "tagline": "You Know, for Search"
}

For more blog posts on Elasticsearch have a look at: - blog.ruanbekker.com:elasticsearch - sysadmins.co.za:elasticsearch

Get Blogpost Titles Links and Tags From a RSS Link Using Python Feedparser

I wanted to get metadata from my other blog sysadmins.co.za, such as each post’s title, link and tags using the RSS link. I stumbled upon feedparser, where I will use it to scrape all the posts details from the link and append it to a list, which I can then use to ingest it into a database or something like that.

Installing Dependencies:

Install feedparser and requests:

1
$ pip install feedparser requests

The Python Code:

I’m not too sure at this point how to get pagination going, so I’ve set a range to check, and if a status code of 200 is received, it will check if the title is in the list that I defined, if not, it will append it to the list.

At the end of the loop, the script will return the list that was defined, which will provide the info mentioned earlier:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import feedparser
import time
import requests

rss_url = "https://sysadmins.co.za/rss"

posts = []

def get_posts_for_ghost(rss_url):
    response = feedparser.parse(rss_url)
    for each in response['entries']:
        if each['title'] in [x['title'] for x in posts]:
            pass
        else:
            posts.append({
                "title": each['title'],
                "link": each['links'][0]['href'],
                "tags": [x['term'] for x in each['tags']],
                "date": time.strftime('%Y-%m-%d', each['published_parsed'])
            })
    return posts

count = 12

for x in range(count):
    if requests.get("{0}/{1}/".format(rss_url, count)).status_code == 200:
        print("get succeeded, count at: {}".format(count))
        get_posts_for_ghost("{0}/{1}/".format(rss_url, count))
        count -= 1
    else:
        print("got 404, count at: {}".format(count))
        count -= 1

    get_posts_for_ghost(rss_url)

print(posts)

Running the script:

Running the script will look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ python rssfeed.py
got 404, count at: 12
got 404, count at: 11
got 404, count at: 10
get succeeded, count at: 9
get succeeded, count at: 8
get succeeded, count at: 7
get succeeded, count at: 6
get succeeded, count at: 5
get succeeded, count at: 4
get succeeded, count at: 3
get succeeded, count at: 2
get succeeded, count at: 1
[
  {
    'title': 'Tutorial on DynamoDB using Bash and the AWS CLI Tools to Interact with a Music Dataset',
    'link': 'https://sysadmins.co.za/tutorial-on-dynamodb-using-bash-and-the-aws-cli-tools-to-interact-with-a-music-dataset/',
    'tags': ['DynamoDB', 'Bash', 'AWS'],
    'date': '2018-08-15'
  },
  {
    'title': 'Setup a PPTP VPN on Ubuntu 16',
    'link': 'https://sysadmins.co.za/setup-a-pptp-vpn-on-ubuntu-16/',
    'tags': ['Networking', 'VPN'],
    'date': '2018-06-27'
  },
  ...