In this post we will setup a Python Flask Application which includes the APM Agent which will collect metrics, that gets pushed to the APM Server. If you have not setup the Elastic Stack with / or APM Server, you can follow this post to setup the needed.
Then we will make a bunch of HTTP Requests to our Application and will go through the metrics per request type.
Our Application will have the following Request Paths:
/- Returns static text
/delay- random delays to simulate increased response latencies
/upstream- get data from a upstream provider, if statements to provide dummy 200, 404 and 502 reponses to visualize
/5xx- request path that will raise an exception so that we can see the error via apm
/sql-write- inserts 5 rows into a sqlite database
/sql-read- executes a select all from the database
/sql-group- sql query to group all the cities and count them
This is just simple request paths to demonstrate the metrics via APM (Application Performance Monitoring) on Kibana.
Install Flask and APM Agent
Create a virtual environment and install the dependencies:
1 2 3 4 5
For more info on APM Configuration.
Instrument a Bare Bones Python Flask app with APM:
A Barebones app with APM Configured will look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13
This will provide metrics on the
/ request path. In order to trace transaction ids from the metrics, we need to configure the index on Kibana. To do this, head over to Kibana, Management, Index Patterns, Add Index Pattern,
@timestamp as the time filter field name.
This will allow you to see the data when tracing the transaction id’s via the Discover UI.
Create the Python Flask App
Create the Flask App with the request paths as mentioned in the beginning:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76
Run the app:
At this point, we wont have any data on APM as we need to make requests to our application. Let’s make 10 HTTP GET Requests on the
/ Request Path:
Visualize the Root Request Path
Head over to Kibana, Select APM and you will see something similar like below when selecting the timepicker to 15 minutes at the right top corner. This page will give you the overview of all your configured applications and the average response times over the selected time, transactions per minute, errors per minute etc:
When you select your application, you will find the graphs on you response times and requests per minute, also a breakdown per request path:
When selecting the request path, in this case
GET /, you will find a breakdown of metrics only for that request and also the response time distribution for that request path, if you select frame from the response time distribution, it will filter the focus to that specific transaction.
When you scroll a bit down to the Transaction Sample section, you will find data about the request, response, system etc:
From the Transaction Sample, you can select the View Transaction in Discover button, which will trace that transaction id on the Discover UI:
Increasing the http curl clients running simultaneously from different servers and increasing the time for 15 minutes to have more metrics will result in the screenshot below, notice the 6ms response time can easily be traced selecting it in the response time distribution, then discovering it in the UI, which will give you the raw data from that request:
Viewing Application Errors in APM
Make a couple of requests to
Navigate to the app, select Errors, then you will see the exception details that was returned. Here we can see that in our code we tried to concatenate integers with strings:
Furthermore we can select that error and it will provide us a direct view on where in our code the error gets generated:
Pretty cool right?! You can also further select the library frames, which will take you to the lower level code that raised the exception. If this errors can be drilled down via the discover ui, to group by source address etc.
Simulate Response Latencies:
Make a couple of requests to the
/delay request path, and you should see the increased response times from earlier:
Requests where Database Calls are Executed
The while loop to call random request paths:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
When we look at our applications performance monitoring overview, we can see the writes provide more latencies as the group by’s:
/sql-write request overview:
When selecting a transaction sample, we can see the timeline of each database call:
When looking at the
/sql-group request overview, we can see that the response times increasing overtime, as more data is written to the database, it takes longer to read and group all the data from the database:
The transaction details shows the timeline of the database query from that request:
When you select the database select query on the timeline view, it will take you to the exact database query that was executed:
When we include a database call with a external request to a remote http endpoint, we will see something like:
Viewing 4xx and 5xx Response Codes
From the application code we are returning 2xx, 4xx, and 5xx response codes for this demonstration to visualize them:
Configuring more Applications
Once more apps are configured, and they start serving traffic, they will start appearing on the APM UI as below:
APM is available for other languages as well and provides a getting started snippets from the APM UI. For more information on APM, have a look at their Documentation
Hope this was useful.