With k3d we can mount the host to container path, and with persistent volumes we can set a hostPath for our persistent volumes. With k3d, all the nodes will be using the same volume mapping which maps back to the host.
We will test the data persistence by writing a file inside a container, kill the pod, then exec into the pod again and test if the data persisted
The k3d Cluster
Create the directory on the host where we will persist the data:
In this post we will explore how to use asynchronous functions in OpenFaas.
What are we doing
A synchronous request blocks the client until operation completes, where a asynchronous request doesn’t block the client, which is nice to use for long-running tasks or function invocations to run in the background through the use of NATS Streaming.
We will be building a Python Flask API Server which will act as our webhook service. When we invoke our function by making a http request, we also include a callback url as a header which will be the address where the queue worker will post it’s results.
Then we will make a http request to the synchronous function where we will get the response from the function and a http request to the asynchronous function, where we will get the response from the webhook service’s logs
Deploy OpenFaas on a k3d Kubernetes Cluster if you want to follow along on your laptop. You can follow this post to deploy a kubernetes cluster and deploying openfaas:
To explain what we are doing, we are bringing up localstack as a service container, then using the aws cli tools we point to the localstack kinesis endpoint, creating a kinesis stream, put 100 records to the stream, then we read from the stream and delete thereafter.
Trigger the Pipeline
Then I head to drone activate my new git repository and select the repository as “Trusted”. I commited a dummy file to trigger the pipeline and it should look like this:
Drone services allow you to run a service container and will be available for the duration of your build, which is great if you want a ephemeral service to test your applications against.
Today we will experiment with services on drone and will deploy a k3s (a kubernetes distribution built by rancher) cluster as a drone service and interact with our cluster using kubectl.
I will be using multiple pipelines, where we will first deploy our “dev cluster”, when it’s up, we will use kubectl to interact with the cluster, once that is done, we will deploy our “staging cluster” and do the same.
This is very basic and we are not doing anything special, but this is a starting point and you can do pretty much whatever you want.
What is Drone
If you are not aware of Drone, Drone is a container-native continious deliver platform built on Go and you can check them out here: github.com/drone
Setup Gitea and Drone
If you don’t have the stack setup, have a look at this post where I go into detail on how to get that setup.
Create your Git Repo
Go ahead and create a git repo, you can name it anything, then it should look something like this:
Create a drone configuration, .drone.yml my pipeline will look like this:
In this pipeline you can see that the staging pipeline depends on dev, so dev pipeline will start by creating the k3s service container, once its up I am using a step just to sleep for 30 seconds to allow it to boot.
Then I have defined a volume that will be persistent during the build time, which we will use to dump our kubeconfig file and update the hostname of our kubernetes endpoint. Once that is done our last step will set that file to the environment and use kubectl to interact with kubernetes.
Once our dev pipeline has finished, our staging pipeline will start.
Activate the Repo in Drone
Head over to drone on port 80 and activate the newly created git repo (and make sure that you select “Trusted”) and you will see the activity feed being empty:
Commit a dummy file to git and you should see your pipeline being triggered:
Once your pipeline has finished and everything succeeded, you should see the output of your nodes in your kubernetes service container:
As I mentioned earlier, we are not doing anything special but service containers allows us to do some awesome things.
Thank you for reading. If you like my content, feel free to visit me at ruan.dev or follow me on twitter at @ruanbekker
This post will show how to setup gitea and drone on a docker host with docker-compose. The drone example in this tutorial will be very basic, but in future posts I will focus more on pipeline examples using drone.
As this post I will use to link back for those who needs to setup the stack first.
Verify the environment variables and adjust the defaults if you want to change something, if you want your git clone ssh url to point to a dns name as well as the url for gitea, then change the following to your dns:
The default port for Gitea in this setup is port 3000:
Initial configuration will be pre-populated from our environment variables:
From the additional settings section, create your admin user (this user is referenced in our docker-compose as well)
Because I am using gitea as my hostname, you will be redirected to http://gitea:3000/user/login, if you don’t have a host entry setup for that it will fail, but you can just replace your servers ip in the request url and it will take you to the login screen, and after logging on, you should see this screen:
Access drone on port 80, you will be directed to the login screen:
Use the same credentials that you have used to sign up with gitea, and after logging on, you should see this:
If ever your login does not work, just delete the drone access token on gitea (gitea:3000/user/settings/applications)
Create a Git Repository
On gitea, create a new git repository:
You should now see your git repository:
Create a new file .drone.yml with the following content:
Commit the file in your git repository and head over to drone (which should be available on port 80) and select “Sync”, after a couple of seconds you should see the git repository:
Select “Activate” and “Activate Repository”, on the next screen select “Trusted”, verify that the configuration file name is the same as which we created, then select save:
Trigger the Build
If you click on “Activity Feed” you should see a empty feed. Head back to git and commit a dummy file to trigger the build to start. I will create a file name trigger with the value as 1 for my dummy file.
After committing the file, you will see on drone that the build started:
When we select the build, you can see we have a clone step and the step that we defined to echo “hello-world”:
This was a basic introduction for gitea and drone, but I will use this post in conjunction with more gitea examples in the future.
Today we will look at Thanos, a open source, highly available prometheus setup with long term storage capabilites, that we will run on docker to simplify the setup.
Note that running this proof of concept does not make it highly available as we will run everything on one host, but it will give you a feel what Thanos is about. In a future post, I will setup Thanos in a multi node environment.
If you are not familiar with Prometheus, then have a look at their documentation, but in short, prometheus is a open source monitoring system and time series database developed by soundcloud.
Prometheus is a monitoring system includes a rich, multidimensional data model, a concise and powerful query language called PromQL, an efficient embedded timeseries database, and over 150 integrations with third-party systems.
Thanos is a highly available prometheus setup with long term storage capabilities.
Thanos allows you to ship your data to S3/Minio for long storage capabilites, so you could for example only store your “live” data on prometheus for 2 weeks, then everything older than that gets sent to object storage such as amazon s3 or minio. This helps your prometheus instance not to be flooded with data or prevents you from running out of storage space. The nice thing is, when you query for data older than 2 weeks, it will fetch the data from object storage.
Thanos has a global query view, which essentially means you can query your prometheus metrics from one endpoint backed by multiple prometheus servers or cluster.
You can still use the same tools such as Grafana as it utilizes the same Prometheus Query API.
Thanos provides downsampling and compaction, so that you downsample your historical data for massive query speedup when querying large time ranges.
Thanos is a clustered system of components which can be categorized as follows:
Thanos provides two components that act as data sources: Prometheus Sidecar and Rule Nodes
Sidecar implements gRPC service on top of Prometheus
Rule Node directly implements it on top of the Prometheus storage engine it is running
Data sources that persist their data for long term storage, do so via the Prometheus 2.0 storage engine
Storage engine periodically produces immutable blocks of data for a fixed time range
A blocks top-level directory includes chunks, index and meta.json files
Chunk files hold a few hundred MB worth of chunks each
The index file holds all information needed to lookup specific series by their labels and the positions of their chunks.
The meta.json file holds metadata about block like stats, time range, and compaction level
A Store Node acts as a Gateway to block data that is stored in an object storage bucket
It implements the same gRPC API as Data Sources to provide access to all metric data found in the bucket
Continuously synchronizes which blocks exist in the bucket and translates requests for metric data into object storage requests
Implements various strategies to minimize the number of requests to the object storage
Prometheus 2.0 storage layout is optimized for minimal read amplification
At this time of writing, only index data is cached
Stores and Data Sources are the same, store nodes and data sources expose the same gRPC Store API
Store API allows to look up data by a set of label matchers and a time range
It then returns compressed chunks of samples as they are found in the block data
So it’s purely a data retrieval API and does not provide complex query execution
Queriers are stateless and horizontally scalable instances that implement PromQL on top of the Store APIs exposed in the cluster
Queriers participate in the cluster to be able to resiliently discover all data sources and store nodes
Rule nodes in return can discover query nodes to evaluate recording and alerting rules
Based on the metadata of store and source nodes, they attempt to minimize the request fanout to fetch data for a particular query
The only scalable components of Thanos is the query nodes as none of the Thanos components provide sharding
Scaling of storage capacity is ensured by relying on an external object storage system
Store, rule, and compactor nodes are all expected to scale significantly within a single instance or high availability pair
The information from above was retrieved from their website, feel free to check them out if you want to read more on the concepts of thanos.
The Architecture Overview of Thanos looks like this:
What are we doing today
We will setup a Thanos Cluster with Minio, Node-Exporter, Grafana on Docker. Our Thanos setup will consist of 3 prometheus containers, each one running with a sidecar container, a store container, 2 query containers, then we have the remotewrite and receive containers which node-exporter will use to ship its metrics to.
The minio container will be used as our long-term storage and the mc container will be used to initialize the storage bucket which is used by thanos.
Deploy the Cluster
Below is the docker-compose.yml and the script to generate the configs for thanos:
Once you have saved the compose as docker-compose.yml and the script as configs.sh you can create the configs:
$ bash configs.sh
The script from above creates the data directory and place all the configs that thanos will use in there. Next deploy the thanos cluster: