I was trying to install grafana on ubuntu when I got faced with: “the following signatures couldn’t be verified because the public key is not available” error as seen below:
1 2 3 4 5 6 7 8 9 10 11 | |
In order to continue, we need to import the trusted key:
1 2 3 4 5 | |
Now that the key has been imported, we can update and continue:
1 2 3 4 5 6 7 8 9 | |
Vault is one of Hashicorp’s awesome services, which enables you to centrally store, access and distribute dynamic secrets such as tokens, passwords, certificates and encryption keys.
We will setup a Vault Server on Docker and demonstrate a getting started guide with the Vault CLI to Initialize the Vault, Create / Use and Manage Secrets.
For related posts:
Create the directory structure:
1 2 | |
Populate the vault config vault.json. (As you can see the config is local, in the next couple of posts, I will show how to persist this config to Amazon S3)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
Populate the docker-compose.yml:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Start the Vault Server:
1
| |
The UI is available at http://localhost:8200/ui and the api at http://localhost:8200
I will demonstrate how to use the Vault CLI to interact with Vault. Let’s start by installing the vault cli tools, I am using mac, so I will be using brew:
1
| |
Set environment variables:
1
| |
Initialize new vault cluster with 6 key shares:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | |
In order to unseal the vault cluster, we need to supply it with 3 key shares:
1 2 3 | |
Ensure the vault is unsealed:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
Authenticate against the vault:
1 2 3 4 | |
Using the cli your auth token will be saved locally at ~/.vault-token.
Enable the secret kv engine:
1
| |
Write a secret to the path enabled above:
1
| |
List your secrets:
1 2 3 4 | |
Read the secret (defaults in table format):
1 2 3 4 5 | |
Read the secret in json format:
1 2 3 4 5 6 7 8 9 10 11 | |
Read only the password value in the secret:
1 2 | |
Create a key with multiple secrets:
1
| |
Read all the secrets:
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
Read only the username field in the key:
1 2 | |
Delete the secret:
1
| |
Create a key and set the metadata to max of 5 versions:
1
| |
Get the metadata of the key:
1 2 3 4 5 6 7 8 9 10 | |
Write a secret appname to our key: secret/fooapp/appname:
1 2 3 4 5 6 7 | |
Overwrite the key with a couple of requests:
1 2 | |
Read the current value:
1 2 | |
Get the version=2 value of this file:
1 2 | |
Thanks for reading, hope this was informative. Have a look at Hashicorp’s Vault Documentation for more information on the project. I will post more posts on Vault under the #vault category.
In this tutorial we will use Concourse to Deploy our application to Docker Swarm.
The code for this tutorial is available on my github repository
The application structure for our code looks like this:
Our ci/pipeline.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 | |
Our ci/credentials.yml which will hold all our secret info, which will remain local:
1 2 3 4 | |
The first step of our deploy will invoke a shell script that will establish a ssh tunnel to the docker host, mounting the docker socket to a tcp local port, then exporting the docker host port to the tunneled port, ci/scripts/deploy.sh:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | |
The deploy script references the docker-compose files, first our ci/docker/docker-compose.staging.yml:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Also, our docker-compose for production, ci/docker/docker-compose.production.yml:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Create 2 branches in your github repository for versioning: version-staging and version-prod, then logon to concourse and save the target:
1
| |
Set the pipeline, point the config, local variables definition and name the pipeline:
1
| |
You will find that the pipeline will look like below and that it will be in a paused state:
Unpause the pipeline:
1
| |
The pipeline should kick-off automatically due to the trigger that is set to true:
Deployed automatically to staging, prod is a manual trigger:
For demonstration purposes we have deployed staging on port 81 and production on port 80.
Testing Staging on http://
Testing Production on http://
Another nice thing about Drone CI is the “Services” configuration within your pipeline. At times your unit or integration testing steps might be dependent of a database such as MongoDB, MySQL etc.
Drone allows you to spin up a ephemeral database service such as MongoDB using a Docker container as the fist step within your pipeline, defined in the services section. This step will always run first.
The service container will be reachable via the configured container name as its hostname. Keep note that if you run multiple paralel jobs that the service container will only be reachable from the container where the mongodb container is running.
We will setup a really basic (and a bit useless) pipeline that will spin up a mongodb service container, use a step to write random data to mongodb and a step that reads data from mongodb.
For demonstration purposes, the data is really random but more focused on the service section.
All the source code for this demonstration is available on my github repository
First we define our service, mongodb. Once the mongodb service is running, we will have our build step, our step that runs the mongodb version against our database, write data into our mongodb database, then read the data from mongodb, then the last step running a shell command with the date.
Our .drone.yml pipeline definition:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | |
Our scripts referenced in our steps:
The first will be our script that write random data into mongodb, scripts/write_mongo.sh:
1 2 3 4 5 | |
We are referencing a scripts/write.js file which is a function that randomizes data and generates a 1000 documents to write to mongodb:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
Our script that will read data from mongodb, scripts/read_mongo.sh:
1 2 3 4 5 6 7 8 | |
The README.md to include the build status:
1
| |
Once your source code is set in github, enable the repository on drone and push to github to trigger the build.
After pushing to github to trigger the build, heading over to drone, I can see that mongodb is running and our step has completed that executes the db.version() against mongodb:
Next our step executes to write the random data into mongodb:
After the data has been written to mongodb, our next step will read the number of documents from mongodb, and also run a query for transaction prices more than 990:
Once that has completed, we will have a shell command returning the time when the last step completed:
Beeline Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)
Some time ago, I assisted a customer who was trying to do a select count(*) via beeline and failed with:
1 2 3 | |
When reproducing this I found a jira: https://issues.apache.org/jira/browse/HIVE-14631 which related to the same issue and the workaround was to switch your execution engine to mapreduce. By doing that, it worked, but wanted a better resolution for the customer.
When setting enabling debugging, I found that the error is related to permissions:
1 2 3 4 5 6 7 8 | |
So it seems that when the client (anonymous) is trying to copy the hive execution jar to is home path in HDFS, in this case (/home/anonymous/.hiveJars/) it fails due to permissions.
By passing the hadoop user, I was able to get the expected results:
1 2 3 4 5 6 | |
Listing the mentioned jar:
1 2 3 | |
Hope this might help someone facing the same issue
CICD Pipelines! <3
In this post I will show you how to setup a cicd pipeline using drone to build a jekyll site and deploy to docker swarm.
Jekyll’s Codebase: Our code will be hosted on Github (I will demonstrate how to set it up from scratch)
Secret Store: Our secrets such as ssh key, swarm host address etc will be stored in drones secrets manager
Docker Swarm: Docker Swarm has Traefik as a HTTP Loadbalancer
Drone Server and Agent: If you dont have drone, you can setup drone server and agent on docker or have a look at cloud.drone.io
Workflow:
1 2 3 4 5 | |
Install Jekyll locally, as we will use it to create the initial site. I am using a mac, so I will be using brew. For other operating systems, have a look at this post.
I will be demonstrating with a weightloss blog as an example.
Install jekyll:
1
| |
Go ahead and create a new site which will host the data for your jekyll site:
1
| |
First we need to create an empty github repository, in my example it was github.com/ruanbekker/blog-weightloss.git. Once you create the repo change into the directory created by the jekyll new command:
1
| |
Now initialize git, set the remote, add the jekyll data and push to github:
1 2 3 4 5 | |
You should see your data on your github repository.
Logon to the Drone UI, sync repositories, activate the new repository and head over to settings where you will find the secrets section.
Add the following secrets:
1 2 3 4 5 6 7 8 | |
You should see the following:
Drone looks from a .drone.yml file in the root directory for instructions on how to do its tasks. Lets go ahead and declare our pipeline:
1
| |
And populate the drone config:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | |
If you want to be notified about your builds, you can add a slack notification step as the last step.
To do that, create a new webhook integration, you can follow this post for a step by step guide. After you have the webhook, go to secrets and create a slack_webhook secret.
Then apply the notification step as shown below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
Based on the status, you should get a notification similar like this:
Next we need to declare our docker compose file which is needed to deploy our jekyll service to the swarm:
1
| |
And populate this info (just change the values for your own environment/settings):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | |
Now we need to push our .drone.yml and docker-compose.yml to github. Since the repository is activated on drone, any push to master will trigger the pipeline, so after this push we should go to drone to look at our pipeline running.
Add the untracked files and push to github:
1 2 3 4 | |
As you head over to your drone ui, you should see your pipeline output which will look more or less like this (just look how pretty it is! :D )
If your deployment has completed you should be able to access your application on the configured domain. A screenshot of my response when accessing Jekyll:
Absolutely Amazingness! I really love drone!
In this post we will setup a blog on hugo and using the theme pickles.
Hugo is a Open-Source Static Site Generator which runs on Golang.
Im using a mac so I will be installing hugo with brew, for other operating systems, you can have a look at their documentation
1
| |
Create your new site:
1
| |
We will use a 3rd party theme, go ahead and install the pickles theme:
1
| |
Generate syntax highlight css, for a list of other styles see this post
1 2 | |
Append this below style.css in themes/pickles/layouts/partials/head.html
1
| |
set pygments settings in config.toml:
1 2 3 4 5 | |
Create your first post:
1 2 | |
Populate your page with some data:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | |
You can set the flags in your main config as well. Go ahead and run the server:
1 2 3 4 5 | |
When you access your blog on port 8080 you should see your post. Some screenshots below:
Drone is a self-service continuous delivery platform which can be used for CICD pipelines, devopsy stuff which is really awesome.
With Configuration as Code, Pipelines are configured with a simple, easy‑to‑read file that you commit to your git repository such as github, gitlab, gogs, gitea etc.
Each Pipeline step is executed inside an isolated Docker container that is automatically downloaded at runtime, if not found in cache.
A pipeline can look as easy as:
1 2 3 4 5 6 7 8 9 10 11 12 | |
I have enabled public access, so please go ahead and launch your cicd pipelines on my drone setup as I want to test the stability of it:
We will deploy a drone server which is responsible for the actual server and 2 drone agents which will receive instructions from the server whenever steps need to be executed. Steps run on agents.
I’m using VULTR to deploy 3 nodes on coreos, 1 drone server and 2 drone agents as seen below:
Documentation: https://docs.drone.io/installation/github/multi-machine/ https://github.com/settings/developers
We will use Github for version control and to delegate auth, therefore we need to register a new application on Github.
Register New Application on Github at https://github.com/settings/developer :
Get your Drone-Server Host Endpoint, and update the fields:
You will receive a Github Client ID, Secret which we will need later, which will look like this:
1 2 3 4 | |
Generate the shared secret which will be used on the server and agent:
1 2 | |
Create the Startup Script for Drone Server, which will just be a docker container running in detached mode. Note that you should use your own domain at SERVER_HOST and if you want to issue an certificate automatically keep DRONE_TLS_AUTOCERT to true.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | |
Create the startup script for the drone agent, note that this script needs to be placed on the agent nodes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | |
Logon to the server node and start the drone server:
1
| |
Login to the agent nodes and start the agents:
1
| |
The server should show that it’s listening on port 80 and 443:
1 2 3 | |
Access your Drone instance on port 80 eg. http://drone.yourdomain.com you should be automatically redirected to port 443, which should direct you to a login page, which will look like this:
Login with your github account and allow drone some time to sync your repositories:
Clone this repository: https://github.com/ruanbekker/drone-ci-testing which will contain the .drone.yml config which drone gets its instructions from.
Select a repository to activate, (drone-ci-testing in this case) head over to settings:
Adding secret:
Add more secrets:
Your build list should be empty:
Edit any of the files in the clone repository and you should see your build running:
When your build has completed:
You can also find out where the step ran:
Run a couple of tests:
Get notified via slack:
If your build fails, its most likely that you need the slack_webhook secret. You can remove the slack step which shouldhelp you get going with drone.
Have a look at this document for more examples or have a look at their documentation as well as their extensive list of plugins and their setup documentation to become familiar with their configuration.
Slack is amazing and I cant live without it.
We can also use custom webhook integrations to allow applications to notify us via slack in response of events.
We will be configuring a custom slack webhook integration and test out the api to show you how easy it is to use it to inform us via slack, whenever something is happening.
Head over to: - https://{your-team}.slack.com/apps/manage/custom-integrations
Select Incoming Webhooks:
Select Add Configuration:
Select the channel it should post to:
Select Add Incoming Webhook Integration.
Save the webhook url that will look like this:
1
| |
You can then further configure the integration.
1
| |
Will result in:
Message Attachment, Error:
1
| |
Results in:
Message Attachment, OK:
1
| |
Results in:
If you want to join my slack workspace, use this invite link
While looking into working with mongodb using golang, I found it quite frustrating getting it up and running and decided to make a quick post about it.
Examples using the golang driver for mongodb to connect, read, update and delete documents from mongodb.
Provision a mongodb server in docker:
1 2 | |
Drop into a golang environment using docker:
1
| |
Get the dependencies:
1
| |
Change to your project path:
1 2 | |
Download the golang mongodb driver:
1
| |
First example will be to connect to your mongodb instance:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | |
Running our app:
1 2 | |
Let’s insert a single document to MongoDB:
1 2 3 4 5 6 7 8 9 10 11 12 | |
Running it will produce:
1 2 3 | |
Writing more than one document:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
This will output in:
1 2 | |
Updating Frankie’s age:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
Running that will update Frankie’s age:
1 2 | |
Reading the data:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
1 2 | |
Finding multiple documents and returning the cursor
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | |
Running the example:
1 2 | |
Deleting our data and closing the connection:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Running the example:
1 2 3 | |
The code for this example can be found at github.com/ruanbekker/code-examples/mongodb/golang/examples.go