In this tutorial I will demonstrate how to run Loki v2.0.0 behind a Nginx Reverse Proxy with basic http authentication enabled on Nginx and what to do to configure Nginx for websockets, which is required when you want to use tail in logcli via Nginx.
Assumptions
My environment consists of a AWS Application LoadBalancer with a Host entry and a Target Group associated to port 80 of my Nginx/Loki EC2 instance.
Health checks to my EC2 instance are being performed to instance:80/ready
I have a S3 bucket and a DynamoDB table already running in my account which Loki will use. But NOTE that boltdb-shipper is now production ready since v2.0.0, which is awesome, because now you only require a object store such as S3, so you don’t need DynamoDB.
More information on this topic can be found under their changelog
What can you expect from this blogpost
We will go through the following topics:
Install Loki v2.0.0 and Nginx
Configure HTTP Basic Authentication to Loki’s API Endpoints
Bypass HTTP Basic Authentication to the /ready endpoint for our Load Balancer to perform healthchecks
Enable Nginx to upgrade websocket connections so that we can use logcli --tail
Test out access to Loki via our Nginx Reverse Proxy
Install and use LogCLI
Install Software
First we will install nginx and apache2-utils. In my use-case I will be using Ubuntu 20 as my operating system:
Next we will install Loki v2.0.0, if you are upgrading from a previous version of Loki, I would recommend checking out the upgrade guide mentioned on their releases page.
As you’ve noticed, we are providing a auth_basic_user_file to /etc/nginx/passwords, so let’s create a user that we will be using to authenticate against loki:
1
$ htpasswd -c /etc/nginx/passwords lokiisamazing
Enable and Start Services
Because we created a systemd unit file, we need to reload the systemd daemon:
You will notice that I have a /ready endpoint that I am proxy passing to loki, which bypasses authentication, this has been setup for my AWS Application Load Balancer’s Target Group to perform health checks against.
We can verify if we are getting a 200 response code without passing authentication:
So let’s access the labels API endpoint by passing our basic auth credentials. To leave no leaking passwords behind, create a file and save your password content in that file:
12
$ vim /tmp/.pass
-> then enter your password and save the file <-
Expose the content as an environment variable:
1
$ pass=$(cat /tmp/.pass)
Now make a request to Loki’s labels endpoint by passing authentication:
And unset your pass environment variable, to clean up your tracks:
1
$ unset pass
LogCLI
Now for my favorite part, using logcli to interact with Loki, but more specifically using --tail as it requires websockets, nginx will now be able to upgrade those connections:
Install logcli, in my case I am using a mac, so I will be using darwin: