In this tutorial we will build a ci pipeline using concourse to build and push a image to dockerhub automatically, whenever a new git commit is made to the master branch.
Now that we have our pipeline definition, credentials and application code (stored in version control), go ahead and set the pipeline, which will save the pipeline configuration in concourse:
Filebeat by Elastic is a lightweight log shipper, that ships your logs to Elastic products such as Elasticsearch and Logstash. Filbeat monitors the logfiles from the given configuration and ships the to the locations that is specified.
Filebeat Overview
Filebeat runs as agents, monitors your logs and ships them in response of events, or whenever the logfile receives data.
Below is a overview (credit: elastic.co) how Filebeat works
Installing Filebeat
Let’s go ahead and install Filebeat. I will be using version 6.7 as that will be the same version that I am running on my Elasticsearch. To check the version of your elasticsearch cluster:
1
$ curl http://127.0.0.1:9200/_cluster/health?pretty # i have es running locally
Let’s configure our main configuration in filebeat, to specify our location where the data should be shipped to (in this case elasticsearch) and I will also like to set some extra fields that will apply to this specific server.
Open up /etc/filebeat/filebeat.yml and edit the following:
Above, just setting my path to nginx access logs, some extra fields, including that it shoulds seed kibana with example visualizations and the output configuration of elasticsearch.
Filebeat Modules
Filebeat comes with modules that has context on specific applications like nginx, mysql etc. Lets enable system (syslog, auth, etc) and nginx for our web server:
12
$ filebeat modules enable system
$ filebeat modules enable nginx
Example of my /etc/filebeat/modules.d/system.yml configuration:
You can have a look at the logs, should you need to debug:
1
tail -f /var/log/filebeat/filebeat
Your data should now be shipped to elasticsearch, by default under the filebeat-YYYY.mm.dd index pattern.
123
$ curl 'http://127.0.0.1:9200/_cat/indices/filebeat*?v'health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open filebeat-6.7.1-2019.03.27 CBdV7adjRKypN1wguwuHDA 314532200 230.2mb 115.9mb
Kibana
You can head over to Kibana at http://localhost:5601 (in this case) to visualize the data that is ingested into your filebeat index. I will write a tutorial on how to graph up most common dashboards later this week.
We will deploy a 3 node docker swarm cluster with terraform on scaleway. I have used the base source code from this repository but tweaked the configuration to my needs.
Pre-Requisites
Ensure terraform and jq is instaled:
12
$ brew install terraform
$ brew install jq
Terraform
You can have a look at the linked source at the top for the source code, but below I will provide each file that will make up our terraform deployment.
Note that we will be deploying 3x SMART1-M servers with Debian Stretch. At this moment the image id is the one of debian stretch but may change in the future. If you want to change the distro, update the install script, and the terraform files.
Make sure that your ssh private key is the intended one as in the config, in my example: ~/.ssh/id_rsa and that they are allowed in your servers authorized_keys file
Once your deployment is done you will be prompted with the public/private ip addresses of your nodes as seen above, you can also manually retrieve them:
Go ahead and ssh to your manager nodes and list the swarm nodes, boom, easy right.
12345
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
2696o0vrt93x8qf2gblbfc8pf * swarm-manager Ready Active Leader 18.09.3
72ava7rrp2acnyadisg52n7ym swarm-worker-1 Ready Active 18.09.3
sy2otqn20qe9jc2v9io3a21jm swarm-worker-2 Ready Active 18.09.3
importrequestsimportjsonimporttimeSCW_API_KEY="<your-api-key>"SCW_OGRA_ID="<your-organization-id>"SCW_REGION="ams1"SCW_COMPUTE_API_URL="https://cp-{region}.scaleway.com/{resource}".format(region=SCW_REGION,resource='servers')SCW_VOLUME_API_URL="https://cp-{region}.scaleway.com/{resource}".format(region=SCW_REGION,resource='volumes')SCW_HEADERS={"X-Auth-Token":SCW_API_KEY,"Content-Type":"application/json"}SCW_IMAGES={"ubuntu/18":"6a601340-19c1-4ca7-9c1c-0704bcc9f5fe","debian/stretch":"710ff1fa-0d16-4f8f-93ac-0647c44fa21d"}defget_status(server_id):response=requests.get(SCW_COMPUTE_API_URL+"/"+server_id,headers=SCW_HEADERS)state=response.json()returnstatedefcreate_server(instance_name,instance_type,instance_tag,os_distro):count=0compute_payload={"name":instance_name,"image":SCW_IMAGES[os_distro],"commercial_type":instance_type,"tags":[instance_tag],"organization":SCW_OGRA_ID}print("creating server")r_create=requests.post(SCW_COMPUTE_API_URL,json=compute_payload,headers=SCW_HEADERS)server_id=r_create.json()["server"]["id"]action_payload={"action":"poweron"}r_start=requests.post(SCW_COMPUTE_API_URL+"/"+server_id+"/action",json=action_payload,headers=SCW_HEADERS)r_describe=requests.get(SCW_COMPUTE_API_URL+"/"+server_id,headers=SCW_HEADERS)server_state=get_status(server_id)['server']['state']whileserver_state!="running":ifcount>90:r_delete=requests.delete(SCW_COMPUTE_API_URL+"/"+server_id,json=action_payload,headers=SCW_HEADERS)return{"message":"error","description":"task timed out while waiting for server to boot"}count+=1print("waiting for server to become ready")time.sleep(10)server_state=get_status(server_id)['server']['state']time.sleep(5)resp=get_status(server_id)["server"]output={"id":resp["id"],"hostname":resp["hostname"],"instance_type":resp["commercial_type"],"public_ip":resp["public_ip"]["address"],"private_ip":resp["private_ip"],"status":resp["state"]}returnoutputresponse=create_server("swarm-manager","START1-M","swarm","ubuntu/18")print(response)
Deploying a server with the hostname: swarm-manager, instance-size: START1-M, tag: swarm and os distribution: ubuntu/18:
123456
$ python scw.py
creating server
waiting for server to become ready
waiting for server to become ready
waiting for server to become ready
{'status': u'running', 'hostname': u'swarm-manager', 'public_ip': u'51.x.x.x', 'instance_type': u'START1-M', 'private_ip': u'10.x.x.x', 'id': u'xx-xx-xx-xx-xx'
If you have not setup the Nagios Server have a look at that link to setup the Nagios server.
Nagios NRPE
Nagios Remote Plugin Executor (NRPE) allows you to remotely execute Nagios plugins on other linux systems. This allows you to monitor remote machine metrics (disk usage, CPU, local listening services, etc.).
NRPE has 2 sections:
The nagios server side.
The client side.
For nagios to execute remote plugins, the client configuration needs to allow the nrpe server which will be nagios.
Download, extract, configure and install NRPE server:
123456789
$ wget 'https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-3.2.1/nrpe-3.2.1.tar.gz'
$ tar -xvf nrpe-3.2.1.tar.gz
$ cd nrpe-3.2.1
$ ./configure --enable-command-args --with-nagios-user=nagios --with-nagios-group=nagcmd --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu
$ make all
$ make install
$ make install-init
$ make install-config
$ systemctl enable nrpe.service
If you have not setup the Nagios Server have a look at that link to setup the Nagios server.
Configure Nagios to Monitor our first Host
I like to setup an isolated path for my custom host/service configigurations. First we will declare the configuration path for our servers.
Open up: /usr/local/nagios/etc/nagios.cfg and add a new cfg_dir:
1
cfg_dir=/usr/local/nagios/etc/servers
Now, create the directory:
1
$ mkdir /usr/local/nagios/etc/servers
Configure your email address for notifications in /usr/local/nagios/etc/objects/contacts.cfg and configure:
1
email youremail@yourdomain.com;
Let’s say we want to configure a web server named web01 that sits at the location 10.10.10.10:
1
$ vi /usr/local/nagios/etc/servers/webservers.cfg
First we define our host configuration:
We are using the linux-server template that is defined in /usr/local/nagios/etc/objects/templates.cfg
We set the hostname, alias and address as well as notification prediods
12345678910
define host {
use linux-server
host_name WEB01
alias WEB01
address 10.10.10.10
max_check_attempts 5
check_period 24x7
notification_interval 30
notification_period 24x7
}
While you have the config open, we want to define the services that we would like to monitor, and associate the services to the host that we defined.
In this example, we want to ping the server and check port tcp 22 and 80. Ensure that your web server is allowing the mentioned ports from the nagios server ip.
In the config, we are declaring the following:
Use the generic-service template
Map the hostname which the service should be associated to
The description that you will see in nagios
Use the check_ping / check_ssh / check_http plugin and set the thresholds for ok, warning, critical
12345678910111213141516171819202122
define service {
use generic-service
host_name WEB01
service_description PING
check_command check_ping!100.0,20%!500.0,60%
}
define service {
use generic-service
host_name WEB01
service_description SSH
check_command check_ssh
notifications_enabled 1
}
define service {
use generic-service
host_name WEB01
service_description HTTP
check_command check_http
notifications_enabled 1
}
Head over to nagios user interface at http://nagios-ip/nagios and you should see that the services are scheduled to be checked and should be reflecting in a minute or two.
$ apt install libnet-snmp-perl libsensors4 libsnmp-base libtalloc2 libtdb1 libwbclient0 snmp whois mrtg libcgi-pm-perl librrds-perl libgd-perl libnagios-object-perl nagios-plugins-contrib
$ ./install.pl --check-prereq
$ ./install.pl --layout standalone --prefix /usr/local/nagiosgraph
Destination directory (prefix)? [/usr/local/nagiosgraph]
Location of configuration files (etc-dir)? [/usr/local/nagiosgraph/etc]
Location of executables? [/usr/local/nagiosgraph/bin]
Location of CGI scripts? [/usr/local/nagiosgraph/cgi]
Location of documentation (doc-dir)? [/usr/local/nagiosgraph/doc]
Location of examples? [/usr/local/nagiosgraph/examples]
Location of CSS and JavaScript files? [/usr/local/nagiosgraph/share]
Location of utilities? [/usr/local/nagiosgraph/util]
Location of state files (var-dir)? [/usr/local/nagiosgraph/var]
Location of RRD files? [/usr/local/nagiosgraph/var/rrd]
Location of log files (log-dir)? [/usr/local/nagiosgraph/var/log]
Path of log file? [/usr/local/nagiosgraph/var/log/nagiosgraph.log]
Path of CGI log file? [/usr/local/nagiosgraph/var/log/nagiosgraph-cgi.log]
Base URL? [/nagiosgraph]
URL of CGI scripts? [/nagiosgraph/cgi-bin]
URL of CSS file? [/nagiosgraph/nagiosgraph.css]
URL of JavaScript file? [/nagiosgraph/nagiosgraph.js]
URL of Nagios CGI scripts? [/nagios/cgi-bin]
Path of Nagios performance data file? [/tmp/perfdata.log]
username or userid of Nagios user? [nagios]
username or userid of web server user? [www-data]
Modify the Nagios configuration? [n] y
Path of Nagios configuration file? [/usr/local/nagios/etc/nagios.cfg]
Path of Nagios commands file? [/usr/local/nagios/etc/objects/commands.cfg]
Modify the Apache configuration? [n] y
Path of Apache configuration directory? /etc/apache2/sites-enabled
Ensure that your nagiosgraph configuration under apache: /etc/apache2/sites-enabled/nagiosgraph.conf has the following config (might be standard)
Ensure the following configuration is set under nagios main config:
12345678
$ vi /usr/local/nagios/etc/nagios.cfg
process_performance_data=1
service_perfdata_file=/usr/local/nagios/var/service-perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata-for-nagiosgraph
Ensure that we have the following commands in place for nagiosgraph:
Create the template graphed-service, this will be mapped to each service that needs to be graphed in nagiosgraph:
1234567
$ vi /usr/local/nagios/etc/objects/templates.cfg
define service {
name graphed-service
action_url /nagiosgraph/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$' onMouseOver='showGraphPopup(this)' onMouseOut='hideGraphPopup()' rel='/nagiosgraph/cgi-bin/showgraph.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&period=week&rrdopts=-w+450+-j
register 0
}
Next configure the services that needs to be graphed on nagios graph. Note, we only need to append the service template that we defined in our template configuration from above:
Note, if you have not checked out Nagios Server Setup post, in that post the inital configuration of the below config is explained.
Access your nagios server at http://nagios-ip/nagios and you will find that the graph icon next to the service will open the graph in a new tab, like the screenshot below:
Good old nagios! Nagios is a great Open Source Monitoring Server that monitors your servers and services/applications that is hosted on top of them, and has the ability to notify in the event when they go down.
I’ve been using Nagios for the last 7 years and worked for 3 business that chose Nagios as their preferred server monitoring solution.
All Nagios related posts are grouped under the #nagios category.
What we are doing today
Today we will setup a Nagios server and its plugins. The plugins helps to check different endpoints, such as custom tcp checks, ssh, snmp etc.
In this nagios tutorial series, I will publish a couple of post which will include:
Setup the Nagios Server and its Plugins - this post
Setup the NRPE Server and NRPE Client Server (this is nice for local ports or custom checks)
Setup Nagiosgraph (Graph performance data and add it as extra host configuration)
Setup a custom Bash and Python Nagios Plugin for Custom Checks
Setup a Telegram / Slack Plugin
Installing Dependencies:
Go ahead and install all the dependencies needed by nagios and add the nagios user and group:
Configure with nagios user and nagcmd group, install and change the ownership of the generated data:
12345678
$ ./configure --with-nagios-group=nagios --with-command-group=nagcmd
$ make -j4 all
$ make install
$ make install-commandmode
$ make install-init
$ make install-config
$ /usr/bin/install -c -m 644 sample-config/httpd.conf /etc/apache2/sites-available/nagios.conf
$ usermod -a -G nagcmd www-data
Install Nagios Plugins
Get the nagios plugins tarball, extract and install:
123456
$ wget nagios-plugins.tar.gz 'https://nagios-plugins.org/download/nagios-plugins-2.2.1.tar.gz#_ga=2.250909126.1332751653.1552389134-913645931.1552389134'$ tar xpf nagios-plugins*.tar.gz
$ cd nagios-plugins-2.2.1
$ ./configure --with-nagios-user=nagios --with-nagios-group=nagcmd --with-openssl
$ make -j4
$ make install
Nginx is a great product! And today we will use nginx to setup a http reverse proxy to access our backend applications.
Our Setup
We will have a flask backend application listening on 127.0.0.1:5000 and our nginx reverse proxy will listen on 0.0.0.0:80 which will proxy requests through to our flask upstream.
Our Backend Application
Our Flask application:
123456789
from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
return 'Hello'
if __name__ == '__main__':
app.run(host='127.0.0.1', port=5000)