Monitoring servers using Prometheus, Node Exporter and Grafana on Ubuntu 18.04

1. Installing Prometheus

Step 1 — Creating Service User

sudo useradd --create-home -s /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /home/prometheus/data
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /home/prometheus
sudo chmod 750 /home/prometheus
sudo chmod 750 /home/prometheus/data

Step 2 — Download Prometheus

Download latest version from the URL: https://prometheus.io/download/#prometheus

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz
tar xvf prometheus-2.0.0.linux-amd64.tar.gz
sudo cp prometheus-2.0.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.0.0.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo cp -r prometheus-2.0.0.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.0.0.linux-amd64/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
rm -rf prometheus-2.0.0.linux-amd64.tar.gz prometheus-2.0.0.linux-amd64

Step 3 — Configuring Prometheus

Create prometheus config file

tee /etc/prometheus/prometheus.yml << "EOF"
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
EOF

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Create Prometheus service

mkdir /var/run/prometheus
chown prometheus:prometheus /var/run/prometheus
tee /etc/systemd/system/prometheus.service << "EOF"
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
RuntimeDirectory=prometheus
RuntimeDirectoryMode=0750
Type=simple
PIDFile=/var/run/prometheus/prometheus.pid
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/sbin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /home/prometheus/data/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries
ExecStartPost=/bin/sh -c 'umask 022; pgrep prometheus > /var/run/prometheus/prometheus.pid'
Restart=always
SyslogIdentifier=prometheus
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus
sudo systemctl enable prometheus

2. Installing Node Exporter on local server

sudo useradd --no-create-home --shell /bin/false node_exporter

curl -fsSL https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz \
  | sudo tar -zxvf - -C /usr/local/bin --strip-components=1 node_exporter-0.18.1.linux-amd64/node_exporter \
  &amp;&amp; sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

tee /etc/systemd/system/node_exporter.service <<"EOF"
[Unit]
Description=Node Exporter

[Service]
User=node_exporter
Group=node_exporter
EnvironmentFile=-/etc/sysconfig/node_exporter
ExecStart=/usr/local/bin/node_exporter $OPTIONS

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload &amp;&amp; \
systemctl start node_exporter &amp;&amp; \
systemctl status node_exporter &amp;&amp; \
systemctl enable node_exporter

3. Installing Node Exporter on remote server

sudo useradd --no-create-home --shell /bin/false node_exporter

curl -fsSL https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz \
  | sudo tar -zxvf - -C /usr/local/bin --strip-components=1 node_exporter-0.18.1.linux-amd64/node_exporter \
  &amp;&amp; sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

tee /etc/systemd/system/node_exporter.service <<"EOF"
[Unit]
Description=Node Exporter

[Service]
User=node_exporter
Group=node_exporter
EnvironmentFile=-/etc/sysconfig/node_exporter
ExecStart=/usr/local/bin/node_exporter $OPTIONS

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload &amp;&amp; \
systemctl start node_exporter &amp;&amp; \
systemctl status node_exporter &amp;&amp; \
systemctl enable node_exporter

tee /etc/nginx/sites-available/nodex <<"EOF"
server {
    listen 80;
    listen [::]:80;

    server_name {SUBDOMAIN-OF-REMOTE-HOST};

    allow {MASTER-SERVER-IP};
    deny  all;

    location / {
        proxy_pass http://localhost:9100/;
    }
}
EOF
ln -s /etc/nginx/sites-available/nodex /etc/nginx/sites-enabled/nodex

nginx -t &amp;&amp; service nginx reload

In case the remote system is old, like Ubuntu 12.04 or 14.04 without systemd, this can help

tee /init/node_exporter << "EOF"
description     "node_exporter"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn
umask 022
chdir /
setuid node_exporter
setgid node_exporter

console none

exec /usr/local/bin/node_exporter
EOF
service node_exporter start

4. Configuring Prometheus to scrape Node Exporters

For local node exporter, add this to the main server’s /etc/prometheus/prometheus.yml

- job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

For remote node exporters, add this to the main server’s /etc/prometheus/prometheus.yml

- job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']
      - targets: ['{SUBDOMAIN-OF-REMOTE-HOST}']
      - targets: ['{SUBDOMAIN-OF-ANOTHER-REMOTE-HOST}']

Reload Prometheus after changing config by

sudo service prometheus reload

5. Installing Grafana

Latest version can be found at https://grafana.com/grafana/download?platform=linux

wget https://dl.grafana.com/oss/release/grafana_6.3.4_amd64.deb
sudo apt-get install -y adduser libfontconfig1
sudo dpkg -i grafana_6.3.4_amd64.deb 

Configure APT repo for future udpates or install via repository

apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana

Start the Grafana Service and enable it to load at startup

systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
sudo systemctl enable grafana-server.service

Create nginx config for to pass requests to Grafana

tee /etc/nginx/sites-available/grafana << "EOF"
server {
    server_name grafana.my.host;
    location / {
        proxy_pass http://localhost:3000/;
    }
}
EOF
nginx -t &amp;&amp; service nginx reload

Speed Test results on Cloudrino Server

Recently I moved a few sites to Cloudrino to test it out in terms of performance, but it turned out that somehow I was not allocated the network I/O as advertised, but after contacting support, things went well. Though I ran speedtest across multiple servers to find out if there’s still an issue, listed below are the speedtest results, though in short, the network speed of Cloudrino is not good enough, the uplink speed is less than 50Mbps which is half of what they say they provide and thats what is needed to serve the sites.
Continue reading Speed Test results on Cloudrino Server

Issues with Slack? Use IRC instead

Slack, even though quite useful, usually keep having problems of its own. Sometimes you can’t even open the app. Also it consumes a lot of CPU resources draining down the battery to say the least, but not anymore. They already are providing way to use custom gateways, jabber(XMPP) and IRC. Continue reading Issues with Slack? Use IRC instead

Page specific templates for Custom Post Types

By default WordPress allows us to use page specific templates by having files like

page-contact.php (page slug specific)

page-10.php (page ID specific)

but when it comes to custom post types, we’re left with no option but to use same template for all the custom posts of that custom post type.
Continue reading Page specific templates for Custom Post Types

Adding an additional paginated page to the content

We know that adding <!–nextpage–> allows you to paginate the content of the post/page in WordPress.

Also if you want to append the content you can use the_content filter.

This filter though works but adds that content to all the pages instead of creating a new page for that content.
Continue reading Adding an additional paginated page to the content