Monitoring servers using Prometheus, Node Exporter and Grafana on Ubuntu 18.04

1. Installing Prometheus

Step 1 — Creating Service User

sudo useradd --create-home -s /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /home/prometheus/data
sudo chown prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /home/prometheus
sudo chmod 750 /home/prometheus
sudo chmod 750 /home/prometheus/dataCode language: Bash (bash)

Step 2 — Download Prometheus

Download latest version from the URL: https://prometheus.io/download/#prometheus

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz
tar xvf prometheus-2.21.0.linux-amd64.tar.gz
sudo cp prometheus-2.21.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.21.0.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo cp -r prometheus-2.21.0.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.21.0.linux-amd64/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
rm -rf prometheus-2.21.0.linux-amd64.tar.gz prometheus-2.21.0.linux-amd64Code language: Bash (bash)

Step 3 — Configuring Prometheus

Create prometheus config file

tee /etc/prometheus/prometheus.yml << "EOF"
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
EOF
sudo chown prometheus:prometheus /etc/prometheus/prometheus.ymlCode language: Bash (bash)

Create Prometheus service

mkdir /var/run/prometheus
chown prometheus:prometheus /var/run/prometheus
tee /etc/systemd/system/prometheus.service << "EOF"
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
RuntimeDirectory=prometheus
RuntimeDirectoryMode=0750
Type=simple
PIDFile=/var/run/prometheus/prometheus.pid
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /home/prometheus/data/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries
ExecStartPost=/bin/sh -c 'umask 022; pgrep prometheus > /var/run/prometheus/prometheus.pid'
Restart=always
SyslogIdentifier=prometheus
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus
sudo systemctl enable prometheusCode language: Bash (bash)

2. Installing Node Exporter on local server

sudo useradd --no-create-home --shell /bin/false node_exporter
curl -fsSL https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz \
  | sudo tar -zxvf - -C /usr/local/bin --strip-components=1 node_exporter-1.3.1.linux-amd64/node_exporter \
  && sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
tee /etc/systemd/system/node_exporter.service <<"EOF"
[Unit]
Description=Node Exporter
[Service]
User=node_exporter
Group=node_exporter
EnvironmentFile=-/etc/sysconfig/node_exporter
ExecStart=/usr/local/bin/node_exporter $OPTIONS
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && \
systemctl start node_exporter && \
systemctl status node_exporter && \
systemctl enable node_exporterCode language: Bash (bash)

3. Installing Node Exporter on remote server

sudo useradd --no-create-home --shell /bin/false node_exporter
curl -fsSL https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz \
  | sudo tar -zxvf - -C /usr/local/bin --strip-components=1 node_exporter-1.3.1.linux-amd64/node_exporter \
  && sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
tee /etc/systemd/system/node_exporter.service <<"EOF"
[Unit]
Description=Node Exporter
[Service]
User=node_exporter
Group=node_exporter
EnvironmentFile=-/etc/sysconfig/node_exporter
ExecStart=/usr/local/bin/node_exporter $OPTIONS
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && \
systemctl start node_exporter && \
systemctl status node_exporter && \
systemctl enable node_exporter
tee /etc/nginx/sites-available/nodex <<"EOF"
server {
    listen 80;
    listen [::]:80;
    server_name {SUBDOMAIN-OF-REMOTE-HOST};
    allow {MASTER-SERVER-IP};
    deny  all;
    location / {
        proxy_pass http://localhost:9100/;
    }
}
EOF
ln -s /etc/nginx/sites-available/nodex /etc/nginx/sites-enabled/nodex
nginx -t && service nginx reloadCode language: Bash (bash)

In case the remote system is old, like Ubuntu 12.04 or 14.04 without systemd, this can help

tee /init/node_exporter << "EOF"
description     "node_exporter"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]
respawn
umask 022
chdir /
setuid node_exporter
setgid node_exporter
console none
exec /usr/local/bin/node_exporter
EOF
service node_exporter startCode language: Bash (bash)

4. Configuring Prometheus to scrape Node Exporters

For local node exporter, add this to the main server’s /etc/prometheus/prometheus.yml

- job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']Code language: Bash (bash)

For remote node exporters, add this to the main server’s /etc/prometheus/prometheus.yml

- job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']
      - targets: ['{SUBDOMAIN-OF-REMOTE-HOST}']
      - targets: ['{SUBDOMAIN-OF-ANOTHER-REMOTE-HOST}']Code language: Bash (bash)

Reload Prometheus after changing config by

sudo service prometheus reloadCode language: Bash (bash)

5. Installing Grafana

Latest version can be found at https://grafana.com/grafana/download?platform=linux

wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb
sudo apt-get install -y adduser libfontconfig1
sudo dpkg -i grafana_7.2.0_amd64.deb Code language: Bash (bash)

Configure APT repo for future udpates or install via repository

apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafanaCode language: Bash (bash)

Start the Grafana Service and enable it to load at startup

systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
sudo systemctl enable grafana-server.serviceCode language: Bash (bash)

Create nginx config for to pass requests to Grafana

tee /etc/nginx/sites-available/grafana << "EOF"
server {
    server_name grafana.my.host;
    location / {
        proxy_pass http://localhost:3000/;
    }
}
EOF
ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/grafana
nginx -t && service nginx reloadCode language: Bash (bash)