Monitoring servers using Prometheus, Node Exporter and Grafana on Ubuntu 18.04

1. Installing Prometheus

Step 1 — Creating Service User

sudo useradd --create-home -s /bin/false prometheus sudo mkdir /etc/prometheus sudo mkdir /home/prometheus/data sudo chown prometheus:prometheus /etc/prometheus sudo chown -R prometheus:prometheus /home/prometheus sudo chmod 750 /home/prometheus sudo chmod 750 /home/prometheus/data
Code language: JavaScript (javascript)

Step 2 — Download Prometheus

Download latest version from the URL: https://prometheus.io/download/#prometheus

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz tar xvf prometheus-2.21.0.linux-amd64.tar.gz sudo cp prometheus-2.21.0.linux-amd64/prometheus /usr/local/bin/ sudo cp prometheus-2.21.0.linux-amd64/promtool /usr/local/bin/ sudo chown prometheus:prometheus /usr/local/bin/prometheus sudo chown prometheus:prometheus /usr/local/bin/promtool sudo cp -r prometheus-2.21.0.linux-amd64/consoles /etc/prometheus sudo cp -r prometheus-2.21.0.linux-amd64/console_libraries /etc/prometheus sudo chown -R prometheus:prometheus /etc/prometheus/consoles sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries rm -rf prometheus-2.21.0.linux-amd64.tar.gz prometheus-2.21.0.linux-amd64
Code language: JavaScript (javascript)

Step 3 — Configuring Prometheus

Create prometheus config file

tee /etc/prometheus/prometheus.yml << "EOF" global: scrape_interval: 15s scrape_configs: - job_name: 'prometheus' scrape_interval: 5s static_configs: - targets: ['localhost:9090'] EOF sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Code language: PHP (php)

Create Prometheus service

mkdir /var/run/prometheus chown prometheus:prometheus /var/run/prometheus tee /etc/systemd/system/prometheus.service << "EOF" [Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus RuntimeDirectory=prometheus RuntimeDirectoryMode=0750 Type=simple PIDFile=/var/run/prometheus/prometheus.pid ExecReload=/bin/kill -HUP $MAINPID ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /home/prometheus/data/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries ExecStartPost=/bin/sh -c 'umask 022; pgrep prometheus > /var/run/prometheus/prometheus.pid' Restart=always SyslogIdentifier=prometheus StandardOutput=syslog StandardError=syslog [Install] WantedBy=multi-user.target EOF sudo systemctl daemon-reload sudo systemctl start prometheus sudo systemctl status prometheus sudo systemctl enable prometheus
Code language: JavaScript (javascript)

2. Installing Node Exporter on local server

sudo useradd --no-create-home --shell /bin/false node_exporter curl -fsSL https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz \ | sudo tar -zxvf - -C /usr/local/bin --strip-components=1 node_exporter-1.0.1.linux-amd64/node_exporter \ && sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter tee /etc/systemd/system/node_exporter.service <<"EOF" [Unit] Description=Node Exporter [Service] User=node_exporter Group=node_exporter EnvironmentFile=-/etc/sysconfig/node_exporter ExecStart=/usr/local/bin/node_exporter $OPTIONS [Install] WantedBy=multi-user.target EOF systemctl daemon-reload && \ systemctl start node_exporter && \ systemctl status node_exporter && \ systemctl enable node_exporter
Code language: JavaScript (javascript)

3. Installing Node Exporter on remote server

sudo useradd --no-create-home --shell /bin/false node_exporter curl -fsSL https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz \ | sudo tar -zxvf - -C /usr/local/bin --strip-components=1 node_exporter-1.0.1.linux-amd64/node_exporter \ && sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter tee /etc/systemd/system/node_exporter.service <<"EOF" [Unit] Description=Node Exporter [Service] User=node_exporter Group=node_exporter EnvironmentFile=-/etc/sysconfig/node_exporter ExecStart=/usr/local/bin/node_exporter $OPTIONS [Install] WantedBy=multi-user.target EOF systemctl daemon-reload && \ systemctl start node_exporter && \ systemctl status node_exporter && \ systemctl enable node_exporter tee /etc/nginx/sites-available/nodex <<"EOF" server { listen 80; listen [::]:80; server_name {SUBDOMAIN-OF-REMOTE-HOST}; allow {MASTER-SERVER-IP}; deny all; location / { proxy_pass http://localhost:9100/; } } EOF ln -s /etc/nginx/sites-available/nodex /etc/nginx/sites-enabled/nodex nginx -t && service nginx reload
Code language: JavaScript (javascript)

In case the remote system is old, like Ubuntu 12.04 or 14.04 without systemd, this can help

tee /init/node_exporter << "EOF" description "node_exporter" start on filesystem or runlevel [2345] stop on runlevel [!2345] respawn umask 022 chdir / setuid node_exporter setgid node_exporter console none exec /usr/local/bin/node_exporter EOF service node_exporter start
Code language: JavaScript (javascript)

4. Configuring Prometheus to scrape Node Exporters

For local node exporter, add this to the main server’s /etc/prometheus/prometheus.yml

- job_name: 'node_exporter' scrape_interval: 5s static_configs: - targets: ['localhost:9100']
Code language: JavaScript (javascript)

For remote node exporters, add this to the main server’s /etc/prometheus/prometheus.yml

- job_name: 'node_exporter' scrape_interval: 5s static_configs: - targets: ['localhost:9100'] - targets: ['{SUBDOMAIN-OF-REMOTE-HOST}'] - targets: ['{SUBDOMAIN-OF-ANOTHER-REMOTE-HOST}']
Code language: JavaScript (javascript)

Reload Prometheus after changing config by

sudo service prometheus reload

5. Installing Grafana

Latest version can be found at https://grafana.com/grafana/download?platform=linux

wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb sudo apt-get install -y adduser libfontconfig1 sudo dpkg -i grafana_7.2.0_amd64.deb
Code language: JavaScript (javascript)

Configure APT repo for future udpates or install via repository

apt-get install -y software-properties-common sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main" wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - sudo apt-get update sudo apt-get install grafana
Code language: JavaScript (javascript)

Start the Grafana Service and enable it to load at startup

systemctl daemon-reload systemctl start grafana-server systemctl status grafana-server sudo systemctl enable grafana-server.service
Code language: CSS (css)

Create nginx config for to pass requests to Grafana

tee /etc/nginx/sites-available/grafana << "EOF" server { server_name grafana.my.host; location / { proxy_pass http://localhost:3000/; } } EOF ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/grafana nginx -t && service nginx reload
Code language: JavaScript (javascript)

Published
Categorized as Server

Leave a comment

Your email address will not be published. Required fields are marked *