How to monitor Red Hat Ansible Tower Using Prometheus, Node Exporter Grafana

Previously, we have posted "How to install and configure Prometheus & grafana in Red Hat Enterprise Linux 7",Now, let's try to monitor our Ansible Tower activity from there. So we need to some configuration for that.

Setup of node exporter on Ansible Tower Node:

Step:1 To download Node_Exporter from Prometheus Download Page.

# cd /tmp
# wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
--2020-07-27 13:19:58--  https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz 

::::::::::::: CUT SOME OUTPUT :::::::::::::

100%[========================================================================================================>] 9,520,728    149KB/s   in 57s    
::::::::::::: CUT SOME OUTPUT :::::::::::::
2020-07-27 13:20:57 (164 KB/s) - ‘node_exporter-1.0.1.linux-amd64.tar.gz’ saved [9520728/9520728]

Step:2 To prepared prerequisite configurations for Node_Exporter.

# useradd --no-create-home -s /bin/false prometheus
# mkdir -p /var/lib/prometheus/node_exporter

Step:3 To extract downloaded zip file & configure Node_Exporter.

# tar xvf node_exporter-1.0.1.linux-amd64.tar.gz 
node_exporter-1.0.1.linux-amd64/
node_exporter-1.0.1.linux-amd64/NOTICE
node_exporter-1.0.1.linux-amd64/node_exporter
node_exporter-1.0.1.linux-amd64/LICENSE
# mv node_exporter-1.0.1.linux-amd64/* /var/lib/prometheus/node_exporter
# chown -R prometheus:prometheus /var/lib/prometheus/node_exporter/

Step:4 To create a systemd configuration file for Node_Exporter.

# vi /usr/lib/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/var/lib/prometheus/node_exporter/node_exporter

[Install]
WantedBy=default.target

Step:5 To start the Node_Exporter service.

# systemctl enable --now node_exporter.service
Created symlink from /etc/systemd/system/default.target.wants/node_exporter.service to /usr/lib/systemd/system/node_exporter.service.
# systemctl status node_exporter.service

Step:6 To allow node_exporter service port port in firewall, if firewall service is running.

# firewall-cmd --add-port=9100/tcp --permanent 
success
# firewall-cmd --reload
success

Generate the token at Ansible Tower Server:

Step:1 Need to generate the token that will grant access to Ansible Tower without the need to enter username and passwords each time it is accessed.

Click Login User (Admin) at top right corner and then click TOKENS tab, as below.

Click "+" and select the read or write under the SCOPE, as below

Click save and copy the tocken, as below.

Okay, now we have the tocken.

Verify Ansible Tower metrics:

To verify the Ansible Tower metrics are being displayed correctly by accessing the URL, as below:

https://atower.lab.munshibari.biz/api/v2/metrics/

Okay, all available Ansible Tower metrics has shown.

Add Ansible Tower Node as Target in Prometheus Server:

Step:1 To edit Prometheus configuration file and add the target.

# vi /etc/prometheus/prometheus.yml
::::::::::::: CUT SOME OUTPUT :::::::::::::
## Scrape Config - Tower
  - job_name: 'tower'
    metrics_path: /api/v2/metrics
    scrape_interval: 5s
    scheme: https
    bearer_token: hLKFdTP5fjfx7M7Q3SqzkDPVnwO4My
    static_configs:
    - targets:
      - atower.lab.munshibari.biz
    tls_config:
      insecure_skip_verify: true

## Add Node Exporter
  - job_name: 'tower-01'
    scrape_interval: 5s
    static_configs:
    - targets: ['192.168.122.80:9100']

Step:2 To restart the Prometheus service to effect changes.

# systemctl restart prometheus
# systemctl status prometheus

Step:3 To verify the newly added Target in the Prometheus service, open the URL http://<prometheus_ipaddress>:9090 using a client's browser.

Configure Grafana to import the dashboards:

We can download the dashboards from the Grafana web site or import the dashboards through json files into Grafana.

Here I am going to try few dashboard as below:

• Import json files "metric_servers.json" https://github.com/redhat-cop/tower_grafana_dashboards

• From the Grafana at https://grafana.com/grafana/dashboards/12609

Step:1 To import json files and create the dashboards.

Click the Manage from the Dashboard, then lick the Import from the Manage, and Upload the JASON file, as below.

Click the Import and save the dashboard, as below.

Step:2 1st dashboard looks like.

Step:3 I have created the second dashboard, it's looks like.

Reference Link: Ansible Blog