Recent Posts

How To Install Ansible Tower with Customer Provided Database

Ansible Tower can be installed with an external database that is provided by the customer, whether on bare metal, virtual machine, container, or cloud-hosted service. I have simulated this in my lab before the actual deployment. Let see.


FYI: https://access.redhat.com/articles/4010491


My Scenario:

Prepared DB1:

Step:1 Verify and install required RPMS

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.122.11	 at1.lab.munshibari.biz	at1
192.168.122.12	 db1.lab.munshibari.biz	db1
192.168.122.13	 db2.lab.munshibari.biz	db2

# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                              repo name                                                                              status
!rhel-7-server-ansible-2.9-rpms/x86_64               Red Hat Ansible Engine 2.9 RPMs for Red Hat Enterprise Linux 7 Server                      21
!rhel-7-server-extras-rpms/x86_64                    Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                       1,303
!rhel-7-server-rpms/x86_64                           Red Hat Enterprise Linux 7 Server (RPMs)                                               29,413
!rhel-server-rhscl-7-rpms/x86_64                     Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server                12,735
repolist: 43,472

# yum install rh-postgresql10-postgresql-server -y

# rpm -qa|grep postgresql
rh-postgresql10-postgresql-libs-10.12-2.el7.x86_64
rh-postgresql10-postgresql-server-10.12-2.el7.x86_64
rh-postgresql10-runtime-3.1-1.el7.x86_64
rh-postgresql10-postgresql-10.12-2.el7.x86_64
postgresql-libs-9.2.24-4.el7_8.x86_64

Step:2 Initialize the new PostgreSQL database server installation.

# /opt/rh/rh-postgresql10/root/usr/bin/postgresql-setup  --initdb
 * Initializing database in '/var/opt/rh/rh-postgresql10/lib/pgsql/data'
 * Initialized, logs are in /var/lib/pgsql/initdb_rh-postgresql10-postgresql.log

Step:3 Enable the PostgreSQL database port in the firewall service.

# firewall-cmd --add-port=5432/tcp --permanent
success
# firewall-cmd --add-port=5432/tcp 
success

Step:4 To set the Environment variable. And read and execute a shell environment.

# cat /etc/profile.d/python.sh 
export PATH="/opt/rh/python27/root/usr/bin:/opt/rh/rh-postgresql10/root/usr/bin:$PATH"
export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64:/opt/rh/rh-postgresql10/root/usr/lib64:$LD_LIBRARY_PATH"
export MANPATH="/opt/rh/python27/root/usr/share/man:$MANPATH"
export XDG_DATA_DIRS="/opt/rh/python27/root/usr/share:$XDG_DATA_DIRS"
export PKG_CONFIG_PATH="/opt/rh/python27/root/usr/lib64/pkgconfig:$PKG_CONFIG_PATH"

# source /etc/profile.d/python.sh

Step:5 To start the postgresql10 service.

# systemctl start rh-postgresql10-postgresql.service
# systemctl enable rh-postgresql10-postgresql.service

Step:6 To create the database and users as required for the Ansible Tower.

# su -  postgres
$ ls -la .pgpass 
-rw-------. 1 postgres postgres 34 Aug 13 00:18 .pgpass
$ cat .pgpass 
192.168.122.13:5432:*:replica:redhat123

$ psql
psql (10.12)
Type "help" for help.

postgres=# CREATE ROLE awxdbuser1 WITH LOGIN encrypted PASSWORD 'redhat123' VALID UNTIL 'infinity';
CREATE ROLE

postgres=# CREATE USER replica REPLICATION LOGIN ENCRYPTED PASSWORD 'redhat123';
CREATE ROLE
postgres=# \du
                                    List of roles
 Role name  |                         Attributes                         | Member of 
------------+------------------------------------------------------------+-----------
 awxdbuser1 | Password valid until infinity                              | {}
 postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 replica    | Replication                                                | {}

postgres=# CREATE DATABASE tower1 WITH ENCODING='UTF8' OWNER=awxdbuser1 CONNECTION LIMIT=-1;
CREATE DATABASE
postgres=# \l
                                   List of databases
   Name    |   Owner    | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+------------+----------+-------------+-------------+-----------------------
 postgres  | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |            |          |             |             | postgres=CTc/postgres
 template1 | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |            |          |             |             | postgres=CTc/postgres
 tower1    | awxdbuser1 | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
(4 rows)

Step:7 To change PostgreSQL listening IP Address and the PostgreSQL Client Authentication Configuration.

# grep listen_addresses  /var/opt/rh/rh-postgresql10/lib/pgsql/data/postgresql.conf 
listen_addresses = '*'		# what IP address(es) to listen on;

# vi /var/opt/rh/rh-postgresql10/lib/pgsql/data/pg_hba.conf
::::::::::::: CUT SOME OUTPUT :::::::::::::
# TYPE  DATABASE        USER            ADDRESS                 METHOD
# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            ident
host    all             awxdbuser1      192.168.122.11/32       md5
host    all             awxdbuser1      192.168.122.12/32       md5
host    all             awxdbuser1      192.168.122.13/32       md5
# IPv6 local connections:
host    all             all             ::1/128                 ident
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            ident
host    replication     all             ::1/128                 ident
host    replication     replica		192.168.122.12/32       md5
host    replication     replica		192.168.122.13/32       md5

Step:8 To restart the PostgreSQL 10 service to effect the change.

# systemctl restart rh-postgresql10-postgresql.service

Prepared DB2:

Step:1 Verify and install required RPMS

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.122.11	 at1.lab.munshibari.biz	at1
192.168.122.12	 db1.lab.munshibari.biz	db1
192.168.122.13	 db2.lab.munshibari.biz	db2

# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                              repo name                                                                              status
!rhel-7-server-ansible-2.9-rpms/x86_64               Red Hat Ansible Engine 2.9 RPMs for Red Hat Enterprise Linux 7 Server                      21
!rhel-7-server-extras-rpms/x86_64                    Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                       1,303
!rhel-7-server-rpms/x86_64                           Red Hat Enterprise Linux 7 Server (RPMs)                                               29,413
!rhel-server-rhscl-7-rpms/x86_64                     Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server                12,735
repolist: 43,472

# yum install rh-postgresql10-postgresql-server -y

# rpm -qa|grep postgresql
rh-postgresql10-postgresql-libs-10.12-2.el7.x86_64
rh-postgresql10-postgresql-server-10.12-2.el7.x86_64
rh-postgresql10-runtime-3.1-1.el7.x86_64
rh-postgresql10-postgresql-10.12-2.el7.x86_64
postgresql-libs-9.2.24-4.el7_8.x86_64

Step:2 Enable the PostgreSQL database port in the firewall service.

# firewall-cmd --add-port=5432/tcp --permanent
success
# firewall-cmd --add-port=5432/tcp 
success

Install the Ansible Tower:

I have customized the role and playbooks atower-pgsql to automate some of the tasks for me which is based on automate-tower-ha-dr. both role are available in the GitHub.


Step:1 To download the atower-pgsql role.

# git clone git@github.com:mh2ict/atower-pgsql.git
Cloning into 'atower-pgsql'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 67 (delta 0), reused 3 (delta 0), pack-reused 64
Receiving objects: 100% (67/67), 26.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (3/3), done.

# cd atower-pgsql/
# ls
ansible.cfg                  hosts              README.md  setup-pgsql-replication.yml  tower-setup.yml           tower-stop-services.yml
check-pgsql-replication.yml  pgsql-promote.yml  roles      tower-set-inventory-ha.yml   tower-start-services.yml

Step:2 To download the Ansible Tower Installer under the atower-pgsql, as below.

# wget https://releases.ansible.com/ansible-tower/setup-bundle/ansible-tower-setup-bundle-latest.el7.tar.gz

# tar xzf ansible-tower-setup-bundle-latest.el7.tar.gz

Step:3 To verify and change two inventory file, as below.

# pwd
atower-pgsql

# grep ^inventory ansible.cfg 
inventory      = hosts
# cat hosts 
[tower]
at1.lab.munshibari.biz ansible_host=192.168.122.11

[database]
db1.lab.munshibari.biz pgsqlrep_role=master

[database_replica]
db2.lab.munshibari.biz pgsqlrep_role=replica

[all:vars]
pg_host='db1.lab.munshibari.biz'


# cat ansible-tower-setup-bundle-3.7.1-1/inventory 
[tower]
at1.lab.munshibari.biz ansible_host=192.168.122.11

[database]

[all:vars]
admin_password='redhat123'

pg_host='db1.lab.munshibari.biz'
pg_port='5432'

pg_database='tower1'
pg_username='awxdbuser1'
pg_password='redhat123'

rabbitmq_port=5672

rabbitmq_username=tower
rabbitmq_password='redhat123'
rabbitmq_cookie=secretvalue
rabbitmq_vhost=at1.lab.munshibari.biz
rabbitmq_use_long_name=true

# 80/443 4369/TCP 5672/TCP 25672/TCP 15672/TCP 5432/TCP
# Isolated Tower nodes automatically generate an RSA key for authentication;
# To disable this behavior, set this value to false
# isolated_key_generation=true

Step:4 To create Passwordless Login Using SSH Keygen.

# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:YIjyODF7jlsLOkRYRsynncVPFJftXKdMf3gTarMfgk0 root@at1.lab.munshibari.biz
The key's randomart image is:
+---[RSA 2048]----+
| +.  . .o..o     |
|  =...o ... . o..|
|=o.+.ooo   o +.=.|
|.O. o. ..   oE+.+|
|= o     S   = o.o|
| *         . + . |
|+ o           o .|
|o+ .           . |
|o..              |
+----[SHA256]-----+

# ssh-copy-id root@at1
# ssh-copy-id root@db1
# ssh-copy-id root@db2

# ssh root@db2 date
# ssh root@db1 date
# ssh root@at1 date
Note: I'm using the root user for the installation, but you can you any non-root account as well.

Step:5 To verify the required RPMS repository.

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.122.11	 at1.lab.munshibari.biz	at1
192.168.122.12	 db1.lab.munshibari.biz	db1
192.168.122.13	 db2.lab.munshibari.biz	db2

# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                              repo name                                                                              status
!rhel-7-server-ansible-2.9-rpms/x86_64               Red Hat Ansible Engine 2.9 RPMs for Red Hat Enterprise Linux 7 Server                      21
!rhel-7-server-extras-rpms/x86_64                    Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                       1,303
!rhel-7-server-rpms/x86_64                           Red Hat Enterprise Linux 7 Server (RPMs)                                               29,413
!rhel-server-rhscl-7-rpms/x86_64                     Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server                12,735
repolist: 43,472

Step:6 To start the Ansible Tower installation.

 # pwd 
atower-pgsql
# cd ansible-tower-setup-bundle-3.7.1-1/
# ./setup.sh

once the installation is successful, we will have the installation summary as below.

Setup PostgreSQL replication:

Step:1 To execute the setup-pgsql-replication.yml playbook, as below.

# pwd 
atower-pgsql
# ansible-playbook -i hosts setup-pgsql-replication.yml

once the replication setup is successful, we will have the replication setup summary as below.

Step:2 To execute the check-pgsql-replication.yml playbook to verify the replication setup, as below.

# pwd 
atower-pgsql
# ansible-playbook -i hosts check-pgsql-replication.yml

once the check-pgsql-replication.yml playbook is successful, we will have the summary as below.

Note: db2 has recovery mode enabled and db1 does not have. and the ansible tower configures with db1.

We can also use the "netstat -atnp" command to verify the connection, as below.


# netstat -atnp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name  
::::::::::::: CUT SOME OUTPUT :::::::::::::
tcp        0      0 192.168.122.13:38474    192.168.122.12:5432     TIME_WAIT   -                   
tcp        0      0 192.168.122.13:38482    192.168.122.12:5432     ESTABLISHED 25783/postgres: wal 
tcp        0      0 192.168.122.13:22       192.168.122.1:42364     ESTABLISHED 11210/sshd: root@pt 
::::::::::::: CUT SOME OUTPUT :::::::::::::

# netstat -atnp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
::::::::::::: CUT SOME OUTPUT :::::::::::::tcp        0      0 192.168.122.12:5432     192.168.122.11:49642    ESTABLISHED 23285/postgres: awx 
tcp        0      0 192.168.122.12:5432     192.168.122.11:49644    ESTABLISHED 23286/postgres: awx 
tcp        0      0 192.168.122.12:5432     192.168.122.13:38482    ESTABLISHED 23283/postgres: wal 
tcp        0      0 192.168.122.12:5432     192.168.122.11:49664    ESTABLISHED 23361/postgres: awx 
tcp        0      0 192.168.122.12:5432     192.168.122.11:49674    ESTABLISHED 23366/postgres: awx 
tcp        0      0 192.168.122.12:5432     192.168.122.11:49682    ESTABLISHED 23395/postgres: awx 
tcp        0      0 192.168.122.12:5432     192.168.122.11:49646    ESTABLISHED 23291/postgres: awx 
tcp        0      0 192.168.122.12:5432     192.168.122.11:49686    ESTABLISHED 23405/postgres: awx 
::::::::::::: CUT SOME OUTPUT :::::::::::::

HA failover to DB2 Server:

Step:1 To stop the PostgreSQL 10 service in DB1 Server.

# systemctl stop rh-postgresql10-postgresql.service

Step:2 To execute the pgsql-promote.yml playbook from the Ansible Tower Server (at1) to promote standby database server as primary.

# pwd 
atower-pgsql
# ansible-playbook -i hosts pgsql-promote.yml

once the pgsql-promote.yml playbook is successful, we will have the summary as below.

Now, the db2 database server is the primary.

Repair Ansible Tower with New PostgreSQL:

Method 1: To change PostgreSQL Database server IP address in the "/etc/tower/conf.d/postgres.py" file and restart the Ansible Tower Services.

FYI: https://access.redhat.com/solutions/3390001

# vi /etc/tower/conf.d/postgres.py
# Ansible Tower database settings.

DATABASES = {
   'default': {
       'ATOMIC_REQUESTS': True,
       'ENGINE': 'awx.main.db.profiled_pg',
       'NAME': 'tower1',
       'USER': 'awxdbuser1',
       'PASSWORD': """redhat123""",
       'HOST': 'db2.lab.munshibari.biz',
       'PORT': '5432',
       'OPTIONS': { 'sslmode': 'prefer',
                    'sslrootcert': '/etc/pki/tls/certs/ca-bundle.crt',
       },
   }
}

# ansible-tower-service restart
# ansible-tower-service status

Method 2: To modify our setup inventory file according to changed Database IP addresses/hostnames and run setup.sh

FYI: https://access.redhat.com/solutions/3785201


# pwd
ansible-tower-setup-bundle-3.7.1-1
# cat inventory 
[tower]
at1.lab.munshibari.biz ansible_host=192.168.122.11

[database]

[all:vars]
admin_password='redhat123'

pg_host='db2.lab.munshibari.biz'

::::::::::::: CUT SOME OUTPUT :::::::::::::
# ./setup.sh

You can verify the Ansible Tower User Interface (GUI), after every step as we have mentioned above from the browser as well.


We can also configure the database replication from the db2 server to db1 as will by swaping database swapping database host information in the hosts inventory file.


Hope this post will help.

1,270 views2 comments