Rootless Podman Container And UID/GID Mapping in Ansible Automation Platform 2

Sometimes, we have found the below errors in the Ansible Automation Platform 2 web console while the pulled images in podman failed.

"ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1"

Let's walk through the troubleshooting steps that I followed during the resolution.

Step 1: To verify /etc/subuid and /etc/subgid are set properly.

[root@ansible4 ~]#
[root@ansible4 ~]# su - awx
Last login: Tue Nov 16 12:49:45 +08 2021 on pts/0

[awx@ansible4 ~]$ cat /etc/subuid
ansible:100000:65536
awx:165536:65536

[awx@ansible4 ~]$ cat /etc/subgid
ansible:100000:65536
awx:165536:65536

[root@ansible4 ~]# rpm -qf /etc/subuid
setup-2.12.2-6.el8.noarch

Note: The awx user's own id should not be in the range set in /etc/subuid or /etc/subgid.

[awx@ansible4 ~]$ id

uid=991(awx) gid=986(awx) groups=986(awx),988(nginx),989(redis) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Step 2: To verify if the max_pid_namespaces is a reasonable number.

[awx@ansible4 ~]$ sysctl -a | grep namespaces

:::::::::::::CUTSOMEOUTPUT:::::::::::::

user.max_cgroup_namespaces = 18844
user.max_ipc_namespaces = 18844
user.max_mnt_namespaces = 18844
user.max_net_namespaces = 18844
user.max_pid_namespaces = 18844
user.max_time_namespaces = 128
user.max_user_namespaces = 18844
user.max_uts_namespaces = 18844

Step 3: To verify the podman command execution using awx user.

[awx@ansible4 ~]$ podman ps
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1

[awx@ansible4 ~]$ podman --log-level debug ps
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level debug ps)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/var/lib/awx/.config/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/awx/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/awx/.local/share/containers/storage
DEBU[0000] Using run root /tmp/podman-run-991/containers
DEBU[0000] Using static dir /var/lib/awx/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-991/libpod/tmp        
DEBU[0000] Using volume path /var/lib/awx/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 4              
DEBU[0000] error from newuidmap: newuidmap: open of uid_map failed: Permission denied
DEBU[0000] Failed to add pause process to systemd sandbox cgroup: <nil>
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1

Note: we can see, the awx user does not have permission to execute newuidmap command and/or to do some necessary changes using that command.

Step 4: To verify the below file permission and examine file capabilities on that files.

[awx@ansible4 ~]$ getcap /usr/bin/newuidmap && stat /usr/bin/newuidmap
/usr/bin/newuidmap = cap_setuid+ep
  File: /usr/bin/newuidmap
  Size: 44752         Blocks: 88         IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 50380637    Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:bin_t:s0
Access: 2021-11-16 12:49:58.960000000 +0800
Modify: 2020-10-26 21:11:02.000000000 +0800
Change: 2021-11-16 12:34:47.565000000 +0800
 Birth: -

[awx@ansible4 ~]$ getcap /usr/bin/newgidmap && stat /usr/bin/newgidmap
/usr/bin/newgidmap = cap_setgid+ep
  File: /usr/bin/newgidmap
  Size: 44784         Blocks: 88         IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 50380635    Links: 1
Access: (4755/<