Rootless Podman Container And UID/GID Mapping in Ansible Automation Platform 2
Sometimes, we have found the below errors in the Ansible Automation Platform 2 web console while the pulled images in podman failed.
"ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1"

Let's walk through the troubleshooting steps that I followed during the resolution.
Step 1: To verify /etc/subuid and /etc/subgid are set properly.
[root@ansible4 ~]#
[root@ansible4 ~]# su - awx
Last login: Tue Nov 16 12:49:45 +08 2021 on pts/0
[awx@ansible4 ~]$ cat /etc/subuid
ansible:100000:65536
awx:165536:65536
[awx@ansible4 ~]$ cat /etc/subgid
ansible:100000:65536
awx:165536:65536
[root@ansible4 ~]# rpm -qf /etc/subuid
setup-2.12.2-6.el8.noarch
Note: The awx user's own id should not be in the range set in /etc/subuid or /etc/subgid.
[awx@ansible4 ~]$ id
uid=991(awx) gid=986(awx) groups=986(awx),988(nginx),989(redis) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Step 2: To verify if the max_pid_namespaces is a reasonable number.
[awx@ansible4 ~]$ sysctl -a | grep namespaces
:::::::::::::CUTSOMEOUTPUT:::::::::::::
user.max_cgroup_namespaces = 18844
user.max_ipc_namespaces = 18844
user.max_mnt_namespaces = 18844
user.max_net_namespaces = 18844
user.max_pid_namespaces = 18844
user.max_time_namespaces = 128
user.max_user_namespaces = 18844
user.max_uts_namespaces = 18844
Step 3: To verify the podman command execution using awx user.
[awx@ansible4 ~]$ podman ps
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1
[awx@ansible4 ~]$ podman --log-level debug ps
INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level debug ps)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/var/lib/awx/.config/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/awx/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/awx/.local/share/containers/storage
DEBU[0000] Using run root /tmp/podman-run-991/containers
DEBU[0000] Using static dir /var/lib/awx/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-991/libpod/tmp
DEBU[0000] Using volume path /var/lib/awx/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend file
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 4
DEBU[0000] error from newuidmap: newuidmap: open of uid_map failed: Permission denied
DEBU[0000] Failed to add pause process to systemd sandbox cgroup: <nil>
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1
Note: we can see, the awx user does not have permission to execute newuidmap command and/or to do some necessary changes using that command.
Step 4: To verify the below file permission and examine file capabilities on that files.
[awx@ansible4 ~]$ getcap /usr/bin/newuidmap && stat /usr/bin/newuidmap
/usr/bin/newuidmap = cap_setuid+ep
File: /usr/bin/newuidmap
Size: 44752 Blocks: 88 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 50380637 Links: 1
Access: (4755/-rwsr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:bin_t:s0
Access: 2021-11-16 12:49:58.960000000 +0800
Modify: 2020-10-26 21:11:02.000000000 +0800
Change: 2021-11-16 12:34:47.565000000 +0800
Birth: -
[awx@ansible4 ~]$ getcap /usr/bin/newgidmap && stat /usr/bin/newgidmap
/usr/bin/newgidmap = cap_setgid+ep
File: /usr/bin/newgidmap
Size: 44784 Blocks: 88 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 50380635 Links: 1
Access: (4755/<