Rootless Podman Container And UID/GID Mapping in Ansible Automation Platform 2
Sometimes, we have found the below errors in the Ansible Automation Platform 2 web console while the pulled images in podman failed.
"ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1"
Let's walk through the troubleshooting steps that I followed during the resolution.
Step 1: To verify /etc/subuid and /etc/subgid are set properly.
[root@ansible4 ~]#
[root@ansible4 ~]# su - awx
Last login: Tue Nov 16 12:49:45 +08 2021 on pts/0
[awx@ansible4 ~]$ cat /etc/subuid
ansible:100000:65536
awx:165536:65536
[awx@ansible4 ~]$ cat /etc/subgid
ansible:100000:65536
awx:165536:65536
[root@ansible4 ~]# rpm -qf /etc/subuid
setup-2.12.2-6.el8.noarchNote: The awx user's own id should not be in the range set in /etc/subuid or /etc/subgid.
[awx@ansible4 ~]$ id
uid=991(awx) gid=986(awx) groups=986(awx),988(nginx),989(redis) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Step 2: To verify if the max_pid_namespaces is a reasonable number.
[awx@ansible4 ~]$ sysctl -a | grep namespaces
:::::::::::::CUTSOMEOUTPUT:::::::::::::
user.max_cgroup_namespaces = 18844
user.max_ipc_namespaces = 18844
user.max_mnt_namespaces = 18844
user.max_net_namespaces = 18844
user.max_pid_namespaces = 18844
user.max_time_namespaces = 128
user.max_user_namespaces = 18844
user.max_uts_namespaces = 18844Step 3: To verify the podman command execution using awx user.
[awx@ansible4 ~]$ podman ps
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1[awx@ansible4 ~]$ podman --log-level debug ps
INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level debug ps)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/var/lib/awx/.config/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/awx/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/awx/.local/share/containers/storage
DEBU[0000] Using run root /tmp/podman-run-991/containers
DEBU[0000] Using static dir /var/lib/awx/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-991/libpod/tmp
DEBU[0000] Using volume path /var/lib/awx/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend file
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 4
DEBU[0000] error from newuidmap: newuidmap: open of uid_map failed: Permission denied
DEBU[0000] Failed to add pause process to systemd sandbox cgroup: <nil>
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1Note: we can see, the awx user does not have permission to execute newuidmap command and/or to do some necessary changes using that command.
Step 4: To verify the below file permission and examine file capabilities on that files.
[awx@ansible4 ~]$ getcap /usr/bin/newuidmap && stat /usr/bin/newuidmap
/usr/bin/newuidmap = cap_setuid+ep
File: /usr/bin/newuidmap
Size: 44752 Blocks: 88 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 50380637 Links: 1
Access: (4755/-rwsr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:bin_t:s0
Access: 2021-11-16 12:49:58.960000000 +0800
Modify: 2020-10-26 21:11:02.000000000 +0800
Change: 2021-11-16 12:34:47.565000000 +0800
Birth: -[awx@ansible4 ~]$ getcap /usr/bin/newgidmap && stat /usr/bin/newgidmap
/usr/bin/newgidmap = cap_setgid+ep
File: /usr/bin/newgidmap
Size: 44784 Blocks: 88 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 50380635 Links: 1
Access: (4755/-rwsr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:bin_t:s0
Access: 2021-11-15 22:12:43.257000000 +0800
Modify: 2020-10-26 21:11:02.000000000 +0800
Change: 2021-11-16 12:35:41.569000000 +0800
Birth: -[root@ansible4 ~]# ll /usr/bin/podman
-rwxr-xr-x. 1 root root 50271624 Jul 29 23:03 /usr/bin/podmanThe issue is the setuid bit on newuidmap and newgidmap. And we are going to remove that Set UID permission on those files.
For example, if the user is running as UID 1000 and has an entry of USER:100000:65536, Podman executes the setuid and setgid apps, /usr/bin/newuidmap and /usr/bin/newgidmap, to configure the user namespace. The user namespace then gets the following mapping:
0 1000 1
1 100000 65536To test whether newuidmap is working correctly or not for the user awx.
[awx@ansible4 ~]$ unshare -U sleep 1000 &
[1] 2315[awx@ansible4 ~]$ echo $!
2315
[awx@ansible4 ~]$ newuidmap $! 0 165536 65536
newuidmap: open of uid_map failed: Permission denied[awx@ansible4 ~]$ kill -9 2315It's failed and awx can not map uid.
Step 5: To change the below file permission and remove the Set UID permission.
[awx@ansible4 ~]$ logout[root@ansible4 ~]# ll /usr/bin/newuidmap
-rwsr-xr-x. 1 root root 44752 Oct 26 2020 /usr/bin/newuidmap
[root@ansible4 ~]# chmod u-s /usr/bin/newuidmap[root@ansible4 ~]# ll /usr/bin/newgidmap
-rwsr-xr-x. 1 root root 44784 Oct 26 2020 /usr/bin/newgidmap
[root@ansible4 ~]# chmod u-s /usr/bin/newgidmap[root@ansible4 ~]# ll /usr/bin/newuidmap /usr/bin/newgidmap
-rwxr-xr-x. 1 root root 44784 Oct 26 2020 /usr/bin/newgidmap
-rwxr-xr-x. 1 root root 44752 Oct 26 2020 /usr/bin/newuidmapTo test again whether newuidmap is working correctly or not for the user awx.
[awx@ansible4 ~]$ unshare -U sleep 1000 &
[1] 2334
[awx@ansible4 ~]$ echo $!
2334
[awx@ansible4 ~]$ newuidmap $! 0 165536 65536
[awx@ansible4 ~]$Its works fine.
Step 5: To verify the podman command with uid mapping inside the user namespace.
[awx@ansible4 ~]$ podman unshare cat /proc/self/uid_map
[awx@ansible4 ~]$ podman unshare cat /proc/self/gid_map
Now, the podman command is working fine with awx users. And the above output indicates the uid=0 in the container maps to uid=991 in the host, and uid=1~65536 maps to 165535~231070 in the host.
Note: For more information, see man page user_namespaces
Now, we can execute an ad-hoc command on the target host to verify.
Now, my Ansible Automation Platform is working fine. Below document link may help you to understand more on that.
Understanding user namespaces with rootless containers:
Troubleshooting podman rootless uid/gid mapping:
New container capabilities in Red Hat Enterprise Linux 8.2:
What happens behind the scenes of a rootless Podman container?: