Performance assessment and tuning experiences with RHEL

POC on Red Hat Directory Server (RHDS)

Migrate Elasticsearch Index to OpenSearch Index.

Rootless Podman Container And UID/GID Mapping in Ansible Automation Platform 2

How To Change Control Plane Execution Environment Settings

Install SSL Certificates for Ansible Automation Controller and Automation Hub

Search

Munshi Hafizul Haque
- Nov 16, 2021
- 4 min read

Rootless Podman Container And UID/GID Mapping in Ansible Automation Platform 2

Sometimes, we have found the below errors in the Ansible Automation Platform 2 web console while the pulled images in podman failed.

"ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1"

Let's walk through the troubleshooting steps that I followed during the resolution.

Step 1: To verify /etc/subuid and /etc/subgid are set properly.

[root@ansible4 ~]#
[root@ansible4 ~]# su - awx
Last login: Tue Nov 16 12:49:45 +08 2021 on pts/0

[awx@ansible4 ~]$ cat /etc/subuid
ansible:100000:65536
awx:165536:65536

[awx@ansible4 ~]$ cat /etc/subgid
ansible:100000:65536
awx:165536:65536

[root@ansible4 ~]# rpm -qf /etc/subuid
setup-2.12.2-6.el8.noarch

Note: The awx user's own id should not be in the range set in /etc/subuid or /etc/subgid.

[awx@ansible4 ~]$ id

uid=991(awx) gid=986(awx) groups=986(awx),988(nginx),989(redis) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Step 2: To verify if the max_pid_namespaces is a reasonable number.

[awx@ansible4 ~]$ sysctl -a | grep namespaces

:::::::::::::CUTSOMEOUTPUT:::::::::::::

user.max_cgroup_namespaces = 18844
user.max_ipc_namespaces = 18844
user.max_mnt_namespaces = 18844
user.max_net_namespaces = 18844
user.max_pid_namespaces = 18844
user.max_time_namespaces = 128
user.max_user_namespaces = 18844
user.max_uts_namespaces = 18844

Step 3: To verify the podman command execution using awx user.

[awx@ansible4 ~]$ podman ps
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1

[awx@ansible4 ~]$ podman --log-level debug ps
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level debug ps)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/var/lib/awx/.config/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/awx/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/awx/.local/share/containers/storage
DEBU[0000] Using run root /tmp/podman-run-991/containers
DEBU[0000] Using static dir /var/lib/awx/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-991/libpod/tmp        
DEBU[0000] Using volume path /var/lib/awx/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 4              
DEBU[0000] error from newuidmap: newuidmap: open of uid_map failed: Permission denied
DEBU[0000] Failed to add pause process to systemd sandbox cgroup: <nil>
ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": cannot setup namespace using newuidmap: exit status 1

Note: we can see, the awx user does not have permission to execute newuidmap command and/or to do some necessary changes using that command.

Step 4: To verify the below file permission and examine file capabilities on that files.

[awx@ansible4 ~]$ getcap /usr/bin/newuidmap && stat /usr/bin/newuidmap
/usr/bin/newuidmap = cap_setuid+ep
  File: /usr/bin/newuidmap
  Size: 44752         Blocks: 88         IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 50380637    Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:bin_t:s0
Access: 2021-11-16 12:49:58.960000000 +0800
Modify: 2020-10-26 21:11:02.000000000 +0800
Change: 2021-11-16 12:34:47.565000000 +0800
 Birth: -

[awx@ansible4 ~]$ getcap /usr/bin/newgidmap && stat /usr/bin/newgidmap
/usr/bin/newgidmap = cap_setgid+ep
  File: /usr/bin/newgidmap
  Size: 44784         Blocks: 88         IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 50380635    Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:bin_t:s0
Access: 2021-11-15 22:12:43.257000000 +0800
Modify: 2020-10-26 21:11:02.000000000 +0800
Change: 2021-11-16 12:35:41.569000000 +0800
 Birth: -

[root@ansible4 ~]# ll /usr/bin/podman
-rwxr-xr-x. 1 root root 50271624 Jul 29 23:03 /usr/bin/podman

The issue is the setuid bit on newuidmap and newgidmap. And we are going to remove that Set UID permission on those files.

For example, if the user is running as UID 1000 and has an entry of USER:100000:65536, Podman executes the setuid and setgid apps, /usr/bin/newuidmap and /usr/bin/newgidmap, to configure the user namespace. The user namespace then gets the following mapping:

0     1000      1
1     100000    65536

To test whether newuidmap is working correctly or not for the user awx.

[awx@ansible4 ~]$  unshare -U sleep 1000 &
[1] 2315

[awx@ansible4 ~]$ echo $!
2315
[awx@ansible4 ~]$ newuidmap $! 0 165536 65536
newuidmap: open of uid_map failed: Permission denied

[awx@ansible4 ~]$ kill -9 2315

It's failed and awx can not map uid.

Step 5: To change the below file permission and remove the Set UID permission.

[awx@ansible4 ~]$ logout

[root@ansible4 ~]# ll /usr/bin/newuidmap
-rwsr-xr-x. 1 root root 44752 Oct 26  2020 /usr/bin/newuidmap
[root@ansible4 ~]# chmod u-s /usr/bin/newuidmap

[root@ansible4 ~]# ll /usr/bin/newgidmap
-rwsr-xr-x. 1 root root 44784 Oct 26  2020 /usr/bin/newgidmap
[root@ansible4 ~]# chmod u-s /usr/bin/newgidmap

[root@ansible4 ~]# ll /usr/bin/newuidmap /usr/bin/newgidmap
-rwxr-xr-x. 1 root root 44784 Oct 26  2020 /usr/bin/newgidmap
-rwxr-xr-x. 1 root root 44752 Oct 26  2020 /usr/bin/newuidmap

To test again whether newuidmap is working correctly or not for the user awx.

[awx@ansible4 ~]$  unshare -U sleep 1000 &
[1] 2334
[awx@ansible4 ~]$ echo $!
2334
[awx@ansible4 ~]$ newuidmap $! 0 165536 65536
[awx@ansible4 ~]$

Its works fine.

Step 5: To verify the podman command with uid mapping inside the user namespace.

[awx@ansible4 ~]$ podman unshare cat /proc/self/uid_map
[awx@ansible4 ~]$ podman unshare cat /proc/self/gid_map

Now, the podman command is working fine with awx users. And the above output indicates the uid=0 in the container maps to uid=991 in the host, and uid=1~65536 maps to 165535~231070 in the host.