Revisiting the restricted shell

I've been administering Unix boxes since the mid-90s and I've always been told that using restricted shells (rsh, rksh, rbash) was a bad idea because they are easily hackable. Indeed, there are countless known methods to get out of a restriced shell: from finding an application that allows a shell escape, to trying to compile your own, to doing clever hacks with the history file.

I've recently been in a corner case where I was dealing with an embedded product which requires a specific set of commands and also uses some bracket commands that are difficult to wrap with our usual SSH command authenticator. So I decided to revisit using a restricted shell to jail this user and I think I managed to make the jail shatterproof enough.

Here is how I did it:

Create Bob's home directory, but assign it to root:
# mkdir /home/bob
# chown root:root /home/bob
# chmod 755 /home/bob

Force a .bashrc and .profile that changes Bob's PATH to a limited set of commands:
# echo "export PATH=/opt/arcbck/allowed_commands" > .bashrc
# ln -s .bashrc .profile

The reason for having both a .profile and a .bashrc is to ensure that this profile will be loaded both for interactive and non-interactive sessions.

If the user needs to write stuff somewhere, create a directory for Bob, e.g.
# mkdir /home/bob/writable
# chown bob home/bob/writable
# chmod 755 /home/bob/writable

Create the allowed_commands directory and put symlinks in it pointing to allowed binaries:
# mkdir /home/bob/allowed_commands
# ln -s /bin/mycmd allowed_commands/mycmd

Now you must be sure of the following:

1. Bob must NOT have any writable access to /home/bob/.profile or /home/bob/.bashrc, else he can change the PATH value
2. Bob must NOT have any writable access to /home/bob, to prevent any modification of .profile and .bashrc
3. Investigate ANY command that ends up in the allowed_commands jail to be sure that there is NO known way of executing another command from it, showing files or escaping the shell. If there are any, then forfeit giving this command or write a wrapper around it (see below).
4. See the jail escape methods linked above, log in as Bob and see if you can use them to escape the jail.

Example of a wrapper script with scp

Let's say I want to allow Bob to scp files into his account using scp's undocumented -t (i.e. -to) option. I would normally do this:
# ln -s /bin/scp allowed_commands/scp

This is wrong as scp can be coerced with -S to execute random commands.

A solution is to put the following in the allowed_commands jail instead:
lrwxrwxrwx. 1 root root   14 May  5 10:02 scp -> scp_wrapper.sh
-rwxr-xr-x. 1 root root  382 May  5 13:54 scp_wrapper.sh

With scp_wrapper.sh containing this:

#!/bin/sh

if [[ "$1" = "-t" && "$2" != "-"* ]]

then

        /bin/scp -t $2

        returncode=$?

else

        echo "scp_wrapper: Refused SCP command: '$*'"

        returncode=255

fi

exit ${returncode}

Using this wrapper, scp will only allow -t and no other option.

Good luck.

Applications crash on SLES12 due to lock elision

This issue has been discussed in other places, but mostly related to specific applications and I think it needs its own post here for those who would stumble on this following a Google search.

glibc 2.18, released in 2013, came with a new feature named TSX Lock Elision.

Briefly, this feature changes the behaviour of libpthread.so in the way it handles mutexes on some specific processors that support hardware lock elision. Intel Xeon CPUs, in particular, support TSX since around 2013 or so. Lock elision offers significant performance gains for some software such as databases.

You can see if your Linux server's CPU supports lock elision by checking /proc/cpuinfo. If it mentions "hle" (hardware lock elision), it does.

RHEL 7 does not support this as of now. It comes with glibc 2.17, so lock elision is not enabled on these systems. As for SLES12, it comes with glibc 2.19, which means that SLES12 systems will use lock elision if the CPU supports it.

However, if an application unlocks a mutex twice, this can cause problems if lock elision is enabled. This is explained in detail in an LWN article. Let me quote an important paragraph in this article:

pthread_mutex_unlock() detects whether the current lock is executed transactionally by checking if the lock is free. If it is free it commits the transaction, otherwise the lock is unlocked normally. This implies that if a broken program unlocks a free lock, it may attempt to commit outside a transaction, an error which causes a fault in RTM. In POSIX, unlocking a free lock is undefined (so any behavior, including starting World War 3 is acceptable). It is possible to detect this situation by adding an additional check in the unlock path. The current glibc implementation does not do this, but if this programming mistake is common, the implementation may add this check in the future.

The "programming mistake" here is double-unlocking mutexes. I've made a sample C program that does exactly this, and although it works fine with glibc 2.17, it will crash on glibc 2.19 with a segmentation fault in __lll_unlock_elision(), if, and only if, the server's cpuinfo reports "hle".

I've stumbled upon a few applications, which I will not name here, that crash on SLES12. Upon analyzing their cores, I found that they have this same exact problem with __lll_unlock_elision(). So, one can assume that they might double-unlock some mutexes.

The bottom line is that if you have an app that does this, your best bet is to contact the vendor, and ask them to remove double mutex unlocks in their code, if they have any.

If that is not possible, there are two workarounds:

1. The first is to patch /etc/ld.so.conf to override libpthread 2.19 with a version that is compiled with lock elision disabled. This is documented in Novell's KB here.

2. The second (and preferred) solution is to adjust LD_LIBRARY_PATH to override it on a per-application basis. You could therefore change its startup script to add this:

LD_LIBRARY_PATH=/lib64/noelision:/lib/noelision:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH

Hope this helps.