Linux – Check process of a single SSH tunnel in a shell script


On a remote lab system, I have a shell script that initiates a reverse SSH tunnel to my main jumpbox that gets run with a cron job every 5 minutes. If the SSH tunnel is up nothing happens, if it is down it initiates it.

createTunnel() {
/usr/bin/ssh -N -R :2222:localhost:22
if [[ $? -eq 0 ]]; then
 echo Tunnel to jumpbox created successfully
 echo An error occurred creating a tunnel to jumpbox. RC was $?
/bin/pidof ssh
if [[ $? -ne 0 ]]; then
  echo Creating new tunnel connection

This has been extremely reliable to ensure my access to the remote machine if it gets rebooted or as my jumpbox IP changes.
However, I recently added a second SSH tunnel to this system and had a situation where one of the two tunnels went down and was never re-established. It appears that since there was one tunnel up, the pidof output still returned with a PID so the script never ran "createTunnel".
Since I have two SSH tunnels, the pidof output shows both PID's:

$ /bin/pidof ssh
28281 28247

How can I adjust my script to determine if only one of the tunnels is down?

Best Answer

My thoughts:

  1. Pid files. I think this is a general approach.
  2. XY problem, use autossh.

(Some other answers may elaborate these ideas.)

  1. There are two scripts, right? (not exactly complying with DRY). Give them different names. Your script waits for its ssh to exit. The additional instance, instead of checking pid(s) of ssh, should check pid(s) of its own name. Exactly one pid two pids mean no previous script is still running:

    [ $(pidof -x scriptname | wc -w) -eq 2 ] && createTunnel

    Why two? Because $() starts a subshell that also counts. Quite dirty, oh well.

  2. Somewhat outside-the-box, I think. Create two symlinks with unique names, like:

    ln -s /usr/bin/ssh ssh-foo
    ln -s /usr/bin/ssh ssh-bar

    Let the first script run ssh-foo, let the second run ssh-bar. Their pidof invocations should target ssh-foo or ssh-bar respectively as well. This way they won't mix. As a bonus, additional ssh (possibly run for a completely different reason in the future) won't affect them.

And finally:

  1. You don't have to check anything.

    /usr/bin/ssh -o ExitOnForwardFailure=yes -N -R :2222:localhost:22

    If the tunnel already exists, port forwarding will fail for sure and ssh will exit. I guess you can run this line directly from a crontab, no script is required.