Byung Kyu Park's Personal Website

When you have to maintain a few servers and workstations, it’s useful to have remote access. However, this can be difficult if the workstation has to be put behind a NAT router for lack of IP addresses or ethernet ports. By having all these workstations/servers behind NAT routers make ssh connection to one central server with a reverse tunnel, you can maintain ssh accessibility to all your servers.

Basic Idea

Here’s the basic setup. From your workstation, run

ssh -R 31337:localhost:22 central_server

where “central_server” should be replaced with the IP address or FQDN of your central server. Once you have done that, from your central server, you can run,

ssh -p 31337 localhost

in order to access that workstation.

If you need to setup additional workstations, you will need to use different port addresses (31338, 31339, etc., etc.) as only one process can listen to a port at the time. if the connection gets cut for some reason and you need to reconnect, watch out for messages like “Warning: remote port forwarding failed for listen port 31337″. That means some other process (including zombie processes) is using the port and the forwarding is probably not set up correctly. In order to correct this, you can kill all processes named “sshd” by running the following command on the server:

pkill -u user sshd

where “user” is the username of the user which is usually used for setting up these reverse tunnels.

Please note that this will likely cut the reverse tunnel for any other machines connected to the central server as well.

In fact, lost connection is a somewhat serious problem as it defeats the purpose of this setup as well. So it is necessary to use cron to monitor and re-establish this connection automatically if it gets cut.

An Implementation

Following is a shell script (written for ksh, mainly for the handy PID references) that can be run in cron to run, for example, every minute; I use keychain so this script utilizes the ssh-agent initialized by keychain; else you need to otherwise enable passwordless ssh login somehow:

#!/bin/ksh

# connect to ssh-agent instance
source ${HOME}/.keychain/local_server-sh

# read PID of existing session (if any)
OLDPID=`cat ${HOME}/.keepsshpid`

if [ -n "$OLDPID" ]
then
    # check if they are still running
    for pid in $(pgrep keepssh)
    do
        if [ $pid -eq $OLDPID ]
        # it's still running; exit.
        then
            return 0
        fi
    done
fi

# if we haven't exited, old session is gone; go into infinite loop
# maintaining an ssh connection, after storing a new PID

echo $$ > ${HOME}/.keepsshpid

while true
do
   ssh -NR 31337:localhost:22 central_server
   sleep 10
done

In short, there are two mechanisms in place to maintain the ssh connection, if it is cut from the server side. First is the infinite loop which re-runs the ssh command 10 seconds after it finishes (i.e. ssh connection is terminated somehow). Second is this script place in cron—when run by cron, if it detects that the previous instance of this script is not running, it re-starts the infinite loop (otherwise it exits so that existing connections are not disrupted).

Sometimes, the reconnection effort can be made too soon and the port forwarding may not be properly established. This can usually be fixed by cutting the ssh connection from central_server by killing the appropriate sshd process (the next connection attempt will probably establish the port forwarding correctly).