My home is perhaps not located in the most urbanized of places. We had broadband a while back but very limited offerings. The only provider that existed - BSNL provided were absurdly slow and not value for money. At the time Jio came up as an alternative, I had asked BSNL office if they’d ever have any better offerings. They responded negative for the near future. The broadband-connection had been scrapped since then, and I had switched the home connection completely to 4G after some speedtest benchmarks. Something I have to look forward to is Fiber to the Home (FTTH) technology has presently made ways to where my home is and I might just get a good enough connection soon. I might have to bear with a little delay though. Thus fast forward a year or two from my earlier change of internet back home - I am back in a situation where I need a stronger connection, or manage with Jio amidst connection fluctuations etc etc.

On a recent conversation with a colleague at IIIT Hyderabad who started with MobaXterm on Windows with vanilla SSH, I realized many people don’t know these methods. This would be the umpteenth time someone has brought these troubles in front of me, and I’d rather have a document which I can point to going ahead than having to describe each time. On another thread, I have realized Sukanta Sen, soon to be work-colleague and Prof. Kenneth Heafield, my new boss have also shared some troubles with working remotely connecting to servers. They have suggested mosh, which I have already been using for about 2-3 years now and will happily start using in the new environment soon.

With COVID-19 lockdown situation and being stuck to working from home often experiencing interruptions with the usual providers, this is perhaps a good time time to write-up what I have been doing for a while. There are several tools which I used in the past to work around a horrible network connection. This post is specific to to maintain SSH-like sessions resilient to network drops and to avoid disruptions to the an overall programming workflow.

Try to find a decent *nix shell.

I haven’t historically liked the alternative offerings of PuTTY, MobaXterm, Cygwin etc. I would not thus recommend these, but I guess in the end it’s a matter of taste. Normally, working with servers have been easier for me in Linux. Since I got my new laptop and saw Windows Subsystem for Linux as a decent alternative, I have since shifted. I have an Ubuntu 18.04 currently running within my Windows OS. The following should be decent starting points to replicate my recommendation of an environment with a Windows machine.

On this *nix setup and one such on the remote, the content ahead describes how to get the following tooling working:

  • tmux: installing, auto-attaching/save-resume sessions
  • mosh: building connection drop resilience and better response in UX.
  • JuiceSSH: take occassional checks when you are travelling.

Always leave a session on the server: tmux.

I always prefer to leave tmux sessions running on a server that is guaranteed to have network connectivity. At IIIT-Hyderabad, my former university and NAVER LABS Europe where I recently finished my internship, this had been the gateway nodes which allow for such operation. In IIIT - this was atom or ada, the headnodes to the HPC clusters I worked with.

If you have tmux (one of the recent versions) installed already on this server, life-becomes really easy. Often server-admins prefer ancient CentOS and tmux versions, which I do not find to my taste. For the uninitiated, it’s actually really easy to install a more recent version of tmux by yourself without root-permissions. The only dependency tmux has is perhaps libevent.

Installing tmux locally.

The following is subject to change, as the source of the following packages change. At the time this post is being written:

You can find a recent version of tmux at:

How you install any package locally is usually very simple, once you know the linux install routine.

./configure --prefix=$HOME/.local/
make && make install

I have not checked this, but usually, the routine throws up a complaint when you try to do the above alone about a libevent dependency. Normally this is taken care of a package manager, but if you’re not root you’d have some additional trouble. Thankfully, tmux has in the past installed for me with just one additional dependency - which is libevent. Releases can be found somewhere below.

Configuring tmux.

There are a lot of ninja-techniques to configure as well. I often try to stick with the bare minimum and defaults (not changing modifiers etc), so I can get started on the next server with minimum configuration change requirements.

You can find several good enough tutorials to configure tmux online.

auto-attach sessions!

I change machines between my laptop, desktop and phone while connecting to the servers described above. A desirable feature is consistency among all three, and saved sessions with auto-attach at the point of connection. I normally use only one tmux session and multiplex using panes or splits.

I have the following script which auto-attaches a tmux if it is already running, otherwise creates one for me. Add the snippet in your ~/.bashrc or ~/.bash_profile on the server.

# HOSTNAME checks and launches tmux only on the ada head-node.
# Checks if already within a tmux as well.
if [ $HOSTNAME = "ada.iiit.ac.in" ] && [ -z "$TMUX" ] ; then
    # attach if running    || launch new
    tmux attach -dt "main" || tmux new -s "main"
fi

The above snippet is specific to ada.iiit.ac.in, but you can come up with something similar based on your setup.

If you’re not using bash, which is default on most systems, you should be knowledgeble enough to find the runtime-configuration file where this goes. The if guard is so that when ~/.bashrc is run again on a compute node, it doesn’t create tmux-ception.

The mobile shell

Now comes the next major problem. SSH. The problem with SSH is that if it disconnects it won’t reconnect back automatically. It simply hangs. Also, when you SSH, the shell waits for response to come back to echo the result of what you entered. The slowness here is extremely disruptive while developing or running something on the server. To overcome some of this and build resilience to connection disruptions we will discard SSH in favour of Mosh or the Mobile Shell.

The thing is, you need mosh installed at the server. On atom/cosmos, I had been root - so I can install mosh on every node I want with one command and ansible. On ada, I have enough pull to get the HPC admin to install it for me. At NAVER, this was not the case - and I installed a local version of mosh myself and configured it. A rough procedure to accomplish something similar is described below.

Configuring own install.

I am not going to go in detail, but the same configure, make and make-install routine can get mosh in your local folder. But by default, mosh doesn’t look for a local mosh installation or doesn’t assume the path where it is installed. In this case

# fill remote home below.
# In case of ada - this would be /home/$USER/.local
HOME=... 

# Launch mosh specifying location of where the mosh-server is installed.
mosh --server=$HOME/.local/bin/mosh-server <user>@<server-host>

Local edits are always more convenient.

You might think with all the above in space, you are really good to go. While mosh gives you local-echo and connection drop resilience, it still disconnects and disrupts your workflow. Have a look at this comic, which is relatable in the context.

Why you shouldn't interrupt a programmer?

The network part is still a blocking-bottleneck and a point of failure. I believe it’s fairly obvious that it is best to make edits and run locally. However, my personal machine which I optimized for portability barely has the power to run the demanding jobs I run. Editing the text source files which are run-eventually, any tiny machine can - even your android phone!

How I manage the changes consistently across different servers, and my personal machine which I use for edits is git. Any version control system with a central remote you can pull to make everything consistent works.

Bringing git into the workflow…

To enhance the workflow one more level further, I would recommend editing code through a git enabled repository which can sync at the server through a simple git-pull. This ensures your edits are visible as you make them and only running the command is what is dependent on the server.

  1. Create an edit branch and sneak git-pulls into a build/run file at the server.
  2. The point where you get a working version of the source, merge the branch as one commit to the main code to avoid polluting commit history.

On Phone: JuiceSSH

One might not always have a laptop with Windows 10 and WSL, or Ubuntu running while travelling. This was the case at times when I was volunteering as CVIT sys-admin, and while I had been travelling in France. Sadly, there is no mosh for mobile, but I have find JuiceSSH to be decent enough for quick tasks. If you enable the auto-attaching tmux sessions, JuiceSSH can be used with minimum disruptions to occassionally check on your jobs and maybe edit if necessary to tiny degrees.JuiceSSH is available for install on the Google Play Store.

There are other alternatives as well, but JuiceSSH is what I like. Soon, I think I might even end up paying for the premium features the developer provides.

Sharing Text

Often, I end up in a situation where I have to share text generated in these servers to another place (maybe my local machine, maybe to share results with a colleague). The issue with the above setup is that often, these outputs can’t be copy-pasted using mouse (as the terminal-mosh-tmux-server intermediates will lose the clipboard).

Pastebin services

What I rely on is a command-line paste-service. I used to use ptpb.pw. However, this service has gone down since then due to abuse. Of late, I have stuck to ix.io. It even offers a user through .netrc based login, which is not very safe but I have been a happy user of. I have found this service satisfactory enough for now. You can substitute for the same with any paste-provider with a command line service. I will point to the possibilities opened by such a service and leave for you to google for the rest.

Epilogue

This post was drafted on a TGV with the onboard WiFi, as I’m not sure how otherwise to kill time on a 4.5 hour trip to Paris from Toulouse. I have wrapped up my 6-month internship at NAVER LABS Europe, stayed a month with Phani while attempting a UK Visa, which has unfortunately failed. From Paris, I’m hoping I can take the Vande Bharat (CDG - DEL - COK), to arrive home in about 48 hours of travel.

Forgive any typos or errors, and please be kind-enough to point them out so I can correct if they exist. If you know better methods to stay connected and edit code, run them on servers, please do let me know in comments below.