SSH tunnels and IPython

We have an OS X server blade at the lab that’s been suped up with some extra RAM (I think 24GB) and is very useful for our bioinformatics processing, as there are a number of bioinformatics tools installed on it. Additionally, running sequence analysis jobs on the server keeps my personal computer free to do other tasks instead of being swamped by these computationally expensive processes.

With that being the case, it’s also much more convenient to log in to the server remotely instead of having to sit directly at the computer to initiate jobs.

I’ve previously written how I’ve managed to set up SSH and the SSH config file to simplify my life for remotely logging in to the server. That part is pretty straightforward. However, we also use IPython Notebooks to document how we’ve processed data. IPython Notebooks run through a web browser, so how can we still use them when we’ve SSH’d into the remote server via the command line?

It turns out, you can set up a SSH tunnel and then connect to the server through your browser. This does require that you have access to the server, as well as available ports on both your client and server to connect through.

Here’s how I create a tunnel into our server:

ssh -L client_port_Number_over8000:localhost:server_port_Number_over8000 user_name@server_ip_address

That’s a lot to type, especially if you’re logging into the server multiple times a week or even multiple times a day. To get around this, we can edit the SSH config file (found in ~/.ssh/config)

Host easy_to_remember_nickname
HostName IP_address_of_remote_server
User remote_server_login_username
ControlMaster auto
LocalForward localhost:client_portNumber_over8000 localhost:server_portNumber_over8000
ServerAliveInterval 30

With this info in the config file, recreating the tunnel is as easy as typing the following on the command line:

$ssh easy_to_remember_nickname

Once the tunnel is established, open a new Terminal window and SSH into the server. Once the SSH connection is established, change to the directory containing your IPython notebook(s) (Note: this is not needed if you’ve set up an IPython config file that sets the default IPython directory), then launch IPython on the remote machine:

$ipython notebook --no-browser --port=first_port_number_listed_in_tunnel_command

Now, on the machine you’re using, start your browser and enter the following into the URL of your browser:

localhost:first_port_number_listed_in_tunnel_command

This should connect through the tunnel and you’ll have the normal web-based view of your notebooks, but those notebooks are actually housed on a remote machine!

Updating IPython (again)

IPython 2.0 has just been released, so I figured I’d go ahead and update it.

I’ve already had one post dedicated to updating IPython, but that was complicated due to the discrepancy between the version available in the Ubuntu Software Center and the version available in the pip repository. After some struggles, I figured out how to install pip (since pip is not installed with Ubuntu by default) and use pip to download and install the full IPython package:

$sudo pip install ipython[all]

To use pip to update a previously installed package (in my case, IPython), I just typed the following in to Terminal:

$sudo pip install --upgrade ipython

And, it quickly downloaded IPython 2.0 and uninstalled the previous version of IPython (1.2.1). Easy!

samb@Mephistopheles: ~_020

Well, I guess I spoke too soon! Trying to run IPython:

$ipython notebook

I get this error message:

ImportError: The IPython Notebook requires tornado >= 3.1.0, but you have 2.4.1

Great! I have absolutely no idea what “tornado” is, nor am I totally certain how to update it.

Tried

$sudo apt-get tornado

and got this: “E: Invalid operation tornado

For some reason, I decided to try the IPython update command I used above, but add “[all]” to the end of the command like was necessary for the initial installation.

$sudo pip install --upgrade ipython[all]

Guess what! It friggin’ worked! The very first thing that popped up on screen was info regarding updating “tornado”! Can’t believe I got burned by the “[all]” command again when dealing with installing IPython! But, it doesn’t seem logical to me that if you’re updating an existing version of IPython that it wouldn’t update all existing components/dependencies of IPython that are already installed on the system.

Whatever. It’s updated and it’s running!

Customizing IPython default notebook location

Did some searching around and it turns out, the “easiest” way to do this is to simply navigate (using Terminal) to your desired notebook save location and then launching IPython (just enter the text that’s listed after the $):

samb@Mephisto:~$ ipython notebook

I’ve been using IPython for some months now and never realized that it was simply saving notebooks in my current working directory! Doh!  I could’ve just been changing to the desired directory this entire time!

Knowing this, I can just create a symbolic link to my desired default IPython directory and quickly change to the directory when starting IPython without having to go through a bunch of cd steps to get to the location.

However, if you have a default location you’d like IPython to save your IPyton notebooks to, I found out how to edit some of the IPython profile configuration files on this Stack Overflow entry.

Briefly, here’s what needs to be done.

1. Locate where IPython is installed, using Terminal.

Input:

samb@Mephisto:~$ ipython locate

Output:

/home/samb/.config/ipython

2. My IPython location did NOT contain an “ipython_notebook_config.py” file, so one had to be created. Oh, and to figure out that I did NOT have that Python (.py) file, I just navigated there using the GUI (Nautilus), but I did have to allow for viewing Hidden Files by pressing Ctrl-h. Used Terminal to create this files with the following command (and subsequent output):

The input:

samb@Mephisto:~$ ipython profile create

The output:

[ProfileCreate] Generating default config file: u'/home/samb/.config/ipython/profile_default/ipython_config.py'
[ProfileCreate] Generating default config file: u'/home/samb/.config/ipython/profile_default/ipython_notebook_config.py'

3. I opened my “ipython_notebook_config.py” file with the gedit program (the default program that opens text-based files when I double-click on them) and changed this:

# The directory to use for notebooks.
#c.NotebookManager.notebook_dir = u’/home/samb’

to this:

# The directory to use for notebooks.
c.NotebookManager.notebook_dir = u’/media/B0FE4B1FFE4ADD6A/Users/Samb/Dropbox/Lab/IPython_nbs’

Notice, I took out the hash/pound symbol (#) before the line that has the directory to my desired notebook save location, as the hash symbol indicates that a particular line should NOT be interpreted by whatever program is reading the file.

One thing I did notice is that the Stack Overflow entry indicates there should be two lines that need to be adjusted. However, my ipython_notebook_config.py file only had one of the two entries described, so I simply changed that. I’ll launch IPython and see what happens.

My expectation is that IPython will open and see the existing IPython notebooks in the directory I specified in the ipython_notebook_config.py.

The result?Workspace 1_010

It worked perfectly (see the area surrounded by the red rectangle)!

The biggest benefit of this setup is that I will now have a fixed file path for my IPython notebooks, whether I’m working at home, at work, or some other remote location.  After playing around some more, I may decide to set up a Git repository so that the IPython files are publicly accessible.  I’m also going to try out the Synology Cloud Station (basically the same as Dropbox) functionality on our lab’s server.  I think the latter might be ideal, since I’d be saving my IPython notebooks in our “web” folder, which is already publicly accessible.  But, using a Git repository would also have the benefit of version control, which might be nice.

Whatever the case, I can now quickly and easily modify the default IPython notebook save location!