SSH tunnels and IPython

We have an OS X server blade at the lab that’s been suped up with some extra RAM (I think 24GB) and is very useful for our bioinformatics processing, as there are a number of bioinformatics tools installed on it. Additionally, running sequence analysis jobs on the server keeps my personal computer free to do other tasks instead of being swamped by these computationally expensive processes.

With that being the case, it’s also much more convenient to log in to the server remotely instead of having to sit directly at the computer to initiate jobs.

I’ve previously written how I’ve managed to set up SSH and the SSH config file to simplify my life for remotely logging in to the server. That part is pretty straightforward. However, we also use IPython Notebooks to document how we’ve processed data. IPython Notebooks run through a web browser, so how can we still use them when we’ve SSH’d into the remote server via the command line?

It turns out, you can set up a SSH tunnel and then connect to the server through your browser. This does require that you have access to the server, as well as available ports on both your client and server to connect through.

Here’s how I create a tunnel into our server:

ssh -L client_port_Number_over8000:localhost:server_port_Number_over8000 user_name@server_ip_address

That’s a lot to type, especially if you’re logging into the server multiple times a week or even multiple times a day. To get around this, we can edit the SSH config file (found in ~/.ssh/config)

Host easy_to_remember_nickname
HostName IP_address_of_remote_server
User remote_server_login_username
ControlMaster auto
LocalForward localhost:client_portNumber_over8000 localhost:server_portNumber_over8000
ServerAliveInterval 30

With this info in the config file, recreating the tunnel is as easy as typing the following on the command line:

$ssh easy_to_remember_nickname

Once the tunnel is established, open a new Terminal window and SSH into the server. Once the SSH connection is established, change to the directory containing your IPython notebook(s) (Note: this is not needed if you’ve set up an IPython config file that sets the default IPython directory), then launch IPython on the remote machine:

$ipython notebook --no-browser --port=first_port_number_listed_in_tunnel_command

Now, on the machine you’re using, start your browser and enter the following into the URL of your browser:

localhost:first_port_number_listed_in_tunnel_command

This should connect through the tunnel and you’ll have the normal web-based view of your notebooks, but those notebooks are actually housed on a remote machine!

Updating IPython (again)

IPython 2.0 has just been released, so I figured I’d go ahead and update it.

I’ve already had one post dedicated to updating IPython, but that was complicated due to the discrepancy between the version available in the Ubuntu Software Center and the version available in the pip repository. After some struggles, I figured out how to install pip (since pip is not installed with Ubuntu by default) and use pip to download and install the full IPython package:

$sudo pip install ipython[all]

To use pip to update a previously installed package (in my case, IPython), I just typed the following in to Terminal:

$sudo pip install --upgrade ipython

And, it quickly downloaded IPython 2.0 and uninstalled the previous version of IPython (1.2.1). Easy!

samb@Mephistopheles: ~_020

Well, I guess I spoke too soon! Trying to run IPython:

$ipython notebook

I get this error message:

ImportError: The IPython Notebook requires tornado >= 3.1.0, but you have 2.4.1

Great! I have absolutely no idea what “tornado” is, nor am I totally certain how to update it.

Tried

$sudo apt-get tornado

and got this: “E: Invalid operation tornado

For some reason, I decided to try the IPython update command I used above, but add “[all]” to the end of the command like was necessary for the initial installation.

$sudo pip install --upgrade ipython[all]

Guess what! It friggin’ worked! The very first thing that popped up on screen was info regarding updating “tornado”! Can’t believe I got burned by the “[all]” command again when dealing with installing IPython! But, it doesn’t seem logical to me that if you’re updating an existing version of IPython that it wouldn’t update all existing components/dependencies of IPython that are already installed on the system.

Whatever. It’s updated and it’s running!

Can’t update IPython

Well, been trying to get the IPython “nbconvert” command to run, to convert some IPython notebooks to html and/or slides. However, when I type this in Terminal:

samb@Mephisto:~$ ipython nbconvert

I get this output:

[TerminalIPythonApp] File not found: u'nbconvert'

This is weird because “nbconvert” is supposed to be built into IPython. Looking around the web, it seems as though “nbconvert” is only built in to IPython versions >1.0. Let’s check my version of IPython:

samb@Mephisto:~$ ipython --version
0.13.2

So, that explains it. I’m running v0.13.2. This is mildly irritating, as I just installed IPython a week ago. In fact, the latest stable version listed on the IPtyhon install page is 1.2.1 from February.

Selection_011

 

However, I can’t get IPython to update. When I try to use “apt-get”, this is what happens:

samb@Mephisto:~$ sudo apt-get install ipython-notebook
Reading package lists... Done
Building dependency tree
Reading state information... Done
ipython-notebook is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Sooooooo, now what?

I’m going to try to remove IPython via the Ubuntu Software Center.

OK, done.  To be safe, will reboot computer and then verify IPython removal has been completed.

After restarting, typing “ipython notebook” in to Terminal indicates that it is not installed and suggests I can get, and install, IPython by entering “sudo apt-get install ipython”.  However, I’m going to follow the instructions on the IPtyhon install page, which instructs the user to enter: sudo apt-get install ipython-notebook

This is how things turned out:

samb@Mephisto: ~_014

See that?  “Setting up ipython-notebook (0.13.2-2)…”  That’s the same version I had previously!  So, I guess I need to hunt down the more recent version.  At the bottom of the IPtyhon install page there is a link to the IPython Downloads Archive and this is what’s in it:

Selection_015

Lo and behold, the current version, which is MANY versions removed from the version that “apt-get” is retrieving.  Clicking on that link brings me to a page with a “tarball” (a file that ends with “.tar.gz”) and a ZIP file.

Downloaded the tarball http://archive.ipython.org/release/1.2.1/ipython-1.2.1.tar.gz by clicking on the link (I should learn how to do that in the command line).

Un-tarred and un-gzipped the file in Terminal:

samb@Mephisto:~/Downloads$ tar -xzf ipython-1.2.1.tar.gz

I then moved (using the GUI) the un-tarred and un-gzipped ipython-1.2.1 folder to my /home/samb directory and followed the installation directions for installing IPython from source.

The Terminal showed a bunch of stuff. Let’s see if using “nbconvert” does anything.

Result? Typing “ipython nbconvert” spits out a whole bunch of stuff; which is great!

I’ll launch IPython and see if it still sees my configuration file from the initial install/configuration last week.

Didn’t work! Launched IPython (ipython notebook) and this is what I got:

ImportError: No module named jinja2

Looking back at the installing IPython from source page, it doesn’t indicate that required dependencies are not installed by default. Maybe that’s always the case when installing packages from source when using Linux? I have no idea, but it would certainly be nice if the IPython instuctions indicated that, particularly since using “apt-get” doesn’t retrieve the most recent IPython build, thus potentially requiring people to download the source file.

Now that we’ve encountered this problem, I’m going to try to install IPython using “pip”. However, when I try running “pip” from Terminal, I’m informed that “pip” is not installed on the system. So, we’ll get it:

samb@Mephisto:~$ sudo apt-get install python-pip

Now, let’s run (as shown on the IPtyhon install page) :

samb@Mephisto:~$ pip install ipython[all]
Requirement already satisfied (use --upgrade to upgrade): ipython[all] in /usr/local/lib/python2.7/dist-packages
Installing extra requirements: 'all'
Cleaning up...
samb@Mephisto:~$ ipython --version
1.2.1

Not thrilled by the “Requirement already satisfied” output, since I uninstalled the last IPython (used the Ubuntu Software Center Package Manager). And, guess what. The notebook won’t launch and still spits out the error message about no jinja2 module. I’ll try restarting the computer and see if that helps.

And…

It did not help. I’m going to just delete the python2.7 directory found in /usr/local/lib, since that’s where all the IPython stuff seems to live. Deleted the directory:

samb@Mephisto:~$ sudo rm -rv /usr/local/lib/python2.7

rm = remove
-rv = recursively (r), verbose (v) to see all the files as they get deleted

Tried using “pip” again:

pip install ipython

It downloaded the ipython1.2.1 tarball, extracted stuff and then…

couldn’t write to the /usr/local/lib directory due to lack of permissions.

So, I re-ran the command with “sudo” and everything seems to have installed properly and completely. Trying to launch IPython notebook yields…

THE SAME JINJA2 module message!!!

Ahhhhh. Just realized I didn’t run:

sudo pip install ipython[all]

Totally forgot the “[all]”. Let’s try again.

Removed the python2.7 directory, as before. Ran sudo pip install ipython[all]. Ha! Well look at that! Here’s the end portion of the output from the that:

changing mode of /usr/local/bin/rstpep2html.py to 755
changing mode of /usr/local/bin/rst2latex.py to 755
changing mode of /usr/local/bin/rst2html.py to 755
Successfully installed ipython Sphinx pygments jinja2 nose docutils
Cleaning up...

Now, try to launch IPython and… Success!! I have finally managed to upgrade IPython! Not only that, but it has retained my previous configuration set up for the default notebook location. And, finally, test out if “nbconvert” is present. Yep! It’s there (just typed: ipython nbconvert)!

Here’s the quick summary of what I ended up having to do:

1. Had to delete the Python2.7 folder (located at /usr/local/bin).

2. Had to install “pip” (sudo apt-get install python-pip)

3. Used “pip” to retrieve and install most current version of IPython (sudo pip install ipython[all]

 

 

 

Customizing IPython default notebook location

Did some searching around and it turns out, the “easiest” way to do this is to simply navigate (using Terminal) to your desired notebook save location and then launching IPython (just enter the text that’s listed after the $):

samb@Mephisto:~$ ipython notebook

I’ve been using IPython for some months now and never realized that it was simply saving notebooks in my current working directory! Doh!  I could’ve just been changing to the desired directory this entire time!

Knowing this, I can just create a symbolic link to my desired default IPython directory and quickly change to the directory when starting IPython without having to go through a bunch of cd steps to get to the location.

However, if you have a default location you’d like IPython to save your IPyton notebooks to, I found out how to edit some of the IPython profile configuration files on this Stack Overflow entry.

Briefly, here’s what needs to be done.

1. Locate where IPython is installed, using Terminal.

Input:

samb@Mephisto:~$ ipython locate

Output:

/home/samb/.config/ipython

2. My IPython location did NOT contain an “ipython_notebook_config.py” file, so one had to be created. Oh, and to figure out that I did NOT have that Python (.py) file, I just navigated there using the GUI (Nautilus), but I did have to allow for viewing Hidden Files by pressing Ctrl-h. Used Terminal to create this files with the following command (and subsequent output):

The input:

samb@Mephisto:~$ ipython profile create

The output:

[ProfileCreate] Generating default config file: u'/home/samb/.config/ipython/profile_default/ipython_config.py'
[ProfileCreate] Generating default config file: u'/home/samb/.config/ipython/profile_default/ipython_notebook_config.py'

3. I opened my “ipython_notebook_config.py” file with the gedit program (the default program that opens text-based files when I double-click on them) and changed this:

# The directory to use for notebooks.
#c.NotebookManager.notebook_dir = u’/home/samb’

to this:

# The directory to use for notebooks.
c.NotebookManager.notebook_dir = u’/media/B0FE4B1FFE4ADD6A/Users/Samb/Dropbox/Lab/IPython_nbs’

Notice, I took out the hash/pound symbol (#) before the line that has the directory to my desired notebook save location, as the hash symbol indicates that a particular line should NOT be interpreted by whatever program is reading the file.

One thing I did notice is that the Stack Overflow entry indicates there should be two lines that need to be adjusted. However, my ipython_notebook_config.py file only had one of the two entries described, so I simply changed that. I’ll launch IPython and see what happens.

My expectation is that IPython will open and see the existing IPython notebooks in the directory I specified in the ipython_notebook_config.py.

The result?Workspace 1_010

It worked perfectly (see the area surrounded by the red rectangle)!

The biggest benefit of this setup is that I will now have a fixed file path for my IPython notebooks, whether I’m working at home, at work, or some other remote location.  After playing around some more, I may decide to set up a Git repository so that the IPython files are publicly accessible.  I’m also going to try out the Synology Cloud Station (basically the same as Dropbox) functionality on our lab’s server.  I think the latter might be ideal, since I’d be saving my IPython notebooks in our “web” folder, which is already publicly accessible.  But, using a Git repository would also have the benefit of version control, which might be nice.

Whatever the case, I can now quickly and easily modify the default IPython notebook save location!

 

Installing IPython

Installation instructions for IPython are here and it looks like using Linux has an advantage over the other two operating systems; no installation of extraneous software needed!

Just have to open Terminal and enter: sudo apt-get install ipython-notebook

Now, to test out if that worked. In Terminal: ipython notebook

Workspace 1_009

Cool, it’s up and running and I’m not sure it could have been any easier! Next IPython steps will be to configure the default save location for the IPython notebook files.

 

The programs I need

Off the top of my head:

A clipboard manager – Can’t do all the upcoming code copying/pasting without one!

A screen capture tool – One that allows for quick, easy annotation like Skitch.

IPython – An interactive, web-based notebook that allows “recording” and execution of command line (as well as a plethora of other programs like Python and R) programs.

NCBI Standalone BLAST – Used for comparing unknown sequences (nucleotide, proteins) with known sequences.

Dropbox – Synced, cloud-based file storage and sharing.  This will be interesting because I already have Dropbox installed on my Windows partition and I don’t want duplicates of all my files.  Will try to link the Ubuntu Dropbox installation to the Windows Dropbox folder.  Will cover this in a separate post.