Jupyter Notebooks

Owned by Former user (Deleted)
Last updated: Jan 21, 2020
4 min read

Based on documentation from Yale Research Computing

With a small amount of configuration, you can use a compute node to run a jupyter notebook and access it from your local machine. You will need to be on campus for your connections to work. The main steps are:

  1. Start a jupyter notebook job.

  2. Start an ssh tunnel.

  3. Use your local browser to connect.

Starting the Server

Here is a template for submitting a jupyter-notebook server as a batch job. You may need to edit some of the slurm options, including the time limit or the partition. You will also need to either load a module that contains jupyter-notebook or source activate an environment if you're using Anaconda Python.

Sample SLURM submit script
#!/bin/bash
#SBATCH --partition=lts
#SBATCH --nodes=1
#SBATCH --ntasks-per-node 1
#SBATCH --time=1:00:00
#SBATCH --job-name=jupyter-notebook
#SBATCH --output=jupyter-notebook-%J.log 

# Load Python module
module load anaconda/python3

# get tunneling info
export XDG_RUNTIME_DIR=""

ipnport=$(shuf -i8000-9999 -n1)
ipnip=$(hostname -s)

## print tunneling instructions to jupyter-log-{jobid}.txt
echo -e "
    Copy/Paste this in your local terminal to ssh tunnel with remote
    -----------------------------------------------------------------
    ssh -N -L $ipnport:$ipnip:$ipnport ${USER}@sol.cc.lehigh.edu
    ----------------------------------------------------------------- 

    Then open a browser on your local machine to the following address
    ------------------------------------------------------------------
    localhost:$ipnport
    ------------------------------------------------------------------
    and use the token that appears below to login.

    OR replace "$ipnip" in the address below with "localhost" and copy
    to your local browser.
    "

jupyter notebook --no-browser --port=$ipnport --ip=$ipnip

Modify the above script for resources requested and submit using sbatch. A copy of this script is available at /share/Apps/examples/jupyter/jupyter.slurm. Submit it without modification using the command

sbatch /share/Apps/examples/jupyter/jupyter.slurm

Running Jupyter Notebook interactively on the compute node

If you need to run Jupyter Notebooks interactively, then you should request an interactive session on the partition of your choice.

srun -p //partitionname// --ntasks-per-node=//number of cores per node// -N //number of nodes -t //time requested in HH:MM:SS// --pty /bin/bash

Note that resources may not be available readily and you will need to wait until resources become available. Once resources are available, the scheduler will log you into a computer node. Load the appropriate Anaconda Python module (see available python modules for more information) and run the commands as listed in the above slurm script. For your convenience, the required commands are available in a script file, /share/Apps/examples/jupyter/notebook.sh.

sh /share/Apps/examples/jupyter/notebook.sh

The ssh command required to tunnel connections from your local system to the sol compute node and the jupyter notebook token will be printed on the screen.


Starting the Tunnel

Once you have submitted your job and it starts, your notebook server will be ready for you to connect. You can run squeue -u $(whoami) to check. You will see an “R” in the ST or status column for your notebook job if it is running. If you see a “PD” in the status column, you will have to wait for your job to start running to connect. The log file with information about how to connect will be in the directory you submitted the script from, and be named jupyter-notebook-[jobid].log where jobid is the slurm id for your job.

MacOS and Linux

On a Mac or Linux machine, you can start the tunnel with an SSH command. You can check the output from the job you started to get the specific info you need.

Windows

On a windows machine, we recommend you use MobaXterm. See our guide on connecting with MobaXterm for instructions on how to get set up. You will need to take a look at your job's log file to get the details you need. Then start MobaXterm:

  • Under Tools choose “MobaSSHTunnel (port forwarding)”.

  • Click the “New SSH Tunnel” button.

  • Click the radio button for “Local port forwarding”.

  • Use the information in your jupyter notebook log file to fill out the boxes.

  • Click Save.

  • On your new tunnel, click the key symbol under the settings column and choose your ssh private key.

  • Click the play button under the Start/Stop column.

Alternatively, if you comfortable with the command line on MobaXterm, then start the tunnel with an SSH command.

Browse

Finally, open a web browser on your local machine and enter the address http://localhost:port where port is the one specified in your log file. The address Jupyter creates by default (the one with the name of a compute node) will not work outside the cluster's network. Since version 5 of jupyter, the notebook will automatically generate a token that allows you to authenticate when you connect. It is long, and will be at the end of the url jupyter generates. It will look something like

http://sol-a102:8854/?token=a693ea95fc6b239b33c466fc67cdc3a1a3b78198810abec5

Example terminal output

    Copy/Paste this in your local terminal to ssh tunnel with remote
    -----------------------------------------------------------------
    ssh -N -L 8854:sol-a102:8854 alp514@sol.cc.lehigh.edu
    -----------------------------------------------------------------

    Then open a browser on your local machine to the following address
    ------------------------------------------------------------------
    localhost:8854
    ------------------------------------------------------------------
    and use the token that appears below to login.

    OR replace sol-a102 in the address below with localhost and copy
    to your local browser.

[I 09:40:19.466 NotebookApp] [nb_conda_kernels] enabled, 6 kernels found
[I 09:40:22.584 NotebookApp] [nb_anacondacloud] enabled
[I 09:40:22.591 NotebookApp] [nb_conda] enabled
[I 09:40:22.678 NotebookApp] ✓ nbpresent HTML export ENABLED
[W 09:40:22.678 NotebookApp] ✗ nbpresent PDF export DISABLED: No module named 'nbbrowserpdf'
[I 09:40:22.678 NotebookApp] Serving notebooks from local directory: /home/alp514
[I 09:40:22.678 NotebookApp] The Jupyter Notebook is running at:
[I 09:40:22.679 NotebookApp] http://sol-a102:8854/?token=a693ea95fc6b239b33c466fc67cdc3a1a3b78198810abec5
[I 09:40:22.679 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:40:22.680 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://sol-a102:8854/?token=a693ea95fc6b239b33c466fc67cdc3a1a3b78198810abec5
[I 09:41:21.640 NotebookApp] 302 GET /?token=a693ea95fc6b239b33c466fc67cdc3a1a3b78198810abec5 (128.180.1.26) 1.73ms
[I 09:41:30.524 NotebookApp] Shutting down on /api/shutdown request.
[I 09:41:30.527 NotebookApp] Shutting down 0 kernels

While running this example, the command used to tunnel into sol-a102 was

ssh -N -L 8854:sol-a102:8854 alp514@sol.cc.lehigh.edu

Once the tunnel was established, the following link was used to access Jupyter Notebook for the above example.

http://localhost:8854/?token=a693ea95fc6b239b33c466fc67cdc3a1a3b78198810abec5

If you have trouble or need help, please feel free to contact Research Computing Staff.