JupyterHub and RStudio Server kubernetes documentation

Owned by Christopher Creswell
Last updated: Apr 26, 2023

Motivation and intended usage:

This system is designed for relatively light course usage at an introductory level, so that instructors do not need to go through the process of setting up JupyterLab/Notebook or RStudio Server on dozens or more student owned machines with extremely varied configurations and capabilities, or reserve resources in Lehigh's High Performance Computing clusters.  It is specifically not intended for research usage.

This system provides a uniform experience where JupyterLab or RStudio runs in the user's browser.  Everyone has the same version of the software and its libraries, and it is available for students to use at any time.

The trade-off is that resource usage is constrained.  In order to support many simultaneous users, the containers that run both software packages are only guaranteed a half of a CPU and 512 megs of RAM dedicated to them.  This is more than enough for most introductory work that uses data sets of a few thousand rows or fewer.

For research usage, or for usage that requires more hardware resources than this, users will want to install JupyterLab or RStudio on their own machines, or reserve High Performance Computing resources: https://lts.lehigh.edu/services/high-performance-computing-hpc-research-computing

System Architecture:

JupyterLab and RStudio Server instances each run in containerd.io containers, managed by a kubernetes cluster running in Lehigh's data center. 

JupyterLab instances are started by an application called JupyterHub running at jupyter.cc.lehigh.edu.  This "hub" application starts JupyterLab instances for users and cleans them up when they become idle.  Anyone at Lehigh can login at jupyter.cc.lehigh.edu and start an instance. 

RStudio Server does not come with an equivalent "hub" application for managing instances, so we are using the OpenOndemand application (OOD) to handle this.  It is running at kubeportal.cc.lehigh.edu.  This application runs a series of custom-written scripts to start instances and clean up idle ones.  OOD is the same application that is used to reserve and schedule jobs on HPC resources, so that when users do need access to those resources, they are already familiar with the application used to get access to them.

Starting a JupyterLab instance:

Just login at jupyter.cc.lehigh.edu with your normal Lehigh username and password, then pick a profile from the "Server Options" menu and click "Start".  The profiles describe what's included in them - they determine what programming tools and libraries are available in your JupyterLab instance.  When you're done, just close your browser.  Any files you edit and save will still be there the next time you login.

Starting an RStudio Server instance:

To get an RStudio Server in the kubernetes cluster, you go to https://kubeportal.cc.lehigh.edu, then click on the "Class Apps" dropdown menu along the top of the page and select "RStudio server (kubernetes)" for the generic RStudio instance in kubernetes, or the particular application for your course if applicable (a course can get its own entry here if it maps a network shared drive that is specific to the course).  You are brought to a form where you can select the version of R that you want, and then you can click the "Launch" button.  It takes a minute or two for the instance to start, and then a "Connect to RStudio Server" button appears that takes you to your RStudio instance.

The RStudio Server instance will prompt you for your Lehigh username and password the first time you open it in a browser session.

User home directories:

Your home directory that's mounted inside your JupyterLab or RStudio Server instance is actually stored on a remote NFS server, so any files the you create or modify will persist after your session ends.  The next time you log in, the same files will still be there.  The home directory used by JupyterLab is different from RStudio Server though - you won't see the same files in both. 

These home directories are generally archived at the end of each semester, so the data in them does go away eventually - don't leave anything important in them.

Session management:

You can get back to a running JupyterLab session by just going back to jupyter.cc.lehigh.edu and logging in again.  For as long as it is still running, your RStudio Server is available at https://rstudio.cc.lehigh.edu/<your username>/.  You can also get to it by going back to https://kubeportal.cc.lehigh.edu/ and clicking on the "My Interactive Sessions" link along the top menu bar.

You do not need to do anything to stop either your JupyterLab or RStudio Server instance.  It will be automatically cleaned up after a period of inactivity, currently set to 1 hour.

One caveat about this though - when you are finished using RStudio Server, to avoid an annoying warning message the next time you come back to RStudio Server you may want to click the small red power-button icon in the upper-right corner of the browser window.  This tells RStudio Server that you're leaving the session.  If you don't do this, the next time you log in and get a new RStudio Server instance, you will see a red error message in the RStudio console saying "ERROR The previous R session terminated abnormally" surrounded by some debugging information.  This error is unavoidable if you don't click the button to end your session.  It's a harmless error - everything will still work fine, but it can be scary to see that on session startup. 

User management:

Anyone at Lehigh can get a JupyterLab instance by logging in at jupyter.cc.lehigh.edu, so there's no need to request anything to use it as an individual user.  If you are an instructor intending to use JupyterLab for a course, please do fill out this form to let us know that you'll be using it and include the course and number of students so we can make sure the system has the proper resources to handle demand:

https://jira.cc.lehigh.edu/servicedesk/customer/portal/1/create/254

In order to use the RStudio Server in the kubernetes cluster, users must exist in the OpenOndemand application.  If you are an instructor and want to use this system with a course, please create a help request for the Systems Engineering Team (SET) here:
https://jira.cc.lehigh.edu/servicedesk/customer/portal/1/create/254

Please include a list of all users of the system including all instructors, TAs, and students.  SET will make sure the proper accounts are provisioned for everyone.

Getting Support:

If you have an issue with one of these systems, please call the LTS Help Desk and have them put in a ticket for the Systems Engineering Team (SET).  Please say you're using the LTS JupyterHub or RStudio Server in kubernetes environment.  Include what class you're a part of, and work the the Help Desk staff to get a description of the problem into the ticket.  Screenshots are always helpful.

Known issues:

The following error in RStudio is harmless:

2022-01-26T20:04:48.938685Z [rsession-<username>] WARNING No memory control group found in /proc/self/cgroup; LOGGED FROM: std::__cxx11::string rstudio::core::system::{anonymous}::getMemoryCgroup() src/cpp/core/system/LinuxResources.cpp:335

It shows up on starting an RStudio Server session.  We have not yet found a way to prevent it from happening.  The error is related to a version mismatch between kubernetes, the container engine it's using to run RStudio, and the OS running both.  It should eventually go away when we are able to upgrade everything to use the same version of "cgroups", which are used to control the resources granted to containers.  Everything will still work fine in spite of this error.


Upon using the "plot" function in RStudio, this error may appear:

Warning message:
In grSoftVersion() :
  unable to load shared object '/usr/local/lib/R/modules//R_X11.so':
  libXt.so.6: cannot open shared object file: No such file or directory


This is also harmless - the plot function still works.  The error is caused by the way the container was built - the RStudio images are based on the Rocker project (https://www.rocker-project.org/ ) and this is related to the way they built the image.  An update to the image would fix it - we'll raise an issue in the Rocker project's github to make sure they are aware of it.

Troubleshooting

If your JupyterLab instance isn't starting up, one thing you can try is manually stopping and restarting it.  If you are seeing a screen like this, then click on the "Home" link that's highlighted in a red box:


You should see a screen like this, where you can stop and then re-start your JupyterLab instance:

Click the "Stop My Server" button and wait a minute or two - the screen will change on its own.  The "Stop My Server" button will disappear and there will be one button saying "Start My Server".  Press that and you will be presented with the environment selection menu again, and you'll be able to start a new instance.  You can also access this screen to start/stop your JupyterLab instance from a running instance in the File menu by choosing "Hub Control Panel":


If that doesn't help, then the problem will likely require some work at the back-end to fix.  Please put in a ticket for the Systems Engineering Team at https://jira.cc.lehigh.edu/servicedesk/customer/portal/1/create/200 and we'll look into the issue. 

When submitting the ticket, please mention JupyterHub in the title.  Please include the following information in the ticket details:

  • The device you're using (laptop, desktop, tablet)
  • OS you're using (Mac/Windows/Linux/iOS/Android/whatever else) and what version you have
  • Browser you're using (Chrome/Firefox/Safari/Edge) and what version you have
  • Your location - on-campus or off-campus with VPN
  • The error message you saw
    • A screenshot if possible
  • The name of the file you were editing in your JupyterLab instance
  • What time the error occurred (helpful for narrowing down searches of the application's logs)
  • (Optional) You can also check the browser's "Developer Tools" console to see any errors your browser has received that may not be displayed, and include that in the ticket.

In the past, the following have occasionally caused problems:

  • Code that writes a ton of output to a cell
    • All the output of your code in JupyterLab is written directly into the .ipynb file, so if you have too much output, it can cause JupyterLab to crash when trying to open or save the file.  To avoid this, don't have your code print out every line of a thousand line long input file, and be sure to clear output cells if you do print out a bunch of data.
  • Attempting to export and convert notebook files to PDF/HTML when they contain special characters in code or output
    • The conversion process can be finicky.  There's not much we can do about this other than recommend trying a different format, and making sure your output is plain ASCII characters.