Upgrade

Sol Upgrade (2025)

This user guide will outline the steps required for Sol users to continuing using the cluster after the planned January 6-8, 2025 operating system (OS) upgrade.

Contents:

1. Brief Summary
2. Motivation
3. Why should I care?
4. Alma, our new OS
5. Take Action
6. Transition Period
7. Containers as a Backstop
8. Managing Research Software

Brief Summary

1. The LTS Research Computing group will reinstall a new operating system on Sol starting with a downtime on January 6-8, 2025.
2. Those who use custom codes will need to recompile. Research Computing can test your codes before the downtime if you provide advance notice by submitting an HPC help ticket. You must provide high-quality documentation, including a complete list of dependencies and detailed instructions for compiling the code. We will compile and test your code on our test cluster. If we succeed, we will provide you with revised instructions for compiling it yourself after the upgrade.
3. Anyone who uses virtual environments are encouraged to export or freeze their environments to create an explicit list of dependencies. They can use this list to rebuild their environments after the upgrade.
4. Users who leverage our modules system for software are encouraged to review our list of popular software below. If your software is not on this list, please open a ticket to tell us about any modules you want to continue using after the upgrade. Our team will build the most popular modules along with any specific requests from the users. Whenever possible, users are encouraged to upgrade to the latest versions following the upgrade. We expect to retire aging codes that are not used by a significant number of Sol users.

Any users who would like us to test their compiled code, build specific modules for use after the upgrade, or have any questions about this process should submit an HPC ticket.

Important Terminology

Modules. In this guide we will refer extensively to "software modules" or just "modules". These refer to software modules delivered by Lmod. Software modules organize the majority of the software used on most HPC systems. We compile software modules so you can use the system more effectively without building or upgrading the software yourself. We sometimes refer to this as a "module tree" because we utilize hierarchical modules, a feature which ensures consistency across your toolchain. Even in cases where researchers compile their own codes, they almost always use the modules system because it has been certified to work properly on our hardware. In particular, the most important pieces of software on an HPC cluster are the message passing interface (MPI) and the scheduler (SLURM), which cooperate to talk to our high-speed InfiniBand network. This enables our research software can run in parallel across many nodes. The engineers that maintain our cluster guarantee that our compilers and MPI can leverage our specific hardware, meaning that most researchers only need to load the right modules and compile their code against these dependencies to maximize their parallel performance on our cluster.

Partitions. Our cluster is highly heterogeneous. In order to make it usable for parallel computation, we divide the hardware into uniform blocks called "partitions" by our scheduler, SLURM. In the schedule below, we will propose a plan in which we upgrade the largest partitions first, while retaining some legacy partitions for testing and validation during a transition period before the remainder are upgraded to the new operating system.

Schedule

The upgrade will happen in stages, meaning that some partitions will not be upgraded during the early January downtime. We will upgrade each partition according to the following schedule:

This guide explains the action you will need to take as well as a transition period that provides a contingency plan for those who cannot migrate their codes in time. If you know that you will want to run your code on both new and old hardware during the transition period, you should open a ticket to give us some advance notice so we know which hardware your code depends on, and how we should upgrade individual partitions during this period.

Motivation

The high-performance computing (HPC) cluster named Sol houses faculty condominium (condo) hardware alongside our NSF-funded expansion named Hawk, which provides broad access to computing cycles and storage for Lehigh faculty. Our current operating system (OS), named CentOS, is now end-of-life. We need to upgrade the operating system to ensure that the system is secure and performant.

Why should I care?

If you use Sol for computation, the OS upgrade will probably break your code. Since we cannot guarantee that all of our thousands of software packages will function with a new operating system, our team will build a new set of software modules that are compatible with the new operating system.

If you have custom code, you will almost certainly need to recompile it in order to use it after our downtime beginning on January 6, 2025. Jump below to the take action section to learn more about how to migrate your code.

Alma, our new OS

Typically, we try to draw out our upgrade cycle to minimize the disruption to our users’ software and workflows. In this case, CentOS was purchased and sponsored by a large corporation that uses the same software as the basis for a paid Linux distribution. This company made a surprise announcement to terminate the lifecycle early, causing a major disruption for those who deployed it before the surprise in late 2020. The resulting fallout provides an object lesson in the challenges in enforcing open-source licenses.

We have selected Alma Linux 9 as the best operating system for the next phase of Sol’s service life. Alongside Rocky, Alma is a spiritual successor to CentOS and provides many of the same benefits. We take comfort in knowing that CERN and Fermilab will be using this distribution, and hope that it has a longer life than its predecessor.

Most users should not notice our operating system, since most of research codes are built on top of it.

Take action

Nearly all users will need to take some action to continue using Sol by January 6, 2025.

We can identify three broad categories of users, each with their own instructions for preserving their workflows. These categories are not mutually exclusive. You may have software in all three categories.

1. Users with custom code
2. Users with virtual environments
3. Users who only use modules

Each linked category above has instructions below. Users who require additional assistance, should submit an HPC ticket.

A. Users with custom code

Most researchers who use our HPC cluster will compile their own code on top of our existing modules system called Lmod. In these cases, you might compile your code against an existing compiler and possibly also a message-passing interface (MPI) using the build system of your choice. The most common example of this might look like this:

module load gcc/9.3.0 openmpi/4.1.6
./configure
make
make install

This method is the bog-standard way to build your code, but you can easily use other build systems to build your code. The most common runner-up is CMake, which is also available in our modules system.

Users with custom codes should plan to recompile their code after the upgrade, in which case there are two changes to the software environment:

1. The newer operating system will provide a new application binary interface (ABI) as well as supporting libraries. This might cause compatibility problems if you have inflexible or unforeseen dependencies on system-level libraries. We expect this to be rare.
2. Our new software tree may have newer compilers and MPI implementations.

In the ideal case, your code will be robust to both sets of changes. In particular, it is rare for MPI and compilers to create forwards-compatibility problems. It’s possible that additional middleware modules might have breaking changes with your code, in which case you will need to update the code, request the older version, or use a container system.

Our Research Computing group can help in two ways. First, you can request assistance from our team to test your code and review your documentation ahead of time. If you submit an HPC ticket (include "upgrade test" in the title) with explicit instructions for compiling the software, we will compile it against our new software tree and provide you with step-by-step instructions for recompiling it after the upgrade. If we fail, we may give you advice for documenting or improving your code. Second, you can also let us know which modules should be included in the new software tree by also submitting an an HPC ticket (please include "module upgrade request" in the title).

B. Users with Virtual Environments

A large subset of our community uses virtual environments of various kinds. These environments give you explicit control over a large library of dependencies. They represent the most user-friendly way for you to install software packages on your own. Most virtual environments are built against a single imperative or scripting language and may include libraries that are linked to specific software modules. The most common example is the LTS-recommended use of a Python virtual environment, however users may also compile custom R packages in their home directory.

In these cases, your virtual environment will contain libraries that will depend on external software in two ways. First, the environment might point to an outdated or OS-provided version of a supporting library. After the OS upgrade, these pointers might break, and this might prevent you from using the environment. Second, many virtual environments might be built against extremely old versions of the supporting language. For example, an environment might use Python 3.6, which is now eight years old. Since Python is nearly forwards-compatible, it should be trivial to rebuild your virtual environment against a newer version of Python provided by our software tree.

The best way to upgrade your environment is to export a copy of your dependencies and then later rebuild the environment with a similar command, after loading the right modules. You can do this in a Python virtual environment with this command:

# export the old environment after loading it
python -m pip freeze > reqs.txt
# load modules and then install them in a new environment
module load python/3.11
python -m venv ./venv
source ./venv/bin/activate
python -m pip install -f reqs.txt

Inside an Anaconda environment, use this command:

# export the old environment after loading it
conda env export | grep -v "^prefix: " > reqs.yaml
# load modules and then install them in a new environment
# consider pinning conda environments against a specific Python version
module load anaconda3
conda env update -f reqs.yaml

Similar procedures are common across all virtual environments. Users who require assistance should submit an an HPC ticket.

C. Users who only use the modules

Most users do not need to compile code, and instead rely entirely on the module system. You might have BASH or SLURM scripts that rely on these modules. It’s possible to build extremely advanced workflows using the modules system. If you want to make sure your workflows will still work after the upgrade, please submit an an HPC ticket with a complete list of the modules you’re using.

For example, if you are currently module load commands in your SLURM scripts, you can send these to us so we can make sure they are included in our updated software modules. Your request could simply provide a copy of a module list output from your calculations. Our team will try to make sure all of these codes are compiled for the new system.

Our schedule lists a tentative deadline for submitting module requests of December 20, 2024, however users will be able to continue to use the legacy software tree on a small subset of our cluster during the transition period.

Popular software: We are currently planning on upgrading the following, highly-popular modules. If the software you need is on this list, you don’t need to submit a request. We have culled this list from the most popular hits on our module load command:

abaqus, anaconda3, ansys, biopython, cmake, conda, cuda, fftw, gaussian, gcc, gnuplot, gromacs, gsl, hdf5, intel-mkl, intel‑oneapi‑mkl, intel‑tbb, lammps, matlab, matplotlib, mpich, mvapich2, netcdf, netcdf‑c, numpy, openblas, openjdk, openmpi, pandas, perl, pip, python, R, scikit‑learn, scipy, seaborn, vasp

Transition Period

The schedule above outlines the key dates for submitting requests for existing software. We will upgrade the largest partitions, namely hawkcpu and rapids, during the downtime. After that, we will gradually move and often consolidate the remaining partitions.

If you know ahead of time that you will not be ready to use the new modules, perhaps because your research software has compatibility problems with recent-vintage compilers and MPI, or you need more time to update your code, it is critical for you to open a ticket with us to keep us informed of your plans. If many users need access to the older OS and software between January 8 and the end of our upgrade on January 31, we will want to use a slow, gradual approach to our upgrades. If we don’t know that you need the older software, we might upgrade too fast.

During the transition period, the older OS on partitions that have not yet been upgraded will not be compatible with SLURM’s interface to InfiniBand, therefore multi-node jobs will not work on the older operating system during the transition. If you have highly parallel calculations, you should tell us about your workflow so you can use InfiniBand immediately following the upgrade.

During the transition period, we may consolidate some of the partitions to improve the usability of the cluster. Our objective is to minimize the gaps between the instruction set for a particular piece of hardware, and the software modules that support it.

Most partitions will "round down" to a very similar instruction set. This strategy allows us to maintain three separate module trees for Ice Lake, Cascade Lake (the bulk of the cluster) and Haswell (the aging nodes) with minimal loss of performance. The module system should be transparent to users, since each software tree will contain many of the same software modules.

To summarize, as we explained in the schedule, we will first upgrade the OS on the hawkcpu and rapids nodes, and then upgrade the remaining partitions during the transition period. Remember to let us know if you will need extra time to use any of the partitions listed above during the transition period.

Containers as a backstop

In the rare case where you are unable to port, maintain, or update your code to the new software tree, we will provide a method for using Apptainer (a.k.a. Singularity) to access the older software in an environment that simulates our previous Centos 8 Stream operating system. This will not work for multi-node, highly-parallel calculations, and performance may be lower than your original workflow. In the section above, we have asked users to tell us if you need extra time on a specific partition. You should similarly open a ticket to tell us if you need instructions for containers to use un-modified code after the upgrade.

Managing research software

This operating system upgrade is a useful prompt for us to audit our research software habits. In comparison to industry, research codes must be more novel, built by smaller teams in less time, and with fewer incremental upgrades. In particular, building reproducible software can be a great challenge that is distinct from many other enterprises.

Our philosophy for deploying software on Sol is simple. We strive to upgrade software which is mostly forwards compatible in order to guarantee performance and security. Most of the software we will recompile for the new operating system will be middleware which should be agnostic to your research codes, for example, upgrading our MPI and compilers should ideally not change the results of your calculations. Similarly, supporting math libraries such as Numpy, FFTW3, OpenBLAS, and Intel’s MKL might experience minor changes to the API, increased features and performance, etc, without altering the scientific correctness of this software.

If upgrading the middleware affects the correctness of your software, then this can be a useful prompt to identify sensitivity or find bugs. In the ideal case, our supporting software and middleware has no systematic effect on your results or conclusions. If you find that they do, or you want some help understanding the best practices for verification and validation, please schedule a consultation with us so we can learn more about your workflow.

Otherwise, we hope to minimize any disruption to your research by making it easy to make small changes to stay current with the latest research software libraries, either by submitting your workflows to us in advance, or performing some comparisons during the transition period. After the upgrade, every user should be able to understand and document their software dependencies so their research codes can lead a long, productive life.

Communications History

On November 25, 2024, our team sent the following message to the HPC mailing list.

Hi All, This message provides notice that the Sol cluster (and Hawk expansion) will have a scheduled downtime on January 6-8, 2025. During the downtime, our team will upgrade the scheduler (SLURM), the operating system (OS), and make a large set of new software available. All users should expect to modify or update their workflows in order to continue computing after the OS upgrade, since these upgrades will almost certainly disrupt your existing compiled codes, our plentiful but aging sofware modules, and your virtual environments. To explain all of the important details and actions you should take, we have provided a guide on our knowledge base. We encourage users to get in touch with us, by opening a help ticket, in order to (1) request software modules they will need after the upgrade, (2) share instructions for testing existing codes on the new software, and (3) answer any questions about the upgrade. We will repeat this announcement in the coming weeks to remind you. We ask you to provide all requests for software or advance testing before December 20, 2024. Requests submiteed after that will be addressed as time allows. We expect this change to improve the security of our system while also giving us an opportunity to streamline the software and partition system in order to make it more usable to our research community. Thank you in advance for helping us prepare for a seamless transition for Sol. –Ryan