Using the Nix Package Manager on an HPC Cluster

HPC clusters are great. They provide lots of CPUs and vast pools of memory that make computationally intensive analyses possible. Without them, I wouldn’t have been able to do a bunch of the work I’ve done over the past few years. But they can also get in the way. Unless you are running an analysis common in fields like bioinformatics, these systems can make using the software you need difficult. They often run an outdated Linux distribution providing old versions of software packages, libraries, and compilers. Reproducibility can also be an issue, because admins may remove or update the software you originally used, and subsequent runs with different version may produce subtly different results.

One answer is containers. The promise of containerization is that, if you stuff everything into standardized units, you will be able to run your code now, always, and everywhere. However, containers only “ensure reproducibility for computers … not for humans.” Code isn’t only or even primarily meant to communicate with computers, but with humans as well. Containers only provide a partial solution to a multifaceted problem.

Enter Nix. Nix is a powerful way to install exactly the software you need for a specific workflow. It also allows you to make those workflows reproducible—and humanly comprehensible—by specifying the exact versions of each program you use. On top of that, because of the vast amount of software that is packaged for it, it makes experimenting with new packages extremely easy.

Most HPC clusters do not provide Nix. Traditionally, Nix required administrative privileges to set up, because it expects its store to exist at the root of the filesystem hierarchy (/nix/store). That changed a few releases ago, though, putting Nix within reach for many HPC users who previously had to rely on other solutions.

Here’s how to get the Nix package manager to run on Leiden University’s ALICE HPC cluster. ALICE currently runs CentOS 7.9, the distribution’s penultimate release from two years ago, and some of the below is specific to that environment. A similar approach will likely work on different HPC systems, but your mileage may vary.

Getting Set Up

First, login to your shell account on the HPC cluster and download the statically linked version of the Nix program.

$ cd ~
$ mkdir -p .local/bin
$ echo "export PATH=\$PATH:\$HOME/.local/bin" >> .profile
$ curl -L https://hydra.nixos.org/job/nix/master/buildStatic.x86_64-linux/latest/download-by-type/file/binary-dist > .local/bin/.nix-wrapped
$ chmod +x .local/bin/.nix-wrapped

Second, create a wrapper script so you can use Nix in a chroot environment.

$ cat > .local/bin/nix <<'EOF'
#!/bin/sh

export NIX_SSL_CERT_FILE="/etc/ssl/certs/ca-bundle.trust.crt"
NIX_REMOTE="local"
.nix-wrapped --experimental-features "nix-command flakes" "$@"
EOF

$ chmod +x .local/bin/nix

You now have everything that’s needed to get started with Nix.

Using Nix

Now you will be able to use commands like nix shell and nix run. Here’s a standard demonstration of what these commands allow you to do:

$ hello
bash: hello: command not found...
$ nix shell nixpkgs#hello
$ hello
Hello, world!

Here, nix shell creates an environment that provides GNU hello, a program that isn’t provided on a system-wide level.

You can also directly run programs using Nix. Let’s say you’d like to edit files using the micro editor. It’s a simple as this:

$ nix run nixpkgs#micro

For more complex workflows than running hello world or a single program, you can create environments pulling in a variety of different packages. Here you have two possibilities. The first are “flakes.” Flakes are essentially configurations that specify which versions of various interdependent software packages you would like to have available and how they should be built. A lot of people are very excited about the possiblities of these kinds of configurations, so there are a lot of tutorials available. One recent resource that goes into the use of flakes for such purposes is called Zero to Nix.

However, flakes are currently considered “experimental.” Their functionality may change, perhaps even fundamentally, before they become the official way of doing things. If, like me, you mostly work with nix-shell and shell.nix files, that is also an option. You can simply do the following:

$ nix shell nixpkgs#nix
$ nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-22.11.tar.gz

Now the environment defined in shell.nix will be activated, and you have access to all the “legacy” Nix commands.

Both of these approaches can exist side by side, so there is not need to choose one or the other. Either one is a great improvement over the previous status quo.

To Conclude

I was seriously surprised how relatively simple it has become to bring my Nix-based workflow with me when using the HPC cluster. For years I’ve been requesting that the administrators of our HPC facility make it centrally available, but understandably they’ve had other priorities. Note, there are more ambitious attempts to use Nix in HPC—a recent FOSDEM talk went into greater detail on some of them. People in the Guix world—a project closely related to Nix—are doing very impressive HPC-related work as well. For now, I’m pleased with what I’m able to do with the approach outlined above, and I will likely not be bugging the ALICE admins again for a while.

Happy hacking—and don’t forget to occasionally run nix store gc to avoid having your Nix store fill up your entire disk quota on the HPC cluster!