Adventures with WSL2/Spack/Apptainer
This was meant as a deliberate tinker with these tools, to see what worked, and how unpleasant the process would be. I’m not trying to claim this was the best way of acheiving what I was needing to do, but was meant to see whether we could get anywhere with building a piece of software under WSL2, and then carrying that over onto an HPC.
The idea was to use WSL2 under Windows to get ESPResSo working with waLBerla integration. although I ended up a little off piste.
Since there weren’t believed to be packages that offered this, we’d need to build stuff from source to get there.
WSL2 was used for the Linux environment under Windows, Spack was used to build software, and Apptainer was used to build a container at the end, that could be taken and used elsewhere, for example on an HPC system.
Setup a WSL2 instance
I’m assuming you have WSL2 enabled on your system, so I’m not covering that step. It might be as simple as running
wsl --install in a command prompt, but I promise nothing.
I installed the default Ubuntu image from the Microsoft Store.
Creating a WSL instance using this is straighforward. In a command prompt run:
wsl --install Ubuntu
Up pops a new terminal, which asks you to set a username and password, and then you’re up and running with a working WSL2 instance:
Well, before we install Spack, we’re going to install some dependencies:
sudo apt-get update -y sudo apt-get install -y build-essential gfortran
This is the essential bits that Spack’s likely to need - it includes compilers and GNU Make, and I’ve tacked on a Fortran compiler at the end as I know I’ll be needing that later.
Installing Spack itself is just like on any other Linux system:
# Get it from github git clone https://github.com/spack/spack cd spack # Switch the the latest release version git checkout v0.19.0 # Source the config file . share/spack/setup-env.sh
That all works cleanly:
We can have a look at the package that exists for espressomd in Spack:
$ spack list |grep -i espresso py-espresso py-espressopp quantum-espresso
We have a few options there, and we can look at each one:
spack info py-espresso
This shows us it’s the one we want, but doesn’t mention anything about waLBerla, so it looks like we’ll have to sort this bit out ourselves. You can also look at the full list of dependencies it expects to need using:
spack spec py-espresso
It’s never a bad thing to do an early check like this anyway, as it triggers Spack to install its necessary bootstrap packages, providing functionality needed by Spack itself. But sadly, there’s no mention of waLBerla.
There’s no point tweaking the py-espresso package to use waLBerla yet, given we don’t have waLBerla installed.
There isn’t a package for waLBerla, but it is a standard CMake package, so should be easy for Spack to handle.
spack create is able to interrogate the source release, and use that to create a template build script:
spack create https://i10git.cs.fau.de/walberla/walberla/-/archive/v6.1/walberla-v6.1.tar.bz2
Now this isn’t a perfect package file, as it doesn’t have any dependencies defined, but it’s a very nice starting point, given how little effort we’ve had to put in to make it. Let’s give that a try and see what fails, if anything:
spack install walberla
This goes through and installs all the things it thinks it needs to build a generic CMake package, then fails:
Wrapper.h:68:10: fatal error: mpi.h: No such file or directory
Ah okay, it needs MPI to be installed. Edit the recipe (
spack edit walberla) so we can add a dependency on MPI where it says in the file:
How did I know it was a package called “mpi” that I needed to add here? I actually just did
spack info hdf5 to take a look at another package I knew probably had a similar requirement, and learnt from that.
Retry the install (
spack install walberla) and it fails again, this time with a new error:
987 g++: fatal error: Killed signal terminated program cc1plus
What’s happened here is that we’ve run out of memory, since by default, WSL2 gets no more than 50%/8Gbytes of RAM, and 2Gbytes of swap. To increase this, create a file in your Windows home directory (e.g
C:\Users\exuser) called .wslconfig containing:
[wsl2] # Limits VM memory to use no more than 12 GB, defaults to 50% of ram memory=12GB # Sets the amount of swap storage space to 16GB, default is 25% of available RAM swap=16GB
In my case, this is just enough to let it build successfully, but you may want different figures for your system. It appears to need a total of ~15Gbytes to run all in memory, but that wasn’t an option on the 16Gbyte machine I tested on. Pick figures that work for you.
You need to restart wsl at this point, so in a Windows command prompt:
Start your Ubuntu session back up (I just clicked the Ubuntu entry in the Start Menu), reload spack into your environment, and build waLBerla again:
. spack/share/spack/setup-env.sh spack install walberla
Spack will only rebuild what it needs to, so it should skip straight to building waLBerla, and this time it will succeed.
Why I’m off piste…
This is the point I realise that what I thought I understood about how espressomd integrated with waLBerla was total nonsense, and I’d been digging an interesting shaped hole all this time. The integration isn’t present in the main fork, and when it is included, it pulls in its own copy of waLBerla rather than using an existing build version! Ah well, at least we know we’ve got all the pieces in places for this to work…
Actually installing py-espresso with waLBerla support
We’re going to create a new package py-espresso-walberla, and create a config based on py-espresso, but with the other git repo, and targeting the branch we want to build. This is similar to what we did before with
spack create but here’s we’re giving it a name explicitly, and making sure that it understands that it’s a project that builds via CMake.
spack create -n py-espresso-walberla https://github.com/RudolfWeeber/espresso -t cmake
This config is wrong, but if I copy the text out of
spack edit py-espresso and edit it to fix the class name, git repo, and branch, we end up with:
# Copyright 2013-2022 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) from spack.package import * from spack.pkg.builtin.boost import Boost class PyEspressoWalberla(CMakePackage): """ESPResSo is a highly versatile software package for performing and analyzing scientific Molecular Dynamics many-particle simulations of coarse-grained atomistic or bead-spring models as they are used in soft matter research in physics, chemistry and molecular biology. It can be used to simulate systems such as polymers, liquid crystals, colloids, polyelectrolytes, ferrofluids and biological systems, for example DNA and lipid membranes. It also has a DPD and lattice Boltzmann solver for hydrodynamic interactions, and allows several particle couplings to the LB fluid. """ homepage = "https://espressomd.org/" git = "https://github.com/RudolfWeeber/espresso" version("walberla", branch="walberla") depends_on("firstname.lastname@example.org:", type="build") depends_on("mpi") depends_on("boost+serialization+filesystem+system+python+mpi") # TODO: replace this with an explicit list of components of Boost, # for instance depends_on('boost +filesystem') # See https://github.com/spack/spack/pull/22303 for reference depends_on(Boost.with_default_variants) extends("python") depends_on("email@example.com:", type="build") depends_on("py-numpy", type=("build", "run")) depends_on("fftw") depends_on("hdf5+hl+mpi")
That includes everything but turning on the waLBerla support, so we add in:
variant("walberla", default=True, description="Include waLBerla support") def cmake_args(self): spec = self.spec args = [ self.define_from_variant("ESPRESSO_BUILD_WITH_WALBERLA", "walberla") ] return args
Have we arrived at working build?
spack install py-espresso-walberla rather does make it look like we have.
Now was that really the right way to add this new version of py-espresso? I’m not going to lose sleep over it right now.
Putting this in a container
What if I wanted to export this as a container to run on another system? This is where I think we can plug Apptainer and Spack together.
A basic Spack container recipe
Spack has support for making Apptainer compatible recipes for exporting Spack environments into containers, but it’s not quite up to the challenge of working with this modified Spack (that exists on this machine alone).
So let’s do the nearly-but-not quite, then tweak it. If we write a spack.yaml file:
spack: specs: - py-espresso-walberla container: format: singularity images: os: "ubuntu:22.04" spack: develop
Here we create a file for Spack asking for our software (py-espresso-walberla) to be installed, making a recipe suitable to be built with Singularity (which is compatible with Apptainer), and to use ubuntu:22.04 as the base distribution for the build (since we might as well try matching the version we’ve already tested with in WSL2).
We can then ask spack to generate a recipe to be used by Apptainer to make this container:
spack containerize > py-espresso-walberla.def
Here it would probably help you to understand what the Spack created container recipe tries to do.
The idea here is that Spack generates a recipe to build you a container with the software you’ve asked for inside. But to do that, it pulls down a Docker container from a container registry (e.g. Docker Hub), which comes preconfigured with a working Spack. This causes us some bother here, because we don’t want the stock version, we want to use a modified version that contains our changes. In theory there’s a way of telling it to pull a custom version from a git repository, but I had a look at this and it appears to not work properly when building a Singularity container, and only works if you’re building a Docker container, which I don’t want to do. The fix we apply here is to use the default Docker container to build our new container, but before using it, we delete the Spack install they’ve put on, and copy in the one we’ve just tested. It’s a bit messy, but it’ll do.
Additional tweaks to py-espresso-walberla.def:
%files spack /opt/newspack # This is the top of the %post section, not an additional one %post rm -rf /opt/spack ~/.spack mv /opt/newspack /opt/spack
We’re all setup at this point to try to make a container image, but don’t have the necessary software to do so. So next we need to install Apptainer into our WSL2 instance, along with the required dependencies:
sudo apt-get install -y uidmap squashfuse fuse2fs fuse-overlayfs gfortran spack install firstname.lastname@example.org~suid
Then use it to build the container:
spack load apptainer apptainer build -B /run py-espresso-walberla.sif py-espresso-walberla.def
There’s a couple of gotchas in there that I seem to have magically sidestepped with awesome foresight. Ahem. Make sure you don’t use the default OS image, else the compiler used is too old, and we’ve not done anything to load a newer compiler. Also, if you get a cryptic permission denied error, that’s what the
-B /run avoids and you might end up spending far too long scratching you head why it’s complaining. Apptainer is assuming that
/dev/shm is a writable ramdisk, yet in the host environment we’re using,
/dev/shm is actually a symbolic link to somewhere else inside
/run, so without this extra bind mount, which makes
/run outside the container visible as
/run within the container, Spack within the build container is unable to write to the ramdisk as it’s expecting to be able to.
We finish a very long and loopy process with a SIF file containing our custom build of py-espresso with waLBerla integration, along with all its dependencies, that we can now take onto other systems. But we did at least get there, and have poked at the edges of what WSL2, Spack, and Apptainer can do, which might just come in useful with future projects.