Packaging CADET with docker and conda-forge

The distribution of CADET has always been somewhat tricky. We do provide binaries for Windows, but all other platforms need to compile from source. This often makes installing CADET challenging new users.

However, there are more modern approaches, like packaging CADET with conda-forge or distributing it as a docker image. This is also important, since we are planning to use jupterhub in the future which requires docker images (actually there is already a server running that is hosted by the JSC which we might use
Moreover, it would be very convenient if users could simply run

conda install -c conda-forge cadet

For information about building conda packages, see here.

Tagging @j.rao and @j.hassan who offered to support looking into this topic.

There are some questions I’d like to put forth in order to focus our attentions a bit.

  1. Are we going for Docker or Conda (or some combination of both) ?
  2. Should we consider the conda alternatives mentioned in the talk today: Mamba/Boa/Quetz. Link
  3. Since this is to make things easier on the end-user, we’ll have to consider things not only from an easier development/maintenance perspective, but also their ability to install/use either docker or conda (with or without root priveleges).

I believe that setting up a docker container shouldn’t be that hard, and it guarantees that all the necessary dependencies exist in the system. We can just create a script/Dockerfile that does exactly what we need in a controlled environment. But the user might have to wrestle with permissions for the docker daemon, add themselves to a docker group etc. In the end, something like this should work at the end-user level:

docker run -it cadet/cadet

I’m not proficient with Conda or the challenges involved, but I guess the advantage is that we build and run cadet natively. Also, I just checked and conda-forge lists several versions of all the dependencies we need. So it might be as simple as:

conda install -c conda-forge <list-of-deps>
conda install -c conda-forge cadet

I think both. Most people are used to conda and we need Docker for services such as JupyterHub

Should we consider the conda alternatives mentioned in the talk today: Mamba/Boa/Quetz.

Sure, if it is not much additional work, we should definitely have a look!

Since this is to make things easier on the end-user, we’ll have to consider things not only from an easier development/maintenance perspective, but also their ability to install/use either docker or conda (with or without root priveleges).

Conda can (and should) be installed by the user, not system-wide. This way, no root access is required.

I cannot comment on docker but form what I understand about conda-forge is that you basically provide a recipe and that is then built and distributed by them ( And yes, as far as we know, all dependencies are there. Especially sundials with the required version 3.2.1.

Because I’m quite busy at the moment, can you guys investigate and experiment a bit with this?

We should probably stick with conda. That is where all the other python dependencies are and trying to get other users, especially corporate ones, to install something else will be more difficult.

I don’t know much about docker either. It is not something I have ever used.

Both conda-forge and docker should be doable in a limited amount of time. We already have scripts for CI (AppVeyor, Travis, Gitlab Actions in some branch). Those can be adapted for conda-forge recipes, I guess.

Since we can statically link to most deps, our package can have very few actual dependencies (if we want to).

For docker, the same scripts or processes can be used for creating the container. I would also add another container that contains a full development environment (all dependencies, compiler toolchains, git, maybe even a preconfigured VS Code). This way, the bar to develop some C++ code is lowered.

1 Like

I had a talk with Tim Kreuzer from the JSC about jupyterhub. I also asked him about the possiblity to deploy our own images and grant access to external users. Here is his response:

[…] es möglich sein eigene Docker Images zu benutzen. Am einfachsten sind dafür natürlich Images, die auf einem öffentlichen Repository gelagert werden können (z.b. Ein privates Repository haben wir momentan noch nicht, wird aber evtl. auch bald kommen.

Ein Feature der neuen Jupyter-JSC Version (die ebenfalls in den nächsten Wochen bis wenigen Monaten) kommen wird, werden auch “Gruppen / Virtuelle Organisationen” sein. Mit diesen Gruppen wird es möglich sein (fast) alles in Jupyter-JSC mit Restriktionen zu versehen. Also auch den von dir genannten UseCase: Dieses Image dürfen nur Leute benutzen, die in einer bestimmen Gruppe sind.

So that sounds promising. We should definitely get that running as it will enable us to have a running version of CADET readily available for everyone.

Interesting article about integrating conda packaging in travis. Preferably we would do the same with docker. This way, we have only one CI system to maintain.

Just tested a quick docker build

FROM continuumio/miniconda3
COPY / /
RUN conda config --add channels conda-forge
RUN conda install hdf5 superlu suitesparse sundials=3.1.1 cmake
RUN git clone /opt/CADET
RUN apt install -y build-essential
RUN chmod +x
RUN ./

Where just jumps into the CADET dir, sets the right environment variables, and builds it.

If we create and push a conda recipe, it’ll be even simpler. :slight_smile:

1 Like

Nice, @j.rao!
You’ve just volunteered to add Docker support to CADET :wink:

Please add Intel MKL (or this one?), TBB, and upgrade Sundials to 3.2.1.

I suggest to exclusively rely on conda-forge. Continuum has changed its license terms and now charges for commercial usage of its channel (i.e., usage of conda within a company). You can still use Miniconda and other channels (i.e., conda-forge) free of charge.

You should get the official CADET Developer of the month award!

1 Like

I agree about completely sticking to conda-forge.

@j.rao have you looked into conda itself aswell?

Haha no. But I can take a look at it some evening soon.

Will the award come with commenting-without-approval priveleges? :wink:

@s.leweke Done. I assume that using the miniconda docker image shouldn’t be a problem as long as we use the conda-forge channel.

1 Like

A post was split to a new topic: Permissions on the forum

Next call is on Wednesday, 5 pm

This post is to track the work in progress


  • Publication of cadet with conda-forge for all MacOS, Linux, and Windows
  • Docker image e.g. for jupyterlab (JSC)
  • Unify conda/docker with existing CI


Knowledge base

Next meeting, April 7th, 5 p.m. CEST, (Launch Meeting - Zoom)

Update: We’ve created jugit repositories for CADET-Docker and CADET-Conda.

I’ve been working on the Docker side of things, and Jazib on Conda.

So far I’ve tried continuumio/miniconda3 (debian based), ubuntu and archlinux as base images.

  • Trying to build CADET with the compilers package from conda (gcc-9.3.0) fails everywhere due to some issue for testRunner. -DENABLE_TESTS=OFF can be used to sidestep this issue.
  • Trying to build on continuumio/miniconda3 and ubuntu with os-repo gcc gives a link error for when tbb-devel is installed via conda. If we use libtbb-dev from the os-repo everything compiles fine.
  • The build with archlinux base image works with gcc from os-repo and tbb-devel from conda.
  • So currently, I’ve pushed the archlinux build to the jugit repo above. More info in the README.
  • I believe that cadet-cli can be used as an “entrypoint” for the docker container. So instead of opening a shell in the container, we could just type, say docker run <cadet-docker-image> <hdf5-file> and let cadet run in the container while using the file outside. Yet to try this out, though.

A discussion with @j.schmoelder also yielded the idea that we could talk to Martin and Frederick for more help with docker.

From the Conda side of things, building on windows might be a problem because superlu is not available on conda for windows. More on this from @j.hassan

Presently, it seems like our dream of conda install -c conda-forge cadet might not be that easy to achieve on all platforms due to build/link issues and missing dependencies.


@j.rao: CADET only requires either SuperLU or UMFPACK (from SuiteSparse). If both are supplied, the user can choose between them. But UMFPACK is entirely sufficient.

Moreover, the standard TBB should work if it is new enough. We actually do not use preview features.
The libtbb-preview is required for building the native MATLAB MEX binary with multithreading support. Since this will (soon-ish) be replaced by the C API and split into a separate repository, we can move to standard libtbb.

1 Like

@s.leweke Ah. That should simplify things. I think suitesparse is available on conda for windows.

I don’t think the issue is with libtbb-preview per se. I tried disabling it and forcing libtbb in the CMakelists.txt file at some point. The linker then spat out an error about not finding CMake finds the libs when configuring. But the linker can’t seem to, even if I force it with ld /path/to/conda/libs. So, essentially, the linker is having trouble finding tbb-devel libs from conda for whatever reason (on debian/ubuntu).

@s.leweke, would you have time to join the zoom meeting at 1700 tomorrow? Maybe we can make some progress then.

@j.hassan We could temporarily avoid using tbb and/or gcc from conda in the recipe. And then when we figure things out, update it.

Also, another resource for conda packaging: GitHub - ActivisionGameScience/ags_conda_recipes: A tutorial (+ build recipes) describing how to use conda for C, C++, and python package management

@j.rao thanks alot for the information.
I will update my recipe file accordingly.


Summary update post.

1. Followed up with Tim Kreuzer of JSC regarding JSC-Jupyter.

JSC-Jupyter is a hosting service offered by the JSC. They use docker images to host jupyter notebooks. This will fit in line with our goal of providing access to the CADET Tutorials to the general public. More information in this repository.
- Introductory PDF
- Creating a Jupyter Conda Kernel
- Creating a Jupyter General Kernel

The notebooks will be hosted on HDFCloud (Helmholtz Data Federation, unrelated to HDF5). We’ll need access to this machine eventually. Tim told me that we can test creating our own Jupyter Kernel on top of their base image on any machine, so we can try this on JURECA-DC for now.

2. Publishing on conda-forge

@j.schmoelder forked the conda-forge/staged-recipes onto the cadet org. on github: staged-recipes. We’ll just try to build the conda-recipes locally (not with a blank docker container). Once the build is successful, we are to submit a pull request to the main conda-forge repo, where they’ll run automated tests. Once it passes, it will get accepted into the conda-forge repo. More info in this issue

3. Moving conda related development
From jugit/cadet-conda to github/cadet/staged-recipes

Next steps:

- [ ] Finish conda build.
- [ ] Push conda recipe to conda-forge
- [ ] Create a custom Jupyter Kernel or image for JSC-Jupyter.
1 Like