Using nabu at ESRF

This page contains specific instructions on how to use nabu at ESRF. The activation commands will also provide the tomwer graphical interface.

Note

From Q3 2023, all the tomography software is available via “modules”. The same command (module load tomotools) can be used on all ESRF machines (cluster and beamline machines).

The former activation command source /scisoft/tomotools/activate still works but is deprecated.

1. On the compute cluster

For large scale reconstructions (multiple volumes), nabu is run on the ESRF compute cluster. Reconstruction can be done either through an interactive session, or through submission of a shell script.

The first thing to do is to connect to a front-end machine, which in turn is able to request computational resources:

ssh -XC cluster-access

1.1 - Requesting resources in SLURM

You can open an interactive session on a GPU compute node. Currently, the preferred partition (= group of machines) is named gpu (it contains AMD-x86 machines with Nvidia A40 GPUs).

salloc --partition gpu --gres=gpu:1 --time=01:00:00 --mem=200G  srun --pty bash -l

Warning

The important bit is --gres=gpu:1 - you have to explicitly request a GPU, even if the node is in the GPU partition.

Don’t forget the final -l in srun --pty bash -l.

This command will only request one CPU core. If you wish to do some CPU processing (eg. involving numpy), then the -c parameter has to be provided. To ensure that all the cores are on the same CPU socket, a working command is

salloc --partition gpu --gres=gpu:1 --time=01:00:00 --mem=200G -c 32 srun --pty bash

Caution

The maximum duration of jobs in the gpu partition is one hour. For longer jobs, the gpu-long partition has to be used.

You can also pick the p9gpu partition using

salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G  srun --pty bash -l

These nodes have IBM-Power9 CPUs with Nvidia Tesla V100 GPUs. They are slower but tend to be much more available.

1.2 - Pick and activate a nabu version

The command to use is

module load tomotools

This will load the latest stable release by default.

To pick a specific version, eg. a recent (development) version, you can use

module load tomotools/dev

1.3 - Run the reconstruction

Once you have a compute node and activated a version of nabu, you can run the usual command nabu config_file.conf

2. Troubleshooting

Nabu fails to start with a cublasNotInitialized error

It very likely means that you did not request for a GPU in the SLURM/OAR allocation command. A list of available GPUs can be obtained with nvidia-smi. If a No devices were found message is shown, then indeed no GPU is available.

3. Frequently Asked Questions (FAQ)

How can I report a problem ?

For any problem with nabu (crash, bad reconstruction results, installation), there are several ways to report it:

In any case, you have to provide the configuration file giving the error so that the error can be reproduced, and ensure that the dataset to process is accessible by “other” users.

How can I do a quick check that nabu works correctly ?

To check that nabu is installed, simply type nabu -V. It returns the current version of nabu. If it fails, then nabu is not installed/activated.

To check that nabu is actually able to reconstruct, you can run some tests, the recommended one is nabu-test reconstruction. Ensure that all the tests in the line test_fbp are passing.