# Using nabu at ESRF

This page contains specific instructions on how to use nabu at ESRF.
The activation commands will also provide the tomwer graphical interface.

```{note}
From Q3 2023, all the tomography software is available via "modules". The same command (`module load tomotools`) can be used on all ESRF machines (cluster and beamline machines).

The former activation command `source /scisoft/tomotools/activate` still works but is deprecated.
```

## 1. On the compute cluster 

For large scale reconstructions (multiple volumes), nabu is run on the ESRF compute cluster. Reconstruction can be done either through an interactive session, or through submission of a shell script.

The first thing to do is to connect to a front-end machine, which in turn is able to request computational resources:
```bash
ssh -XC cluster-access
```

### 1.1 - Requesting resources in SLURM

You can open an interactive session on a GPU compute node. Currently, the preferred partition (= group of machines) is named `gpu` (it contains AMD-x86 machines with Nvidia A40 GPUs).
```bash
salloc --partition gpu --gres=gpu:1 --time=01:00:00 --mem=200G  srun --pty bash -l
```

```{warning}
The important bit is `--gres=gpu:1` - you have to explicitly request a GPU, even if the node is in the GPU partition. 
```

Don't forget the final `-l` in `srun --pty bash -l`.


This command will only request one CPU core. If you wish to do some CPU processing (eg. involving numpy), then the `-c` parameter has to be provided. To ensure that all the cores are on the same CPU socket, a working command is 

```bash
salloc --partition gpu --gres=gpu:1 --time=01:00:00 --mem=200G -c 32 srun --pty bash
```

```{caution}
The maximum duration of jobs in the `gpu` partition is one hour. For longer jobs, the `gpu-long` partition has to be used.
```

You can also pick the `p9gpu` partition using

```bash
salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G  srun --pty bash -l
```

These nodes have IBM-Power9 CPUs with Nvidia Tesla V100 GPUs. They are slower but tend to be much more available.



### 1.2 - Pick and activate a nabu version

The command to use is

```bash
module load tomotools
```

This will load the latest stable release by default.

To pick a specific version, eg. a recent (development) version, you can use
```bash
module load tomotools/dev
```


### 1.3 - Run the reconstruction

Once you have a compute node and activated a version of nabu, you can run the [usual command](nabu_cli) `nabu config_file.conf`



## 2. Troubleshooting

### Nabu fails to start with a `cublasNotInitialized` error

It very likely means that you did not request for a GPU in the SLURM/OAR allocation command. A list of available GPUs can be obtained with `nvidia-smi`. If a `No devices were found` message is shown, then indeed no GPU is available.


## 3. Frequently Asked Questions (FAQ)

### How can I report a problem ?

For any problem with nabu (crash, bad reconstruction results, installation), there are several ways to report it:
  - Ideally, open an [issue on the bug tracker](https://gitlab.esrf.fr/tomotools/nabu/-/issues)
  - Write a mail to [pierre.paleo@esrf.fr](mailto:pierre.paleo@esrf.fr)
  - Use the ESRF instant messaging (rocket.chat)

In any case, you have to provide the **configuration file** giving the error so that the error can be reproduced, and ensure that the dataset to process is accessible by "other" users.

### How can I do a quick check that nabu works correctly ?

To check that nabu is installed, simply type `nabu -V`. It returns the current version of nabu. If it fails, then nabu is not installed/activated.

To check that nabu is actually able to reconstruct, you can [run some tests](tests), the recommended one is `nabu-test reconstruction`. Ensure that all the tests in the line `test_fbp` are passing.