# Using nabu at ESRF

This page contains specific instructions on how to use nabu at ESRF. It focuses on the usage of `nabu` itself, without the graphical user interface `tomwer`, which has a dedicated documentation.

**One command to remember**: `source /scisoft/tomotools_env/utils/activate_nabu.sh latest`


## 1. On the compute cluster 

For large scale reconstructions (multiple volumes), nabu is run on the ESRF compute cluster. Reconstruction can be done either through an interactive session, or through submission of a shell script.

The first thing to do is to connect to a front-end machine, which in turn is able to request computational resources:
```bash
ssh -XC cluster-access
```

### 1.1 - Requesting resources in SLURM

You can open an interactive session on a GPU compute node. Currently, the preferred partition is `p9gpu` (power9 machines with Tesla V100 GPUs).
```bash
salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G  srun --pty bash
```

```{warning}
The important bit is `--gres=gpu:1` - you have to explicitly request a GPU, even if the node is in the GPU partition. 
```


This command will only request one CPU core. If you wish to do some CPU processing (eg. involving numpy), then the `-c` parameter has to be provided. To ensure that all the cores are on the same CPU socket, a working command is 

```bash
salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G --cores-per-socket 16 -c 64 srun --pty bash
```

```{caution}
The maximum duration of jobs in the `p9gpu` is one hour. For longer jobs, the `p9gpu-long` partition has to be used.
```



### 1.2 - Requesting resources in OAR

The OAR cluster is made of older machines (Kepler generation GPUs). This cluster is bound to be removed in the future. Nabu is less regularly updated on this cluster.

Request one hour of interactive session on the `gpu` queue:
```bash
oarsub -l cpu=1,walltime=01:00:00 -I -q gpu
```

### 1.3 - Pick and activate a nabu version

On the compute clusters (and *most* ESRF machines), the command to use is

```bash
source /scisoft/tomotools_env/utils/activate_nabu.sh VERSION
```
where `VERSION` is replaced with a version of nabu. For example:
```bash
source /scisoft/tomotools_env/utils/activate_nabu.sh 2021.2.0
```

Please see section 2 for instructions specific to machines that cannot access the `/scisoft` disk.


### 1.4 - Run the reconstruction

Once you have a compute node and activated a version of nabu, you can run the [usual command](nabu_cli) `nabu config_file.conf`



## 2. On a local machine (LBS/dedicated machines)

Some machines cannot mount the `/scisoft` disk.

The following machines have special paths for activating nabu (here the `dev` version, but it can be another version):

  - `lbs191`: `source  /nobackup/lbs191tmp1/nabu/utils/activate_nabu.sh dev`
  - `lbs051`:  `source /nobackup/lbs0511/tomwer/nabu/deployment_scripts/activate_nabu.sh dev`

Please contact us if you wish to install nabu on a specific ESRF machine.


## 3. Troubleshooting

### Nabu fails to start with a `cublasNotInitialized` error

It very likely means that you did not request for a GPU in the SLURM/OAR allocation command. A list of available GPUs can be obtained with `nvidia-smi`. If a `No devices were found` message is shown, then indeed no GPU is available.

## 4. Frequently Asked Questions (FAQ)

### How can I report a problem ?

For any problem with nabu (crash, bad reconstruction results, installation), there are several ways to report it:
  - Ideally, open an [issue on the bug tracker](https://gitlab.esrf.fr/tomotools/nabu/-/issues)
  - Write a mail to [pierre.paleo@esrf.fr](mailto:pierre.paleo@esrf.fr)
  - Use the ESRF instant messaging (rocket.chat)

In any case, you have to provide the **configuration file** giving the error so that the error can be reproduced, and ensure that the dataset to process is accessible by "other" users.

### How can I do a quick check that nabu works correctly ?

To check that nabu is installed, simply type `nabu -V`. It returns the current version of nabu. If it fails, then nabu is not installed/activated.

To check that nabu is actually able to reconstruct, you can [run some tests](tests), the recommended one is `nabu-test reconstruct`. Ensure that all the tests in the line `test_fbp` are passing.

### Is any of these above questions really frequently asked ?

No, but let's hope the answers can help.