Writing NXdata¶
This tutorial explains how to write a NXdata group into a HDF5 file.
A basic knowledge of the HDF5 file format, including understanding the concepts of group, dataset and attribute, is a prerequisite for this tutorial. You should also be able to read a python script using the h5py library to write HDF5 data. You can find some information on these topics at the beginning of the Getting started with silx.io tutorial.
Definitions¶
NeXus Data Format¶
NeXus is a common data format for neutron, x-ray, and muon science. It is being developed as an international standard by scientists and programmers representing major scientific facilities in order to facilitate greater cooperation in the analysis and visualization of neutron, x-ray, and muon data.
It uses the HDF5 format, adding additional rules and structure to help people and software understand how to read a data file.
The name of a group in a NeXus data file can be any string of characters, but it must have a NX_class attribute defining a *class type*.
Examples of such classes are:
- NXroot: root group of the file (may be implicit, if the NX_class attribute is omitted)
- NXentry: describes a measurement; it is mandatory that there is at least one group of this type in the NeXus file
- NXsample: contains information pertaining to the sample, such as its chemical composition, mass, and environment variables (temperature, pressure, magnetic field, etc.)
- NXinstrument: encapsulates all the instrumental information that might be relevant to a measurement
- NXdata: describes the plottable data and related dimension scales
You can find all the specifications about the NeXus format on the nexusformat.org website. The rest of this tutorial will focus exclusively on NXdata.
NXdata groups¶
NXdata describes the plottable data and related dimension scales.
It is mandatory that there is at least one NXdata group in each NXentry group. Note that the variable and data can be defined with different names. The signal and axes attributes of the group define which items are plottable data and which are dimension scales, respectively.
In the case of a curve, for instance, you would have a 1D signal dataset (y values) and optionally another 1D signal of identical size as axis (x values). In the case of an image, you would have a 2D dataset as signal and optionally two 1D datasets to scale the X and Y axes.
A NXdata group should define all the information needed to provide a sensible plot, including axis labels and a plot title. It can also include additional metadata such as standard deviations of data values, or errors an axes.
Note
The NXdata specification evolved slightly over the course of time. The complete documentation for the *NXdata* class mentions older rules that you will probably have to take into account if you intend to write a program that reads NeXus files.
If you only need to write such files and only need to read back files you have yourself written, you should adhere to the most recent rules. We will only mention these most recent specifications in this tutorial.
Main elements in a NXdata group¶
Signal¶
The @signal attribute of the NXdata group provides the name of a dataset containing the plottable data. The name of this dataset can be freely chosen by the writer.
This signal dataset may have a @long_name attribute, that can be used as an axis label (e.g. for the Y axis of a curve) or a plot title (e.g. for an image).
Axes¶
The @axes attributes of the NXdata group provides a list of names of datasets to be used as dimension scales. The number of axes in this list should match the number of dimensions of the signal data, in the general case. But in some specific cases, such as scatter plots or stack of images or curves, the number of axes may differ from the number of signal dimensions.
An axis should be a 1D dataset, whose length matches the size of the corresponding signal dimension.
Silx supports also an axis being a dataset with 2 values \((a, b)\). In such a case, it is interpreted as an affine scaling of the indices (\(i \mapsto a + i * b\)).
An axis dataset may have a @long_name attribute, that can be used as an axis label.
An axis dataset may also define a @first_good and @last_good attribute. These can be used to define a range of indices to be considered valid values in the axis.
The name of the dataset can be freely chosen by the writer.
An axis may be omitted for one or more dimensions of the signal. In this case, a “.” should be written in place of the dataset name in the list of axes names.
Signal errors¶
A dataset named errors can be present in a NXdata group. It provides the standard deviation of data values. This dataset must have the same shape as the signal dataset.
Axes errors¶
An axis may have associated errors (uncertainties). These axis errors must be provided in a dataset whose name is the axis name with _errors appended to it.
For instance, an axis whose dataset name is pressure may provide errors in an another dataset whose name is pressure_errors.
This dataset must have the same size as the corresponding axis.
Interpretation¶
Silx supports an attribute @interpretation attached to the signal dataset. The supported values for this attribute are scalar, spectrum or image.
This attribute must be provided when the number of axes is lower than the number of signal dimensions. For instance, a 3D signal with @interpretation=”image” is interpreted as a stack of images. The axes always apply to the last dimensions of the signal, so in this example of a 3D stack of images, the first dimension is not scaled and is interpreted as a frame number.
Note
This additional attribute is not mentionned in the official NXdata specification.
Writing NXdata with h5py¶
The following examples explain how to write NXdata directly using the h5py library.
Note
All following examples should be preceded by:
import h5py
import numpy
import sys
# this is needed for writing arrays of utf-8 strings with h5py
if sys.version_info < (3,):
text_dtype = h5py.special_dtype(vlen=unicode)
else:
text_dtype = h5py.special_dtype(vlen=str)
filename = "./myfile.h5"
h5f = h5py.File(filename, "w")
entry = h5f.create_group("my_entry")
entry.attrs["NX_class"] = "NXentry"
A simple curve¶
The simplest NXdata example would be a 1D signal to be plotted as a curve.
nxdata = entry.create_group("my_curve")
nxdata.attrs["NX_class"] = "NXdata"
nxdata.attrs["signal"] = numpy.array("y", dtype=text_dtype)
ds = nxdata.create_dataset("y",
data=numpy.array([0.1, 0.2, 0.15, 0.44]))
ds.attrs["long_name"] = numpy.array("ordinate", dtype=text_dtype)
To add an axis:
nxdata.attrs["axes"] = numpy.array(["x"],
dtype=text_dtype)
ds = nxdata.create_dataset("x",
data=numpy.array([101.1, 101.2, 101.3, 101.4]))
ds.attrs["long_name"] = numpy.array("abscissa", dtype=text_dtype)
A scatter plot¶
A scatter plot is the only case for which we can have more axes than there are signal dimensions. The signal is 1D, and there can be any number of axes with the same number of values as the signal.
But the most common case is a 2D scatter plot, with a signal and two axes.
nxdata = entry.create_group("my_scatter")
nxdata.attrs["NX_class"] = "NXdata"
nxdata.attrs["signal"] = numpy.array("values",
dtype=text_dtype)
nxdata.attrs["axes"] = numpy.array(["x", "y"],
dtype=text_dtype)
nxdata.create_dataset("values",
data=numpy.array([0.1, 0.2, 0.15, 0.44]))
nxdata.create_dataset("x",
data=numpy.array([101.1, 101.2, 101.3, 101.4]))
nxdata.create_dataset("y",
data=numpy.array([2, 4, 6, 8]))
A stack of images¶
The following examples illustrates how to use the @interpretation attribute to define only two axes for a 3D signal. The first dimension of the signal is considered a frame index and is not scaled.
nxdata = entry.create_group("images")
nxdata.attrs["NX_class"] = "NXdata"
nxdata.attrs["signal"] = numpy.array("frames",
dtype=text_dtype)
nxdata.attrs["axes"] = numpy.array(["y", "x"],
dtype=text_dtype)
# 2 frames of size 3 rows x 4 columns
signal = nxdata.create_dataset(
"frames",
data=numpy.array([[[1., 1.1, 1.2, 1.3],
[1.4, 1.5, 1.6, 1.7],
[1.8, 1.9, 2.0, 2.1]],
[[8., 8.1, 8.2, 8.3],
[8.4, 8.5, 8.6, 8.7],
[8.8, 8.9, 9.0, 9.1]]]))
signal.attrs["interpretation"] = "image"
nxdata.create_dataset("x",
data=numpy.array([0.1, 0.2, 0.3, 0.4]))
nxdata.create_dataset("y",
data=numpy.array([2, 4, 6]))
Writing NXdata with silx¶
silx provides a convenience function to write NXdata groups:
silx.io.nxdata.save_NXdata()
The following examples show how to reproduce the previous examples using this function.
A simple curve¶
To get exactly the same output as previously, you can specify all attributes like this:
import numpy
from silx.io.nxdata import save_NXdata
save_NXdata(filename="./myfile.h5",
signal=numpy.array([0.1, 0.2, 0.15, 0.44]),
signal_name="y",
signal_long_name="ordinate",
axes=[numpy.array([101.1, 101.2, 101.3, 101.4])],
axes_names=["x"],
axes_long_names=["abscissa"],
nxentry_name="my_entry",
nxdata_name="my_curve")
Most of these parameters are optional, only filename and signal are mandatory parameters. Omitted parameters have default values.
If you do not care about the names of the entry, NXdata and of all the datasets, you can simply write:
import numpy
from silx.io.nxdata import save_NXdata
save_NXdata(filename="./myfile.h5",
signal=numpy.array([0.1, 0.2, 0.15, 0.44]),
axes=[numpy.array([101.1, 101.2, 101.3, 101.4])])
A scatter plot¶
import numpy
from silx.io.nxdata import save_NXdata
save_NXdata(filename="./myfile.h5",
signal=numpy.array([0.1, 0.2, 0.15, 0.44]),
signal_name="values",
axes=[numpy.array([2, 4, 6, 8]),
numpy.array([101.1, 101.2, 101.3, 101.4])],
axes_names=["x", "y"],
nxentry_name="my_entry",
nxdata_name="my_scatter")
A stack of images¶
import numpy
from silx.io.nxdata import save_NXdata
save_NXdata(filename="./myfile.h5",
signal=numpy.array([[[1., 1.1, 1.2, 1.3],
[1.4, 1.5, 1.6, 1.7],
[1.8, 1.9, 2.0, 2.1]],
[[8., 8.1, 8.2, 8.3],
[8.4, 8.5, 8.6, 8.7],
[8.8, 8.9, 9.0, 9.1]]]),
signal_name="frames",
interpretation="image",
axes=[numpy.array([2, 4, 6]),
numpy.array([0.1, 0.2, 0.3, 0.4])],
axes_names=["y", "x"],
nxentry_name="my_entry",
nxdata_name="images")
