{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Performing the azimutal integration from shell scripts\n", "\n", "There are three main tools for performing azimuthal integration from the command line. \n", "Two of them specicalized for SAXS and WAXS and the last which is common to the GUI but we will see how to use it in command line mode (CLI).\n", "\n", "They all require 3 mandatory parameters: \n", "\n", "* The PONI-file containing the geometry of the experimental setup\n", "* The number of bins used for the histogram\n", "* The list of input files\n", "\n", "The three tools differ only by the default options selected to fit the best the needs of a given user community.\n", "\n", "## Azimuthal integration for powder diffraction\n", "\n", "The *pyFAI-waxs* which stands for wide angle scattering produces integrated patterns in $2\\theta$ units by default.\n", "\n", "*Note:* while this cookbook is performed in a Jupyter notebook, all commands are run in the command line interpreter (Bash), not in Python." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Detector Detector\t Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline\t PixelSize= 5.168e-05, 5.126e-05 m\n", "Wavelength= 4.217150e-11m\n", "SampleDetDist= 1.182208e-01m\tPONI= 5.394843e-02, 5.551600e-02m\trot1=0.006974 rot2= -0.003313 rot3= -0.000000 rad\n", "DirectBeamDist= 118.224mm\tCenter: x=1066.839, y=1036.336 pix\tTilt=0.442 deg tiltPlanRotation= -154.594 deg\n", "Mask: None\tMethod: splitpixel\n", "Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.xy,\t reading: 0.022s\t 1D integration: 1.789s.\n" ] } ], "source": [ "pyFAI-waxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# == pyFAI calibration ==\n", "# Distance Sample to Detector: 0.118220810284 m\n", "# PONI: 5.395e-02, 5.552e-02 m\n", "# Rotations: 0.006974 -0.003313 -0.000000 rad\n", "# \n", "# == Fit2d calibration ==\n", "# Distance Sample-beamCenter: 118.224 mm\n", "# Center: x=1066.839, y=1036.336 pix\n", "# Tilt: 0.442 deg TiltPlanRot: -154.594 deg\n", "# \n", "# Detector Detector\t Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline\t PixelSize= 5.168e-05, 5.126e-05 m\n", "# Detector has a mask: False \n", "# Detector has a dark current: False \n", "# detector has a flat field: False \n", "# \n", "# Wavelength: 4.21714957131e-11 m\n", "# Mask applied: False\n", "# Dark current applied: False\n", "# Flat field applied: False\n", "# Polarization factor: None\n", "# Normalization factor: 1.0\n", "# \n", "# Headers of the input frame:\n", "# {\n", "# \"info\": {},\n", "# \"photometricInterpretation\": 1,\n", "# \"rowsPerStrip\": 2048,\n", "# \"nColumns\": 2048,\n", "# \"compression\": false,\n", "# \"sampleFormat\": 3,\n", "# \"imageDescription\": \"{\\nEDF_DataBlockID=0.Image.Psd\\nEDF_BinarySize=16777216\\nEDF_HeaderSize=1536\\nByteOrder=LowByteFirst\\nDataType=FloatValue\\nDim_1=2048\\nDim_2=2048\\nImage=0\\nHeaderID=EH:000000:000000:000000\\nSize=16777216\\nmethod=max\\nnframes=21\\ncutoff=None\\nmerged_file_00=ref_lab6_0001.edf\\nmerged_file_01=ref_lab6_0002.edf\\nmerged_file_02=ref_lab6_0003.edf\\nmerged_file_03=ref_lab6_0004.edf\\nmerged_file_04=ref_lab6_0005.edf\\nmerged_file_05=ref_lab6_0006.edf\\nmerged_file_06=ref_lab6_0007.edf\\nmerged_file_07=ref_lab6_0008.edf\\nmerged_file_08=ref_lab6_0009.edf\\nmerged_file_09=ref_lab6_0010.edf\\nmerged_file_10=ref_lab6_0011.edf\\nmerged_file_11=ref_lab6_0012.edf\\nmerged_file_12=ref_lab6_0013.edf\\nmerged_file_13=ref_lab6_0014.edf\\nmerged_file_14=ref_lab6_0015.edf\\nmerged_file_15=ref_lab6_0016.edf\\nmerged_file_16=ref_lab6_0017.edf\\nmerged_file_17=ref_lab6_0018.edf\\nmerged_file_18=ref_lab6_0019.edf\\nmerged_file_19=ref_lab6_0020.edf\\nmerged_file_20=ref_lab6_0021.edf\\nfilename=LaB6_29.4keV.tif\\n\",\n", "# \"nRows\": 2048,\n", "# \"colormap\": null,\n", "# \"nBits\": 32,\n", "# \"date\": \"Fri Mar 2 10:02:31 2018\",\n", "# \"software\": \"fabio.tifimage\",\n", "# \"compression_type\": 1,\n", "# \"stripOffsets\": [\n", "# 1143\n", "# ],\n", "# \"stripByteCounts\": [\n", "# 16777216\n", "# ]\n", "# }\n", "# --> LaB6_29.4keV.xy\n", "# 2th_deg I \n", " 1.668855e-02 2.425633e+00\n", " 5.006564e-02 2.804243e+00\n", " 8.344272e-02 2.121904e+00\n", " 1.168198e-01 2.734025e+00\n" ] } ], "source": [ "#Let's have a look at the header of the produces ascii file:\n", "head -n 50 LaB6_29.4keV.xy" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: pyFAI-waxs [options] -p ponifile file1.edf file2.edf ...\n", "\n", "Azimuthal integration for powder diffraction.\n", "\n", "positional arguments:\n", " FILE Image files to integrate\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n", " -v, --version show program's version number and exit\n", " -p PONIFILE PyFAI parameter file (.poni)\n", " -n NPT, --npt NPT Number of points in radial dimension\n", " -w WAVELENGTH, --wavelength WAVELENGTH\n", " wavelength of the X-Ray beam in Angstrom\n", " -e ENERGY, --energy ENERGY\n", " energy of the X-Ray beam in keV\n", " (hc=12.3984197396keV.A)\n", " -u DUMMY, --dummy DUMMY\n", " dummy value for dead pixels\n", " -U DELTA_DUMMY, --delta_dummy DELTA_DUMMY\n", " delta dummy value\n", " -m MASK, --mask MASK name of the file containing the mask image\n", " -d DARK, --dark DARK name of the file containing the dark current\n", " -f FLAT, --flat FLAT name of the file containing the flat field\n", " -P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR\n", " Polarization factor, from -1 (vertical) to +1\n", " (horizontal), default is None for no correction,\n", " synchrotrons are around 0.95\n", " --error-model ERROR_MODEL\n", " Error model to use. Currently on 'poisson' is\n", " implemented\n", " --unit UNIT unit for the radial dimension: can be q_nm^-1, q_A^-1,\n", " 2th_deg, 2th_rad or r_mm\n", " --ext EXT extension of the regrouped filename (.xy)\n", " --method METHOD Integration method\n", " --multi Average out all frame in a file before integrating\n", " extracting variance, otherwise treat every single\n", " frame\n", " --average AVERAGE Method for averaging out: can be 'mean' (default),\n", " 'min', 'max' or 'median\n", " --do-2D Perform 2D integration in addition to 1D\n", "\n", "pyFAI-waxs is the script of pyFAI that allows data reduction (azimuthal\n", "integration) for Wide Angle Scattering to produce X-Ray Powder Diffraction\n", "Pattern with output axis in 2-theta space.\n" ] } ], "source": [ "pyFAI-waxs -h" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One options which may be of some interest is the *--do-2D* which allow to perform caking.\n", "\n", "## Azimuthal integration for small angle data\n", "\n", "Small angle scattering usually prefer integration in a radial unit independant of the wavelength, typically \n", "$q = \\frac{4 \\pi sin(2\\theta/2)}{\\lambda}$\n", "in inverse nanometers ($nm^{-1}$)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Detector Detector\t Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline\t PixelSize= 5.168e-05, 5.126e-05 m\n", "Wavelength= 4.217150e-11m\n", "SampleDetDist= 1.182208e-01m\tPONI= 5.394843e-02, 5.551600e-02m\trot1=0.006974 rot2= -0.003313 rot3= -0.000000 rad\n", "DirectBeamDist= 118.224mm\tCenter: x=1066.839, y=1036.336 pix\tTilt=0.442 deg tiltPlanRotation= -154.594 deg\n", "Mask: None\tMethod: splitpixel\n", "Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.dat,\t reading: 0.022s\t 1D integration: 1.792s.\n" ] } ], "source": [ "pyFAI-saxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# == pyFAI calibration ==\n", "# Distance Sample to Detector: 0.118220810284 m\n", "# PONI: 5.395e-02, 5.552e-02 m\n", "# Rotations: 0.006974 -0.003313 -0.000000 rad\n", "# \n", "# == Fit2d calibration ==\n", "# Distance Sample-beamCenter: 118.224 mm\n", "# Center: x=1066.839, y=1036.336 pix\n", "# Tilt: 0.442 deg TiltPlanRot: -154.594 deg\n", "# \n", "# Detector Detector\t Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline\t PixelSize= 5.168e-05, 5.126e-05 m\n", "# Detector has a mask: False \n", "# Detector has a dark current: False \n", "# detector has a flat field: False \n", "# \n", "# Wavelength: 4.21714957131e-11 m\n", "# Mask applied: False\n", "# Dark current applied: False\n", "# Flat field applied: False\n", "# Polarization factor: None\n", "# Normalization factor: 1.0\n", "# \n", "# Headers of the input frame:\n", "# {\n", "# \"info\": {},\n", "# \"photometricInterpretation\": 1,\n", "# \"rowsPerStrip\": 2048,\n", "# \"nColumns\": 2048,\n", "# \"compression\": false,\n", "# \"sampleFormat\": 3,\n", "# \"imageDescription\": \"{\\nEDF_DataBlockID=0.Image.Psd\\nEDF_BinarySize=16777216\\nEDF_HeaderSize=1536\\nByteOrder=LowByteFirst\\nDataType=FloatValue\\nDim_1=2048\\nDim_2=2048\\nImage=0\\nHeaderID=EH:000000:000000:000000\\nSize=16777216\\nmethod=max\\nnframes=21\\ncutoff=None\\nmerged_file_00=ref_lab6_0001.edf\\nmerged_file_01=ref_lab6_0002.edf\\nmerged_file_02=ref_lab6_0003.edf\\nmerged_file_03=ref_lab6_0004.edf\\nmerged_file_04=ref_lab6_0005.edf\\nmerged_file_05=ref_lab6_0006.edf\\nmerged_file_06=ref_lab6_0007.edf\\nmerged_file_07=ref_lab6_0008.edf\\nmerged_file_08=ref_lab6_0009.edf\\nmerged_file_09=ref_lab6_0010.edf\\nmerged_file_10=ref_lab6_0011.edf\\nmerged_file_11=ref_lab6_0012.edf\\nmerged_file_12=ref_lab6_0013.edf\\nmerged_file_13=ref_lab6_0014.edf\\nmerged_file_14=ref_lab6_0015.edf\\nmerged_file_15=ref_lab6_0016.edf\\nmerged_file_16=ref_lab6_0017.edf\\nmerged_file_17=ref_lab6_0018.edf\\nmerged_file_18=ref_lab6_0019.edf\\nmerged_file_19=ref_lab6_0020.edf\\nmerged_file_20=ref_lab6_0021.edf\\nfilename=LaB6_29.4keV.tif\\n\",\n", "# \"nRows\": 2048,\n", "# \"colormap\": null,\n", "# \"nBits\": 32,\n", "# \"date\": \"Fri Mar 2 10:02:31 2018\",\n", "# \"software\": \"fabio.tifimage\",\n", "# \"compression_type\": 1,\n", "# \"stripOffsets\": [\n", "# 1143\n", "# ],\n", "# \"stripByteCounts\": [\n", "# 16777216\n", "# ]\n", "# }\n", "# --> LaB6_29.4keV.dat\n", "# q_nm^-1 I \n", " 4.278568e-02 2.427221e+00\n", " 1.283570e-01 2.821874e+00\n", " 2.139284e-01 2.101271e+00\n", " 2.994998e-01 2.708885e+00\n" ] } ], "source": [ "head -n 50 LaB6_29.4keV.dat" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: pyFAI-saxs [options] -n 1000 -p ponifile file1.edf file2.edf ...\n", "\n", "Azimuthal integration for SAXS users.\n", "\n", "positional arguments:\n", " FILE Image files to integrate\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n", " -v, --version show program's version number and exit\n", " -p PONIFILE PyFAI parameter file (.poni)\n", " -n NPT, --npt NPT Number of points in radial dimension\n", " -w WAVELENGTH, --wavelength WAVELENGTH\n", " wavelength of the X-Ray beam in Angstrom\n", " -e ENERGY, --energy ENERGY\n", " energy of the X-Ray beam in keV\n", " (hc=12.3984197396keV.A)\n", " -u DUMMY, --dummy DUMMY\n", " dummy value for dead pixels\n", " -U DELTA_DUMMY, --delta_dummy DELTA_DUMMY\n", " delta dummy value\n", " -m MASK, --mask MASK name of the file containing the mask image\n", " -d DARK, --dark DARK name of the file containing the dark current\n", " -f FLAT, --flat FLAT name of the file containing the flat field\n", " -P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR\n", " Polarization factor, from -1 (vertical) to +1\n", " (horizontal), default is None for no correction,\n", " synchrotrons are around 0.95\n", " --error-model ERROR_MODEL\n", " Error model to use. Currently on 'poisson' is\n", " implemented\n", " --unit UNIT unit for the radial dimension: can be q_nm^-1, q_A^-1,\n", " 2th_deg, 2th_rad or r_mm\n", " --ext EXT extension of the regrouped filename (.dat)\n", " --method METHOD Integration method\n", "\n", "pyFAI-saxs is the SAXS script of pyFAI that allows data reduction (azimuthal\n", "integration) for Small Angle Scattering with output axis in q space.\n" ] } ], "source": [ "pyFAI-saxs -h" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Common tool for azimuthal intration\n", "\n", "pyFAI-integrate offers a graphical user interface for integration which exposes most options from pyFAI. \n", "Once all options have been configured in the the windows, one can either launch the processing or save the configuration into a *JSON* file to be able to create script with all the options.\n", "\n", "![pyFAI-integrate GUI](pyFAI-integrate.png)\n", "\n", "To run *pyFAI-integrate* as a script, i.e. without the graphical interface, the *--no-gui* option is required:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:root:Enter, port=54321.\n", "INFO:root:Enter.\n", "start config ...\n", "INFO:pyFAI.worker:Detector Detector\t Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline\t PixelSize= 5.168e-05, 5.126e-05 m\n", "Wavelength= 4.217150e-11m\n", "SampleDetDist= 1.182208e-01m\tPONI= 5.394843e-02, 5.551600e-02m\trot1=0.006974 rot2= -0.003313 rot3= -0.000000 rad\n", "DirectBeamDist= 118.224mm\tCenter: x=1066.839, y=1036.336 pix\tTilt=0.442 deg tiltPlanRotation= -154.594 deg\n", "INFO:pyFAI.azimuthalIntegrator:AI.integrate1d: Resetting integrator because init\n", "INFO:pyFAI.opencl.processing:234.570MB are needed on device which has 12644.188MB\n", "INFO:pyFAI.opencl.processing:Compiling file ['kahan.cl', 'preprocess.cl', 'memset.cl', 'ocl_azim_CSR.cl'] with options -D NBINS=1000 -D NIMAGE=4194304 -D WORKGROUP_SIZE=32\n", "INFO:pyFAI:Processing done in 2.433s ! \n" ] } ], "source": [ "pyFAI-integrate --no-gui -j azimint.json -o LaB6_29.4keV.cli LaB6_29.4keV.tif " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:root:Enter, port=54321.\n", "INFO:root:Enter.\n", "usage: pyFAI-integrate [options] file1.edf file2.edf ...\n", "\n", "PyFAI-integrate is a graphical interface (based on Python/Qt4) to perform\n", "azimuthal integration on a set of files. It exposes most of the important\n", "options available within pyFAI and allows you to select a GPU (or an openCL\n", "platform) to perform the calculation on.\n", "\n", "positional arguments:\n", " FILE Files to be integrated\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n", " -V, --version show program's version number and exit\n", " -v, --verbose switch to verbose/debug mode\n", " -o OUTPUT, --output OUTPUT\n", " Directory or file where to store the output data\n", " -f FORMAT, --format FORMAT\n", " output data format (can be HDF5)\n", " -s SLOW, --slow-motor SLOW\n", " Dimension of the scan on the slow direction (makes\n", " sense only with HDF5)\n", " -r RAPID, --fast-motor RAPID\n", " Dimension of the scan on the fast direction (makes\n", " sense only with HDF5)\n", " --no-gui Process the dataset without showing the user\n", " interface.\n", " -j JSON, --json JSON Configuration file containing the processing to be\n", " done\n", " --monitor-name MONITOR_KEY\n", " Name of the monitor in the header of each input files.\n", " If defined the contribution of each input file is\n", " divided by the monitor. If the header does not contain\n", " or contains a wrong value, the contribution of the\n", " input file is ignored. On EDF files, values from\n", " 'counter_pos' can accessed by using the expected\n", " mnemonic. For example 'counter/bmon'.\n", "\n", "PyFAI-integrate saves all parameters in a .azimint.json (hidden) file. This\n", "JSON file is an ascii file which can be edited and used to configure online\n", "data analysis using the LImA plugin of pyFAI. Nota: there is bug in debian6\n", "making the GUI crash (to be fixed inside pyqt) http://bugs.debian.org/cgi-\n", "bin/bugreport.cgi?bug=697348\n" ] } ], "source": [ "pyFAI-integrate --help" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# == pyFAI calibration ==\n", "# Distance Sample to Detector: 0.118220810284 m\n", "# PONI: 5.395e-02, 5.552e-02 m\n", "# Rotations: 0.006974 -0.003313 -0.000000 rad\n", "# \n", "# == Fit2d calibration ==\n", "# Distance Sample-beamCenter: 118.224 mm\n", "# Center: x=1066.839, y=1036.336 pix\n", "# Tilt: 0.442 deg TiltPlanRot: -154.594 deg\n", "# \n", "# Detector Detector\t Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline\t PixelSize= 5.168e-05, 5.126e-05 m\n", "# Detector has a mask: False \n", "# Detector has a dark current: False \n", "# detector has a flat field: False \n", "# \n", "# Wavelength: 4.21714957131e-11 m\n", "# Mask applied: False\n", "# Dark current applied: False\n", "# Flat field applied: False\n", "# Polarization factor: None\n", "# Normalization factor: 1.0\n", "# --> /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/LaB6_29.4keV.cli\n", "# r_mm I \n", " 3.913197e-02 2.562033e+00\n", " 1.173959e-01 2.491645e+00\n", " 1.956599e-01 2.354185e+00\n", " 2.739238e-01 2.858854e+00\n", " 3.521877e-01 2.796831e+00\n", " 4.304517e-01 2.496695e+00\n", " 5.087156e-01 2.796695e+00\n", " 5.869795e-01 2.742193e+00\n", " 6.652435e-01 2.834705e+00\n", " 7.435074e-01 2.855904e+00\n", " 8.217714e-01 2.906582e+00\n", " 9.000353e-01 3.021892e+00\n", " 9.782992e-01 3.171136e+00\n", " 1.056563e+00 3.010916e+00\n", " 1.134827e+00 3.027250e+00\n", " 1.213091e+00 3.020955e+00\n", " 1.291355e+00 2.935973e+00\n", " 1.369619e+00 3.207142e+00\n", " 1.447883e+00 2.983296e+00\n", " 1.526147e+00 3.007866e+00\n", " 1.604411e+00 3.052688e+00\n", " 1.682675e+00 2.898143e+00\n", " 1.760939e+00 3.027955e+00\n", " 1.839203e+00 3.147613e+00\n", " 1.917466e+00 3.243755e+00\n", " 1.995730e+00 3.213144e+00\n", " 2.073994e+00 2.948822e+00\n" ] } ], "source": [ "head -n 50 LaB6_29.4keV.cli" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "This cookbook exposed the way to perfrom azimuthal integration in scripts with pyFAI using three tools: pyFAI-waxs, pyFAI-saxs and pyFAI-integrate. \n", "Using shell scripts is probably not as efficient as writing Python scripts but they may be easier to interface with batch queueing system managing compute-clusters. \n" ] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 2 }