was developped for processing of X-ray snapshots from single-crystals in random orientations (Kabsch, 2014). It is based on the rotation data processing program XDS (Kabsch, 2010). The package consists of two components

• nXDS for processing a stream of snapshots taken under similar experimental conditions
• nXSCALE for combining multiple streams into a final data set of fully corrected reflection intensities obtained by post-refinement.

For all images in a stream of X-ray snapshots nXDS assumes:

• The same detector is used with identical internal settings of its segments (in case of a multi-segment detector).
• The detector is at a fixed position and orientation.
• X-rays are approximately of the same wavelength, beam direction and polarization.
• Direction of the rotation axis (if applicable) and oscillation range do not change.
• Detector type and image formats can be handled (these are the same as for XDS).

As described in this chapter, the data images in a stream are processed in 8 steps

• XYCORR
• FILTER
• INIT
• COLSPOT
• POWDER
• IDXREF
• INTEGRATE
• CORRECT (nXSCALE applied to a single stream)

which are called in succession by nXDS.

Information between the steps is communicated by files, which allows repetition of selected steps with a different set of input parameters without rerunning the whole program. The files generated by nXDS are either ASCII type files that can be inspected and modified by using a text editor, or binary image files in the CBF format, a byte-offset variant of the CBFlib format. Such images are indicated by the file name extension ".cbf". All files have a fixed name defined by nXDS, which makes it mandatory to process each stream in a newly created directory to avoid name clashes. Clearly, one should not run more than one nXDS-job simultaneously in the same directory. Also, output files generated by rerunning selected steps (see Table) should first be given another name if their original contents are meant to be saved.

Data processing begins by copying an appropriate input file template into the data processing directory. Input file templates are provided with the nXDS package for a number of frequently used data collection facilities. The copied input file must be renamed nXDS.INP and edited to provide the correct parameter values for the actual data collection experiment.

All parameters in nXDS.INP are named by keywords containing an equal sign as the last character, and many of them will be mentioned here in context to clarify their meaning. Execution of nXDS invokes in succession each of the 8 program steps described below - or a subset of the steps named in the parameter JOB=. Results and diagnostics from each step are saved in files with the extension .LP attached to the program step name. These files should always be studied carefully to see whether processing was satisfactory or - in case of failure - to find out what could have gone wrong.

XYCORR

calculates lookup tables of spatial corrections for each detector pixel which are stored in the files X-CORRECTIONS.cbf and Y-CORRECTIONS.cbf . In subsequent data processing steps, when the true coordinates of a pixel with respect to the laboratory coordinate system are needed, the correction values for the X- and Y-coordinates are retrieved from the tables and added to the pixel's array coordinates in the data image.

Dependent on the detector, XYCORR computes the spatial corrections in different ways.

• If the data images are already corrected for geometrical distortions, as is often the case, XYCORR produces tables of zeros.
• For spiral read-out imaging plate detectors like the MAR and MAR345 detectors, XYCORR computes the small corrections resulting from radial (ROFF=) and tangential (TOFF=) offset errors of the scanner. For the STOE imaging plate detector the manufacturer's software (Stoe & Cie GmbH, D-Darmstadt) provides a set of 8 calibration values in the crystal file that can be copied to the parameter STOE_CALIBRATION_PARAMETERS=.
• For some multiwire- and CCD detectors that deliver geometrically distorted images, corrections are derived from a calibration image (BRASS_PLATE_IMAGE= file name). This image displays the response to a brass plate containing a square grid of holes which is mounted in front of the detector and illuminated by an X-ray point source. The number of calibration holes and the distance between adjacent holes can be specified by the user via the input parameters MXHOLE= and HOLE_DISTANCE=, respectively, to override the detector specific default values. In addition, the user has the option to specify a minimum number of calibration spots (input parameter MNHOLE=) that must be located by XYCORR in the brass-plate image. This serves as a check that the calibration image was not underexposed.
  XYCORR assumes that the source has been placed exactly at the location to be occupied by the crystal during the actual data collection, as photons emanating from the calibration source are meant to simulate all possible diffracted beam directions. XYCORR also generates the file FRAME.cbf which contains the original brass plate image in which the calibration spots located by the program are marked for visual control.
• The PILATUS pixel detector is assembled from 60 modules arranged in 12 banks of 5 modules. Each module consists of 487 X 195 pixels. There are two types of corrections to compensate the spatial distortions; they are applied in succession.
  Geometrical corrections compensate misorientations of the modules with respect to the X- and Y-axes of the specific detector; these corrections are provided by the manufacturer as two files (input parameters X-GEO_CORR= and Y-GEO_CORR=).
  Parallax corrections are necessary due to the oblique incidence of the scattered beam in the sensor material (SILICON=) of finite thickness (SENSOR_THICKNESS=); these corrections depend on wavelength, detector distance and origin.

Problems:

• A misplaced calibration source leads to an incorrect brass plate image and a useless lookup table as a consequence, impairing the correct prediction of the observed diffraction pattern in subsequent program steps.
• Underexposure of the brass plate calibration image results in an incomplete and unreliable list of calibration spots. Inspect the control image FRAME.cbf to find out whether this has been the case.
  

FILTER

tests each snapshot and removes images that cannot result from diffraction if their pixel contents does not obey Poisson statistics. The list of accepted images replaces the original list specified by the parameter IMAGE_LIST=. The presence of a sufficient number of diffraction spots on the saved images is determined in the subsequent step COLSPOT.

INIT

determines three lookup tables, saved as files BLANK.cbf, GAIN.cbf, and BKGINIT.cbf, that are required by the subsequent processing steps for classifying pixels in the data images as background or belonging to a diffraction spot ('strong' pixels).

• BLANK.cbf contains the detector background noise at each image pixel in the absence of any X-rays. Some of the detectors, like the SIEMENS multiwire detector, do not have background noise and the table will contain just zeros. Otherwise, the table is derived in the following order.
  1. If a data image was taken in the absence of any X-rays (input parameter DARK_CURRENT_IMAGE=file name), this image is used for the table BLANK.cbf.
  2. If the detector noise in each pixel was estimated by a non-negative constant (input parameter OFFSET=), the table BLANK.cbf is set to the given constant.
  3. If neither of the above two cases applies, a constant detector noise is determined from the mean value at the four corners of a few data images. The table BLANK.cbf is set to the determined constant.
• GAIN.cbf codes for the expected variation of the pixel contents in the background region of a data image. The variance of the contents of a pixel in the background region is GAIN*(pixel_contents-detector_noise). The variance is determined from the scatter of pixel values within a rectangular box (input parameters NBX=, NBY=) of size (2*NBX+1)*(2*NBY+1) centered at each image pixel in succession. The table GAIN.cbf is used for distinguishing background pixels from "strong" pixels that are part of a diffraction spot.
• BKGINIT.cbf estimates the initial background at each pixel from a few data images specified by the input parameter BACKGROUND_RANGE=. The lookup table is obtained by adding the X-ray background from each image. Shaded regions on the detector (i.e., beam-stop) or pixels outside a user defined circular region (TRUSTED_REGION=) or pixels with an undefined spatial correction value are classified as untrustworthy and marked by -3. The table should be inspected using the XDS-Viewer program.

Problems:
Some detectors with insufficient protection from electromagnetic pulses may generate badly spoiled images whose inclusion leads to a completely wrong X-ray background table. These images can be identified in INIT.LP by their unexpected high mean pixel contents, and this step should be repeated with a different set of images.

COLSPOT

locates strong diffraction spots occurring in the data images listed in the file named by the input parameter value IMAGE_LIST= and saves the spot centroids on the file SPOT.nXDS.

COLSPOT identifies 'strong' pixels (STRONG_PIXEL=) whose values clearly exceed the fluctuations in a rectangular box (input parameters NBX=, NBY=) of size (2*NBX+1)*(2*NBY+1) centered at each image pixel in succession. Spots are defined as sets of 'strong' pixels adjacent in two dimensions.

A spot is accepted if it contains a minimum number of 'strong' pixels (MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT=) and if the spot centroid is sufficiently close to the location of the strongest pixel in the spot ( SPOT_MAXIMUM-CENTROID=). Images containing less that MINIMUM_NUMBER_OF_SPOTS= are excluded from further processing. At most MAXIMUM_NUMBER_OF_SPOTS= of the strongest spots are saved on file SPOT.nXDS. The names of the accepted images are included in the list of 'good' images (COLSPOT.LST)

Problems:
Sharp edges in the images like ice rings or borders of the untrusted detector regions can lead to an excessive number of 'strong' pixels erroneously classified as contributing to diffraction spots. These aliens could lead to a waste of computing time and even prevent IDXREF to recognize the crystal lattice. To avoid this problem an upper limit for the most strongest spots in an image can be set by the user.

POWDER

generates a powder pattern POWDER.cbf from all spots in the file SPOT.nXDS. This control image could help the user to correct input parameter values for the origin of the detector coordinate system which is critical for the correct indexing of the scattering vectors in the subsequent steps of data processing. The generated image from this step should be inspected by the user with the XDS-Viewer program which adopts an interpretation of cursor positions consistent with the nXDS program.

The plane of the control image POWDER.cbf (1024 × 1024 square pixels) is placed at unit distance from the crystal with its normal coinciding with the incident beam direction computed from the input parameter INCIDENT_BEAM_DIRECTION=. The incident beam intersects the image at pixel coordinates x=511, y=511 at the image center which is marked by crosshairs. The scattering vectors from file SPOT.nXDS are marked in this image where they form a set of concentric rings - the powder pattern - in the ideal case.

The center of the rings should be at the tip of the incident beam - which is the image center. Often the center of the powder rings is found instead offset from its ideal place which can be interpreted to result from an incorrect value for the origin of the detector coordinate system (ORGX=, ORGY=). The pixel offset dx,dy (powder center - image center) is specified by the input parameter POWDER_CENTER_CORRECTION=dx dy q where q is the side length (radians) of a square pixel. From this information a corrected origin (ORGX=, ORGY=) is calculated by the POWDER step and reported in POWDER.LP. For further processing with nXDS you should then specify in nXDS.INP the new values for ORGX=, ORGY= and turn off the parameter or set POWDER_CENTER_CORRECTION= 0 0 0.001 which is its default. The POWDER step could be repeated with the new values for visual control or nXDS could proceed if no further corrections are equired.

Problems:
The control image POWDER.cbf does not show a concentric set of circles despite a large number of spots. This could be due to the presence of too many spots or artefacts on file SPOT.nXDS, to varying directions of the incident beam during data collection or to incorrect parameters describing the multi-segment detector.

IDXREF

uses the initial parameters describing the diffraction experiment from the input file nXDS.INP and the centroids of the spots observed on each image from the file SPOT.nXDS to find orientation, metric, and symmetry of the crystal lattice, and saves the refined values on file XPARM.nXDS.

Spots from file SPOT.nXDS are accepted that are

• within the trusted detector region (parameter TRUSTED_REGION=),
• within a given resolution range (parameter INCLUDE_RESOLUTION_RANGE=),
• and not located in an ice ring (parameter EXCLUDE_RESOLUTION_RANGE=).

To determine a crystal lattice that explains the observed locations of the diffraction spots, IDXREF proceeds for each image as follows.

1. The laboratory coordinates of the diffracted beam wave vector (normalized to 1/λ) corresponding to each spot are calculated from the input parameter values specifying the origin and orientation of the multi-segment detector (POWDER_CENTER_CORRECTION=, ORGX=,ORGY=, DETECTOR_DISTANCE=, DIRECTION_OF_DETECTOR_X-AXIS=, DIRECTION_OF_DETECTOR_Y-AXIS=, SEGMENT=, DIRECTION_OF_SEGMENT_X-AXIS=, DIRECTION_OF_SEGMENT_Y-AXIS=, SEGMENT_ORGX=, SEGMENT_ORGY=, SEGMENT_DISTANCE=, X-RAY_WAVELENGTH=, QX=, and QY=).
2. Subtraction of the incident beam wave vector (given by the parameters INCIDENT_BEAM_DIRECTION= and X-RAY_WAVELENGTH=) from the diffracted beam wave vector leads for each spot to a reciprocal lattice vector when the Laue equations are satisfied.
3. Differences between any two such reciprocal lattice vectors which are above a specified minimal length ( SEPMIN=) are then accumulated in a 3-dimensional histogram with a grid size (RGRID=) defined by the user or determined by the program. These difference vectors will form clusters in the histogram since there are many different pairs of reciprocal lattice vectors of nearly identical vector difference. The clusters are found as maxima in the smoothed histogram (CLUSTER_RADIUS=), and only some (MAXIMUM_NUMBER_OF_DIFFERENCE_VECTOR_CLUSTERS=) of the most densely populated cluster vectors are further used.
4. If the space-group is unknown a vector triplett is selected that best explains the observed difference vector clusters by integral indices (INTEGER_ERROR=). Vector clusters that can be consistently indexed with respect to this triplett are connected in a tree. Trees using the same vector triplett as a basis - but offset only by a constant vector in reciprocal space - can be joined into a new, common reciprocal basis explaining all members of the trees. The parameter MERGE_TREE= defines a minimum fraction of explained difference vectors required for inclusion in a common basis.
5. If space-group and cell constants are specified (input parameters SPACE_GROUP_NUMBER=, UNIT_CELL_CONSTANTS=), the observed difference vector clusters are interpreted (indexed) (INTEGER_ERROR=) by a rotated version of the given unit cell using exhaustive search (NUMBER_OF_TESTED_BASIS_ORIENTATIONS=).
6. The basis vectors and the 30 best short lattice vectors with attached indices are listed in IDXREF.LP. If many of the indices deviate significantly from integral values, the program is unable to find a reasonable lattice basis and all further processing will be meaningless for this image.
7. Based on the orientation and metric of the reduced cell, up to 3,000 of the strongest spots are indexed by the local indexing method. This method considers each spot as a node of a tree and identifies the largest subtree of nodes which can be assigned reliable indices. The number of reflections in the ten largest subtrees is reported and usually shows a dominant first tree corresponding to a single lattice, whereas alien spots are found in small subtrees. Input parameters that control the local indexing are INDEX_ERROR=, INDEX_MAGNITUDE=, INDEX_QUALITY=.
8. Reflections in the largest subtree are used for initial refinement of the basis vectors of the reduced cell, the incident beam wave vector, and the detector parameters (controlled by the input parameter REFINE(IDXREF)=). Spots are included in the refinement only if their coordinates can be explained with reasonable accuracy (MAXIMUM_ERROR_OF_SPOT_POSITION=).
9. The refined metric parameters of the reduced cell are used for testing each of the 44 possible lattice types (Kabsch, 1993). Possible lattice symmetries (decision constants MAX_CELL_AXIS_ERROR=, MAX_CELL_ANGLE_ERROR=) are reported but no automatic decision for the space-group is made. If the crystal symmetry is unknown, data processing will continue with the crystal being described by its reduced cell basis vectors and triclinic symmetry. Otherwise, the observed spots are reindexed in terms of the given cell whose basis vectors are found from the reduced cell vectors.
10. After initial refinement based on the reflections in the largest subtree, all spots which can now be indexed are included. Spots not belonging to the crystal lattice are given indices 0,0,0. The initial file SPOT.nXDS is replaced by a file of identical name - now with indices attached to each observed spot.
11. The indexing run is considered successful if some minimum number (MINIMUM_FRACTION_OF_INDEXED_SPOTS=) (default 40%) of the given spots can be explained; otherwise the image will be excluded from further processing.

Problems:

• Indices of many short lattice vectors deviate significantly from integral values. This can be caused by incorrect input parameters, like oscillation angle, detector position, by a large fraction of alien spots in SPOT.nXDS caused by multiple lattices, or by placing the detector too close to the crystal.
• Indexing and refinement is unsatisfactory despite well-indexed short lattice vectors. This probably results from selection of an incorrect index origin caused by inaccurate values for the input parameters ORGX= and ORGY=. A visual check of a data image or the control image POWDER.cbf with the XDS-Viewer program could be quite helpful for finding the approximate coordinates for the detector origin.

INTEGRATE

determines the recorded intensity of reflections of all successfully indexed data images using the parameter values listed in file XPARM.nXDS. The refined parameters together with the recorded intensities are saved on file INTEGRATE.HKL, representing the main result of processing the specified stream of snapshots.

For each image INTEGRATE carries out the following steps.

• Labelling of each image pixel by the indices of the nearest reflection whereby invalid pixels (MINIMUM_VALID_PIXEL_VALUE=) or pixels in the untrusted detector region (EXCLUDE_RESOLUTION_RANGE=) or those outside the allowed resolution range (INCLUDE_RESOLUTION_RANGE=) are ignored. Pixels from each detector segment are sorted by their index labels so that contributions to the same reflection follow each other.
• Classification of image pixels into background and spot. Pixel contents that exceed the background by a given multiple of standard deviations are called strong (input parameters SIGNAL_PIXEL=, BACKGROUND_PIXEL=).
• Determination of extension and form of the diffraction spots from the distributions of the pixels contributing to the strong spots. A spot is described by the optional input parameters
  1. BEAM_DIVERGENCE= is twice the opening angle of a cone with the diffracted beam wave vector as cone axis. The interception of the cone with the data image traces the boundary of the spot and includes some neighbouring background pixels.
  2. BEAM_DIVERGENCE_E.S.D.= characterizes the Gaussian spot shape by its standard deviation.
  3. REFLECTING_RANGE_E.S.D.= is the standard deviation (1/Å) of a Gaussian modeling the rocking curve.
  All of the parameters describing extension and form of the spots can be determined automatically from the idea that the observed spots should be expected.
• Selection of strong spots for refinement. Centroids computed from the strong pixels must be reasonably close to their calculated position using the initial diffraction parameters from XPARM.nXDS (input parameter MAXIMUM_ERROR_OF_SPOT_POSITION=). Control of the refinable parameters is provided by the user input parameter REFINE(IDXREF)=.
• Construction of a specific coordinate system for each spot that is centered on the surface of the Ewald sphere. In this coordinate system α and β span the plane tangential to the Ewald sphere with the α-axis perpendicular to the incident- and diffracted beam wave vector of the reflection. Intensity profiles of the spots become very similar when expressed in such a system (Kabsch, 1988b). The number of grid points in this coordinate system used for representing the transformed reflection profile are usually chosen automatically by nXDS; the user has the option to override the automatic assignment by specifying the input parameter NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA=.
• Generation of a spot template by superimposing transformed profiles of strong reflections that are not overloaded (OVERLOAD=). Grid points with a value above a minimum percentage of the maximum in the template (CUT=) are defined as elements of the integration domain.
• Integration by profile fitting (PROFILE_FITTING=) with respect to the spot template for all reflections predicted to occur in the image. Incompletely recorded reflections with less than MINPK= % intensity compared with the spot template will be discarded. Otherwise, the missing intensity is estimated from the learned profile of the spot template.
  Reflections in the 'blind' region (MINIMUM_ZETA=) or those too far from the Ewald sphere (MINIMUM_EWALD_OFFSET_CORRECTION=) are omitted from the output file INTEGRATE.HKL.

Problems:

• Off-centered profiles indicate incorrectly predicted reflection positions by using the parameters provided by the file XPARM.nXDS (i.e., misindexing by using a wrong origin of the indices).
• Profiles extending to the borders of the box indicate too-small values for BEAM_DIVERGENCE=. This leads to incorrect integrated intensities because of truncated reflection profiles and unreliable background determination.

CORRECT

determines scaling and correction factors for the recorded intensities and standard deviations of all reflections in the file INTEGRATE.HKL, reports quality and completeness of the data, and saves the fully corrected integrated intensities on file nXDS_ASCII.HKL. This file can be directly read by XDSCONV (see XDS) which converts the reflection data into various formats required by software packages for crystal structure determination like CCP4, CNS, X-PLOR, or SHELX.

The integrated intensities of the reflections saved on file INTEGRATE.HKL may or may not have been indexed in the correct space group. For the purpose of integration it is only important that all reflections occurring in the data images have been accurately located and indexed with respect to some unit cell basis. Thus, the INTEGRATE step can be carried out even in the case of an unknown space group because the reflection indices with respect to the correct space group are always a linear transformation of the original indices used in the INTEGRATE step and reported in INTEGRATE.HKL.
Presently, CORRECT requires knowledge of the space group and approximate cell constants. Plausible space group candidates and their conventional cell parameters were already reported in IDXREF.LP and can now be tried in the CORRECT step without the need for rerunning previous program steps (specify JOB=CORRECT, SPACE_GROUP_NUMBER=, UNIT_CELL_CONSTANTS=) in nXDS.INP and rerun nxds_par).

The possibility to compare the new data with a reference data set (REFERENCE_DATA_SET=) is particularly useful for resolving the issue of alternative settings in case the lattice symmetry is higher than the point group symmetry of the crystal (e.g. P4, P6, R3). NOTE: It is assumed that the reference reflections are indexed with respect to the basis specified in nXDS.INP (SPACE_GROUP_NUMBER=, UNIT_CELL_CONSTANTS=).
Also, reference data are quite useful for recognizing misindexing. CORRECT reindexes the input reflections from file INTEGRATE.HKL thereby resolving possible indexing ambiguities by comparison with a given reference data set or by a selective breeding algorithm that yields, on average, the highest correlation with intensities from all other images (Kabsch, 2014).

Reflections from file INTEGRATE.HKL are accepted that are

• not overloaded (parameter OVERLOAD=),
• within a given resolution range (parameter INCLUDE_RESOLUTION_RANGE=),
• recorded on an image sufficiently close to the calculated Bragg peak (parameter MINIMUM_EWALD_OFFSET_CORRECTION=).

Intensity correction factors are calculated for each accepted reflection that are due to

• polarization of the incident beam (parameters FRACTION_OF_POLARIZATION=, POLARIZATION_PLANE_NORMAL=),
• air absorption (parameter AIR=),
• differences in path lengths of the diffracted beam in the detector material (parameters SILICON=, SENSOR_THICKNESS=),
• Ewald offset (defined as angle between the center of the rotation/still image and the calculated angle of the Bragg peak on the Ewald sphere).
• differences a reflection remains in diffracting condition (Lorentz correction factor for still or rotation images).

Recognition and removal of outliers among the images is based on medians of

• the incident beam directions
• unit cell constants (decision parameters MAX_CELL_AXIS_ERROR=, MAX_CELL_ANGLE_ERROR=)

CORRECT improves the final data set by a post-refinement procedure that iteratively changes the crystallographic parameters of all images so that intensities of symmetry equivalent reflections become as similar as possible to each other and simultaneously minimize discrepacies between calculated and observed spot positions on the detector segments. The post-refinement can be controlled by input parameters like USE_REFERENCE_IN_POSTREFINEMENT=, POSTREFINE=, REFINE_SEGMENT=.

After postrefinement a simple method is chosen to reduce the potentially very large number of reflections to a manageable size - yet with little loss of information. For each unique reflection (considering Friedel pairs as different) in general two statistically independent intensity estimates are generated from reflections taken from randomly selected snapshots. Now, comparison of pairs of independent intensity estimates allows to correct the original intensity error estimates, obtained from counting statistics during the INTEGRATE step, by a resolution-dependent factor so that the new error estimates agree with the sample statistics of the intensity pairs. Outliers from the Wilson plot, often arising from ice rings in the data images, are recognized and marked by a negative sign attached to their e.s.d. in the final output file (REJECT_ALIEN=).
The reduced set of statistically independent intensity observations thus obtained is sufficient to assess the quality of the final data set as function of resolution by the various indicators, like I/SIGMA, R_mrgd-F, CC(1/2), etc (Diederichs and Karplus, 1997). This reduced set of fully corrected data are saved on file nXDS_ASCII.HKL.

Problems:

• A reference data set is provided but shows very poor correlation with most of the newly processed snapshot data. This could happen if the reference reflections mistakenly refer to a unit cell basis other than the specified one in nXDS.INP.
• Incomplete data sets may lead to wrong conclusions about the space-group, as some of its symmetry operators might not be involved in the R-factor calculations.
• Often, the CORRECT step is repeated several times. One should remember, that nXDS overwrites the earlier versions of the output files nXDS_ASCII.HKL and CORRECT.LP.