Changelog

Unreleased

Note

All of the changes here are deployed to our current environment, even though a release hasn't been made for them yet. If you want to have these updates in a personal environment you'll need to install the package from git.

Installation command

pip install git+https://github.com/European-XFEL/EXtra.git

Added:

• [PumpProbePattern.pumped_pulses_ratios()][extra.components.PumpProbePattern.pumped_pulses_ratios] determining the ratio of pumped pulses per train (#317).
• TimeserverPulses.plot_xray_patterns to visualize the X-ray pulse patterns present in timeserver data (#499).
• DelayLineDetector.edges can include the edge amplitudes in its result if requested (#505).
• AdqRawChannel.train_data and AdqRawChannel.pulse_data are now reading data in parallel by default (#497).
• AdqRawChannel.pulse_data accepts a dtype parameter for the output array.
• [AdqRawChannel.split_trains][extra.components.AdqRawChannel.split_trains]
• AGIPDConditions now recognizes current source constant types
• CalibrationData.from_data method to find the constants suitable for a given DataCollection
• DetectorData to obtain detector metadata and module mapping from the Calibration Cataloge (#408).
• PulsePattern.select_pulses, PulsePattern.deselect_pulses and PulsePattern.union to modify pulse patterns (#417).
• ManualPulses to generate custom pulse pattern with the same interface as existing PulsePattern-based components (#417).
• SpectrometerCalibration to provide a Jupyter widget for energy calibration of 2D X-ray spectrum data (#363).
• Pre-built packages will be available on PyPI for Python 3.10 - 3.13 from the next release, rather than only Python 3.10 (#377).
• CalibrationData.from_condition supports DataCollection objects to reference a point in time
• CalibrationData.from_correction method to find the constants used to correct a particular run.
• CalibrationData.reports_info method to show a summary of the reports associated with groups of calibration constants.
• lpd_dark_consts_with_fallback function to find LPD dark constants with a fallback to constants recorded with all memory cells.
• Scan.group_data() method to make an xarray or pandas GroupBy object based on scan steps (#379).
• Scan.from_motor_targets() method to make a scan object based on the target positions of the motor, excluding points where the actual position is too far from the target.
• fit_gaussian() has a new nans_on_failure argument to allow returning NaN arrays when fitting fails (#385).
• fit_gaussian() now supports DataArray inputs (#385).
• AdqRawChannel now has a public .pulses property. It is either a PulsePattern object or None if the class is instantiated with the pulses=False option. (#411)
• Added support for plotting wrapped Julia arrays from PythonCall.jl to imshow2().
• PulsePattern.info() to give a brief
• TOFAnalogResponse can deconvolve the Auger-Meitner and photon line if the flag deconvolve is set to True. summary about a pulse pattern (#435).
• StepTimer class for measuring performance within functions.
• Grating1DCalibration can use the calibration mask, removes the background subtraction (leaving it to the calibration DB), and use robust tools to improve the fit quality (#510).

Fixed:

• fit_gaussian() would sometimes return a negative sigma, now it will always return a positive sigma (#405).
• Passing colorbar=False to imshow2() now applies to DataArray inputs as well (#419).
• Fix incorrect time units when using OpticalLaserDelay with pre-2022 BAM data (#425).
• CookieboxCalibration takes polarization into account when calculating transmission.
• CookieboxCalibration avoids crashes if eTOF data is noisy by using RANSAC for the fit, using the Golden method for the t0 search, and added an option for using photo-electron counting for increased robustness. Peaks are also masked if not found to avoid contaminating the fit (#476).
• Scan.plot_bin_by_steps() correctly uses the uncertainty_method parameter.
• Fixed support for plotting images with zeros/negative numbers with imshow2() and lognorm=True (#491).
• Fixed counting threshold for CookieboxCalibration (#498).
• CookieboxCalibration bug fix for new extra.applications.base.SerializableMixin interface (#506).
• CookieboxCalibration transmission corrected based on pulse energy (#512).

Changed:

• extra.recipes package renamed to extra.applications
• XGM.pulse_energy() will now insert NaNs instead of zeros as a fill value for runs with a varying number of pulses (#347).
• imshow2() will now add a colorbar automatically (#351).
• Calibration conditions objects can now be displayed as Markdown in Jupyter notebooks (#381).
• Increased maximal length of pulse masks returned by pulse_mask() to 4096 (#461).
• CookieboxCalibration calculates the transmission using the sum of the trace instead of the fit amplitude for robustness.
• extra.applications.base.SerializableMixin._fromdict becomes a classmethod.
• CookieboxCalibration and TOFAnalogResponse delegate parallelization to the AdqRawChannel class (#509).
• TOFAnalogResponse averages analog pulses using thresholding from AdqRawChannel to clean up the data (#511).
• CookieboxCalibration use processes to parallelize over eTOFs (#513).

2025.1

Added:

• CookieboxCalibration to calibrate data from eTOFs after taking a calibration run (#284).
• Grating2DCalibration to calibrate data from a 2D grating detector (#284).
• Grating1DCalibration to calibrates 1D grating information (e.g.: collected from the Gotthard detector in SQS) (#318).
• Exposed detector data components from extra_data in extra.components (AGIPD1M, AGIPD500K, DSSC1M, JUNGFRAU, LPD1M) (#177).
• imshow2 now supports plotting 2D DataArrays properly (#333).
• Added hyperslicer2() to make plotting image arrays easier (#348).
• Added ridgeplot() function (#354).
• reorder_axes_to_shape utility function (#349).
• ShimadzuHPVX2Conditions to retrieve constants for dynamic flat-field correction (#254).
• MultiModuleConstant and SingleConstant both have new .dimensions() methods to retrieve dimension labels, where these are stored with constants (#350).

Changed:

• Timepix3.spatial_bins() is now a static method.
• The XGM component will now emit warnings when it detects the wrong number of pulses have been saved in the slow data property. There is also a new force_slow_data argument to XGM.pulse_counts() to always return whatever was saved in the slow data (#161).
• CookieboxCalibration has been adapted to do the normalization only by the Auger peak (#367).
• LPDConditions now accepts parallel_gain (#254).
• JUNGFRAUConditions now accepts exposure_timeout (#254).
• CalibrationData.from_condition has a new begin_at_strategy parameter (#254).

Fixed:

• Fixed [PumpProbePulses.is_constant_pattern()][extra.components.PumpProbePulses.is_constant_pattern] to properly take pump probe flags into account when determining whether a pattern is constant (#313).
• AdqRawChannel.pulse_edges() now also supports data where the trace is too short for the actual number of pulses present (#312).
• Fixed issues with pulse separation in AdqRawChannel with variable pulse patterns and those with trains missing ADQ data (#310).
• AdqRawChannel now properly enumerates channels starting with 1 rather than 0 as in the Karabo device.
• Fixed reading of the Scantool.acquisition_time property for newer Scantool versions (#303).
• CookieboxCalibration: allow flagging of bad calibration points, to allow using others in the mask_calibration_point(tof_id, energy) function (#318).
• CookieboxCalibration: Order plot_calibration_data y axis by energy to avoid unordered axis in case the energy scan was done out of monotonically increasing order (#318).
• CookieboxCalibration: Use Auger-Meitner mean to align all plots in plot_calibration_data by the Auger-Meitner peak (#318).
• CookieboxCalibration: New function plot_calibrations, which plots only the calibration factors in the same plot for diagnostics. Often this is all that one is interested in (#318).
• Grating2DCalibration: Crop image based on rotation angle to avoid artifacts caused by out-of-bound effects. Add background root-mean-squared error as uncertainty band in the output (#318).
• Grating2DCalibration: If provided, use extra motor information as independent variable in the fit (useful when data includes motor movements additionally) (#318).
• Fixed Scan.plot_bin_by_steps() to show 2D data (#320).
• Restrict the version of Cython used to build while we figure out an issue with Cython 3.1 (#328).
• Fixed behaviour of Scan.plot_bin_by_steps() when passed a custom axis (#334).

2024.2

Added:

• Timepix3 to access raw hits and centroids from the Timepix3 detector (#231).
• Scan.plot() now allows passing a figsize (#262).
• JF4MHalfMotors to access positions of motors moving halfs of Jungfrau detector (#224).

Fixed:

• Karabacon 3.0.10 is now supported by the Scantool (#212).
• Scan.plot_bin_by_steps() would previously ignore the title/xlabel/ylabel arguments, now it actually uses them (#237).
• `Scan.bin_by_steps() now preserves any additional dimensions in the data to be binned, so it can produce e.g. an average spectrum per scan step (#269).
• AdqRawChannel.pulse_data() no longer erroneously reads in the same train data for every pulse if there is only a single pulse per train (#259).

Changed:

• gaussian() has a new norm parameter to allow disabling normalization, which fit_gaussian() will use by default (#221).
• fit_gaussian() has a new A_sign parameter to specify the expected orientation of the peak (#222).
• AdqRawChannel.pulse_data() no longer throws an exception if the pulse pattern refers to data beyond the acquired traces, but instead fills this with NaN or -1 depending on data type (#245).

2024.1.2

Added:

• AGIPD1MQuadrantMotors to access positions of motors moving quadrants of AGIPD1M detector (#148).
• A module to expose the DAMNIT API.
• AdqRawChannel to access and process raw traces saved by fast ADQ digitizers (#35).
• Hits obtained via DelayLineDetector.hits() can now optionally be restricted to hits reconstructed with a certain method. (#170).
• DldPulses can optionally enumerate PPL-only pulses using negative indices to stay compatible with PPL-unaware data. (#167).
• The slowTrains XGM property is available through XGM.slow_train_energy() (#162).
• XGM.pulse_energy() can now take a series=True argument to return a 1D Series instead of a 2D DataArray (#162).
• Data indexed by pulse or train can now be aligned with and added as additional columns to sparse DLD data (#166, #173).
• OpticalLaserDelay to obtain the time delays between FEL and optical laser pulses (#165).
• Empty trains can now optionally be included when determining constant pulse patterns via [XrayPulses.is_constant_pattern()][extra.components.XrayPulses.is_constant_pattern] (#156).
• Check whether any pulse patterns are interleaved with is_interleaved_with() or directly SA1/SA3 with is_sa1_interleaved_with_sa3() (#155).
• Obtain MachinePulses from any other timeserver-based pulse components directly via machine_pulses() or get machine repetition rate directly from machine_repetition_rate() (#155).
• A helper function named fit_gaussian() (#131).
• A new method Scan.split_by_steps() (#169).
• PumpProbePulses.pulse_mask now has an option to give a mask for only FEL or only pump laser pulses (#174).
• BadPixels flag values for interpreting masks in corrected detector data (#172).
• Deprecate pulse dimension time in favor of pulseTime (#178).
• Constant fraction and dynamic leading edge discriminators to determine arrival time of fast pulses on an analog signal (#25).
• Sinc interpolation method (#25).
• extra.calibration.CalibrationData now has methods to create & display a markdown table of the constants found. This is primarily for use in Jupyter notebooks (#202).
• More informative plots from Scan.plot() (#201).

Fixed:

• Sometimes an XGM will record the wrong number of pulses in its slow data property, which would cause the pulse energy to not be retrieved properly. Now the XGM will rely only on the fast data to find the number of pulses in .pulse_energy() (#153).
• Attempt to convert calibration condition values to floats before defaulting to strings. This prevents misinterpretation of boolean values as strings (e.g., "True") (#193).
• Add tabulate to the package dependencies.

2024.1.1

Added:

• MachinePulses to investigate pulse pattern data beyond only X-rays and optical lasers (#138).
• Implemented Scan.bin_by_steps() and Scan.plot_bin_by_steps() to help with averaging data over scan steps (#124).
• pulse_periods(), pulse_repetition_rates() and train_durations() methods to obtain statistics about the pulses in all pulse pattern components (#114).
• extra.calibration module to find and load calibration constants.

2024.1

Added:

• DelayLineDetector to access raw and reconstructed results from the offline delay line detector processing pipeline (#103).
• An XGM component to access XGM devices (#53).
• PumpProbePulses to combine X-ray and optical laser pulses in a single pattern (#24).
• The Scan component to automatically detect steps within a motor scan (#4).
• DldPulses to access pulse information saved during delay line detector event reconstruction (#42).
• The helper function imshow2 to provide good defaults when plotting images (#38).

Changed:

• The get_ prefix was deprecated for some method names in the pulse pattern components (#106).
• All methods in XrayPulses and OpticalLaserPulses now support labelled results and default to it (#40).
• Scantool's __repr__() functionality to print information was moved to Scantool.info() (#29).

2023.2

The initial release!

Added:

• A Scantool component for the EuXFEL scantool, in #2.
• Components for extracting the X-ray and optical laser pulse patterns: XrayPulses and OpticalLaserPulses in #5.
• A collection of utility functions.
• Sub-modules to forward EXtra-data, EXtra-geom, and karabo-bridge-py.