Calibration constants¤
The extra.calibration module helps you to find & load detector calibration
data stored in the Calibration Catalog.
This is used for most imaging detectors in use at European XFEL to record
and track datasets called calibration constants derived from characterization
measurements and used for data corrections.
To find constants by conditions for a particular point in time, create a
condition object for the relevant detector type and use
CalibrationData.from_condition:
from extra.calibration import CalibrationData, LPDConditions
lpd_cd = CalibrationData.from_condition(
LPDConditions(memory_cells=200, sensor_bias_voltage=250),
"FXE_DET_LPD1M-1",
event_at="2022-05-22T02:00:00",
)
# Load one constant for all found modules
offset = lpd_cd["Offset"].ndarray()
You can also find a group of constants produced by one characterisation process by specifying a CalCat report, e.g. Report 3757:
In a Jupyter notebook, you can show the selected constants in a table:
| Modules | BadPixelsDark | Noise | Offset | ThresholdsDark |
|---|---|---|---|---|
| 0 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 |
| 1 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 |
| 2 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 |
| ... | ||||
| 14 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 |
| 15 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 | 2023-07-29 19:34 |
Found constants¤
extra.calibration.CalibrationData ¤
Bases: Mapping
Collected constants for a given detector
This can represent multiple constant types (offset, gain, bad pixels, etc.)
across multiple modules. It works as a mapping keyed by constant type
(e.g. cd["Offset"]), giving you MultiModuleConstant objects.
from_condition
classmethod
¤
from_condition(condition: ConditionsBase, detector_name, calibrations=None, client=None, event_at=None, pdu_snapshot_at=None, begin_at_strategy='closest')
Look up constants for the given detector conditions & timestamp.
condition should be a conditions object for the relevant detector type,
e.g. DSSCConditions.
event_at and pdu_snapshot_at should either be an ISO 8601
compatible string or a datetime-like object. It may also be a
DataCollection object from EXtra-data to use the beginning of the
run as a point in time.
from_report
classmethod
¤
Look up constants by a report ID or path.
Constants produced together in the same characterisation are grouped in CalCat by their report. This method accepts either the integer report ID or the full filesystem path of the report.
from_correction
classmethod
¤
from_correction(metadata_file_or_proposal: str | Path | int, run: int | None = None, detector_name=None, *, client=None, use_calcat=True)
Find constants used to produce corrected data.
This can be called with a proposal & run number and a detector name (e.g. 'FXE_XAD_JF1M'), or with the path of a YAML metadata file from the EuXFEL offline calibration pipeline.
By default, this method retrieves additional metadata from CalCat. Pass use_calcat=False to read only the minimal info in a YAML file.
from_data
classmethod
¤
from_data(data: DataCollection, detector_name: str, calibrations=None, client=None, begin_at_strategy='closest', **kwargs)
Look up constants applicable to given a dataset.
data should be an EXtra-data DataCollection object containing
the necessary metadata to identify the detector conditions.
detector_name refers to the detector identifer as used in CalCat,
typically identical to its Karabo domain, i.e. the first part of
its device IDs.
The remaining arguments behave in the same way as for
CalibrationData.from_condition and any additional keyword
arguments are passed on to the applicable ConditionsBase.from_data
method, e.g. AGIPDConditions.from_data.
aggregator_names
property
¤
Data aggregator names for modules. May include missing modules.
pdu_names
property
¤
Names of the specific detector units making up the detector.
May include missing modules.
require_calibrations ¤
Drop any modules missing the specified constant types
select_modules ¤
Return a new CalibrationData object with only the selected modules
One of module_nums, aggregator_names or qm_names must be specified.
select_calibrations ¤
Return a new CalibrationData object with only the selected constant types
merge ¤
Combine two or more CalibrationData objects for the same detector.
Where the inputs have different constant types or different modules, the output will include all of them (set union). Where they overlap, later inputs override earlier ones.
summary_table ¤
Make a table overview of the constants found.
Columns are calibration types, rows are modules. If there are >4 calibrations, the table will be split up into several pieces with up to 4 calibrations in each.
The table(s) returned should be rendered within Jupyter notebooks, including when converting them to Latex & PDF.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module_naming
|
str
|
modnum, aggregator or qm, to change how the modules are labelled in the table. Defaults to modnum. |
'modnum'
|
markdown_table ¤
Make a markdown table overview of the constants found.
Columns are calibration types, rows are modules. If there are >4 calibrations, the table will be split up into several pieces with up to 4 calibrations in each.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module_naming
|
str
|
modnum, aggregator or qm, to change how the modules are labelled in the table. Defaults to modnum. |
'modnum'
|
display_markdown_table ¤
Display a table of the constants found (in a Jupyter notebook).
Columns are calibration types, rows are modules. If there are >4 calibrations, the table will be split up into several pieces with up to 4 calibrations in each.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module_naming
|
str
|
modnum, aggregator or qm, to change how the modules are labelled in the table. Defaults to modnum. |
'modnum'
|
extra.calibration.MultiModuleConstant
dataclass
¤
MultiModuleConstant(constants: Dict[str, SingleConstant], module_details: List[Dict], detector_name: str, calibration_name: str)
Bases: Mapping
A group of similar constants for several modules of one detector.
This works as a mapping holding SingleConstant objects.
Keys can be module numbers (offset[0]), data aggregator names
(offset['LPD00']), QxMy names (offset['Q1M1']) or Physical Detector Unit
(PDU) names.
aggregator_names
property
¤
Data aggregator names for the modules where we have this constant
qm_names
property
¤
Names like Q1M3 for the modules where we have this constant, if applicable
pdu_names
property
¤
Names of the specific detector units making up the detector.
Only includes modules where we have this constant.
select_modules ¤
Return a new MultiModuleConstant object with only the selected modules
One of module_nums, aggregator_names or qm_names must be specified.
ndarray ¤
Load this constant as a Numpy array.
If parallel is specified, the per-module constants are loaded in
parallel using N worker processes.
dimension_names ¤
Get the order of dimensions for this constant (if it was saved)
Possible dimension names include "module", "cell", "gain", "fast_scan" and "slow_scan".
This is the same as the .dimensions property, but allows passing the root directory for calibration files.
xarray ¤
Load this constant as an xarray DataArray.
module_naming may be "modnum", "aggregator" or "qm" to use different
styles of labelling for the modules dimension.
If parallel is specified, the per-module constants are loaded in
parallel using N worker processes.
extra.calibration.SingleConstant
dataclass
¤
SingleConstant(path: Path, dataset: str, ccv_id: Optional[int], pdu_name: Optional[str], _metadata: dict = dict(), _have_calcat_metadata: bool = False)
A calibration constant for one detector module
CalCat calls this a calibration constant version (CCV).
dimensions
property
¤
Get the order of dimensions from the constant file (if it was saved)
dimension_names ¤
Get the order of dimensions from the constant file (if it was saved)
This is the same as the .dimensions property, but allows passing the root directory for calibration files.
metadata ¤
Get a specific metadata field, e.g. 'begin_validity_at'
This may make a request to CalCat if the value is not already known.
metadata_dict ¤
Get a dict of available metadata
If this constant didn't come from CalCat but we have a CalCat CCV ID, this will fetch metadata from CalCat.
Conditions objects¤
extra.calibration.AGIPDConditions
dataclass
¤
AGIPDConditions(sensor_bias_voltage: float, memory_cells: int, acquisition_rate: float, gain_setting: Optional[int], gain_mode: Optional[int], source_energy: float = 9.2, integration_time: int = 12, pixels_x: int = 512, pixels_y: int = 128)
Bases: ConditionsBase
Conditions for AGIPD detectors
extra.calibration.DSSCConditions
dataclass
¤
DSSCConditions(sensor_bias_voltage: float, memory_cells: int, pulse_id_checksum: Optional[float] = None, acquisition_rate: Optional[float] = None, target_gain: Optional[int] = None, encoded_gain: Optional[int] = None, pixels_x: int = 512, pixels_y: int = 128)
Bases: ConditionsBase
Conditions for DSSC detectors
extra.calibration.JUNGFRAUConditions
dataclass
¤
JUNGFRAUConditions(sensor_bias_voltage: float, memory_cells: int, integration_time: float, gain_setting: int, gain_mode: Optional[int] = None, exposure_timeout: int = 25, sensor_temperature: float = 291, pixels_x: int = 1024, pixels_y: int = 512)
Bases: ConditionsBase
Conditions for JUNGFRAU detectors
extra.calibration.LPDConditions
dataclass
¤
LPDConditions(sensor_bias_voltage: float = 250.0, memory_cells: int = 512, memory_cell_order: Optional[str] = None, feedback_capacitor: float = 5.0, source_energy: float = 9.3, category: int = 0, pixels_x: int = 256, pixels_y: int = 256, parallel_gain: bool = False)
Bases: ConditionsBase
extra.calibration.ShimadzuHPVX2Conditions
dataclass
¤
Bases: ConditionsBase
Detectors and modules¤
Calibration data is associated with physical detector units (PDUs), with an actual detector consisting of one or more of these PDUs then generally called modules. PDUs can freely move between detectors of the same type in a process called mapping, and take their calibration data alongside with them.
This is automatically taken into account when using the CalibrationData APIs described
here, but you can also obtain the detector metadata and mapping information explicitly:
extra.calibration.DetectorData ¤
Bases: Mapping
Detector consisting of one or more modules
A detector can house one or more detector modules. For the purpose of tracking calibration data, a physical detector unit (PDU) is mapped to these detector modules. When a PDU is moved (i.e. mapped) to a different detector module or even a different detector, existing calibration data is therefore automatically applied correctly.
This object exposes detector modules in a dict-like interface mapping data aggregators to detector modules at a particular point in time. Alternatively, the module index may also be used as a key.
Some attributes may be None if this object was initialized from
incomplete information rather than CalCat, e.g. metadata of
correction jobs.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
int
|
Detector numerical ID. |
identifier |
str
|
Detector identifier. |
karabo_control_domain |
str
|
Karabo domain for control devices. |
number_of_modules |
int
|
Number of modules for the full detector, may be more than currently installed or selected. |
pdu_snapshot_at |
str
|
ISO format date the mapping is taken from. |
from_numeric_id
classmethod
¤
Look up a detector and its modules by CalCat numeric ID.
pdu_snapshot_at should either be an ISO 8601 compatible string
or a datetime-like object. It may also be a DataCollection
object from EXtra-data to use the beginning of the run as a
point in time.
from_identifier
classmethod
¤
Look up a detector and its modules by identifier.
pdu_snapshot_at should either be an ISO 8601 compatible string
or a datetime-like object. It may also be a DataCollection
object from EXtra-data to use the beginning of the run as a
point in time.
from_instrument
classmethod
¤
Look up a detector and its modules by instrument.
identifier may be a string restricting the result using Unix
shell-style glob patterns.
pdu_snapshot_at should either be an ISO 8601 compatible string
or a datetime-like object. It may also be a DataCollection
object from EXtra-data to use the beginning of the run as a
point in time.
list_by_instrument
staticmethod
¤
List all detectors by instrument.
extra.calibration.DetectorModule
dataclass
¤
DetectorModule(pdu_id: int, physical_name: str, aggregator: str, detector: str, virtual_device_name: str | None, module_index: int, module_number: int | None, detector_type: str, legacy_uuid: int | None, source_name: str | None)
Detector module.
A module installed in a detector is represented by a physical detector unit (PDU) mapped to this module's logical position in the detector at a particular point in time.
Calibration data is always associated with PDUs rather than the detector itself even in the case of a single module and thus PDU.
Attributes:
| Name | Type | Description |
|---|---|---|
pdu_id |
int
|
PDU numerical ID. |
physical_name |
str
|
PDU identifier. |
aggregator |
str
|
Data aggregator the PDU is mapped to. |
detector |
str
|
Detector identifier the module is part of |
virtual_device_name |
str
|
Identifier or QM name for the logical module within the detector, e.g. Q1M1. |
module_index |
int
|
Enumerated module index within the detector when sorted by aggregator, contiguous and always starting at 0. |
module_number |
int
|
Logical module number within the detector, may start at any number and have gaps. |
detector_type |
str
|
Detector type string of this PDU. |
Bad pixel values¤
The calibration pipeline produces masks along with corrected data, in keys
called image.mask or data.mask depending on the detector. Zeros in the mask
represent normal, good pixels, while any other value indicates one or more
reasons why the data may be atypical.
Most of these values indicate different kinds of 'bad' data. But it also
includes values like NON_STANDARD_SIZE for pixels, usually at sensor edges,
which are intentionally larger than most, and thus capture more photons.
extra.calibration.BadPixels ¤
Miscellaneous¤
extra.calibration.lpd_dark_consts_with_fallback ¤
lpd_dark_consts_with_fallback(condition: LPDConditions, detector_name, event_at=None, preference_time=timedelta(days=7), **kwargs)
Look up LPD dark constants with fallback to constants for all memory cells
The parameters are mostly the same as for CalibrationData.from_condition(). Constants with the matching memory cell order will be used if they're closer in time than the fallback, or up to preference_time (default 5 days) further.