let_it_be_3D package

Submodules

core.angles_and_distances module

core.angles_and_distances.add_errors_between_computed_and_ground_truth_angles(gt_angles: Dict, anipose_io: Dict) → Dict

Set the errors between the computed and ground truth angles.

Parameters:

• gt_angles – Angles from ground_truth = ground_truth_config[“angles”].

• anipose_io – Containing information to validate calibration in comparison with ground truth.

Returns:

anipose_io – Added “angles_error_ground_truth_vs_triangulated”.

Return type:

dict

core.angles_and_distances.add_errors_between_computed_and_ground_truth_distances_for_different_references(anipose_io: Dict, ground_truth_distances: Dict) → Dict

Calculate errors between computed distances compared to ground_truth.

Parameters:

• anipose_io (dict) – Containing information to validate calibration in comparison with ground truth, such as “bodyparts”.

• ground_truth_distances – Distances defined in ground truth as returned by load_distances_from_ground_truth.

Returns:

anipose_io – Added “distance_errors_in_cm”.

Return type:

dict

core.angles_and_distances.add_reprojection_errors_of_all_calibration_validation_markers(anipose_io: Dict, df_xyz: DataFrame) → Dict

core.angles_and_distances.get_xyz_distance_in_triangulation_space(marker_ids: Tuple[str, str], df_xyz: DataFrame) → Series | float

Calculate the distance between two markers in 3D.

Parameters:

• marker_ids (tuple of str) – The two marker_ids to calculate the distance between.

• df_xyz (pd.DataFrame) – DataFrame of triangulated data.

Returns:

Distance between the two markers. float, if only one frame, else pd.Series.

Return type:

pd.Series or float

core.angles_and_distances.load_distances_from_ground_truth(distances: Dict) → Dict

Return distances from ground_truth_config[“distances”].

core.angles_and_distances.set_distances_and_angles_for_evaluation(parameters: Dict, anipose_io: Dict, df_xyz: DataFrame) → Dict

Compute angles and distances and add them to anipose_io.

Parameters:

• parameters (dict) – ground_truth_config

• anipose_io (dict) – Containing information to validate calibration in comparison with ground truth.

• df_xyz – DataFrame of triangulated data.

Returns:

anipose_io – Added “distances_in_cm” and “computed_angles”.

Return type:

dict

See also

core.utils.KEYS_TO_CHECK_PROJECT

core.camera_intrinsics module

class core.camera_intrinsics.IntrinsicCalibratorFisheyeCamera(filepath_calibration_video: Path | str, max_calibration_frames: int)

Bases: IntrinsicCameraCalibratorCheckerboard

class core.camera_intrinsics.IntrinsicCalibratorRegularCameraCheckerboard(filepath_calibration_video: Path | str, max_calibration_frames: int)

Bases: IntrinsicCameraCalibratorCheckerboard

class core.camera_intrinsics.IntrinsicCameraCalibrator(filepath_calibration_video: Path | str, max_calibration_frames: int)

Bases: ABC

Intrinsic calibration for fisheye and regular cameras on checkerboard videos.

Parameters:

• filepath_calibration_video (Path or str) – The filepath to the intrinsic calibration videos. They have to be recorded in same resolution as the recording/calibration videos without cropping using a 6x6 checkerboard..

• max_calibration_frames (int) – Number of frames to take into account for intrinsic calibration. 300 works well, depending on CPU speed, it can be necessary to reduce.

filepath_calibration_video

The filepath to the intrinsic calibration videos. They have to be recorded in same resolution as the recording/calibration videos without cropping using a 6x6 checkerboard..

Type:

Path or str

max_calibration_frames

Number of frames to take into account for intrinsic calibration. 300 works well, depending on CPU speed, it can be necessary to reduce.

Type:

int

video_reader

imageio Reader object of the calibration video.

Type:

Reader

checkerboard_rows_and_columns

d

Empty camera matrix.

Type:

np.ndarray

imsize

Size of the video.

Type:

tuple of ints

k

Empty camera matrix.

Type:

np.ndarray

objp

Empty object points. Shape as detected in one frame.

Type:

np.ndarray

run()

Detect board in frames and calibrate intrinsics.

References

[1] Kenneth Jiang (2017). Calibrate fisheye lens using OpenCV. medium.com (https://medium.com/@kennethjiang/calibrate-fisheye-lens-using-opencv-333b05afa0b0)

Copyright Kenneth Jiang. This class and its subclasses use code taken from [1]. Changes were made to match our needs here.

Examples

>>> from core.camera_intrinsics import IntrinsicCalibratorFisheyeCamera
>>> intrinsic_calibration_object = IntrinsicCalibratorFisheyeCamera(
    ... "test_data/intrinsic_calibrations/Bottom_checkerboard.mp4",
    ... 100)
>>> intrinsic_calibration_object.run()

property checkerboard_rows_and_columns: Tuple[int, int]

property d: ndarray

property imsize: Tuple[int, int]

property k: ndarray

property objp: ndarray

run() → Dict

Detect board in frames and calibrate intrinsics.

Returns:

calibration_results – Intrinsic calibration results containing camera matrix and distorsion coefficient at keys ‘K’ and ‘D’.

Return type:

dict

class core.camera_intrinsics.IntrinsicCameraCalibratorCheckerboard(filepath_calibration_video: Path | str, max_calibration_frames: int)

Bases: IntrinsicCameraCalibrator, ABC

core.checker_objects module

class core.checker_objects.CheckCalibration(calibration_directory: Path | str, project_config_filepath: Path | str, recording_config_filepath: Path | str, plot: bool = True)

Bases: object

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

Parameters:

• calibration_directory (Path or str) – Directory, where the calibration videos are stored.

• project_config_filepath (Path or str) – Filepath to the project_config .yaml file.

• recording_config_filepath (Path or str) – Filepath to the recording_config .yaml file.

• plot (bool, default True) – If True (default), then the undistorted images are plotted.

calibration_index

Index of a calibration. Together with recording_date, it creates a unique calibration key.

Type:

int

recording_date

Date at which the calibration was done.

Type:

str

See also

CheckRecording

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

CheckCalibrationValidation

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

core.triangulation_calibration_module.Calibration

A class, in which videos are calibrated to each other.

core.filename_checker.FilenameCheckerInterface.create_calibrations

Create CheckCalibration and CheckCalibrationValidation objects for all calibration_directories added to FilenameCheckerInterface.

class core.checker_objects.CheckCalibrationValidation(calibration_validation_directory: Path | str, recording_config_filepath: Path | str, ground_truth_config_filepath: Path | str, project_config_filepath: Path | str, plot: bool = True)

Bases: object

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

Parameters:

• calibration_validation_directory (Path or str) – Directory, where the calibration_validation iamges are stored.

• recording_config_filepath (Path or str) – Filepath to the recording_config .yaml file.

• ground_truth_config_filepath (Path or str) – The path to the ground_truth config file.

• project_config_filepath (Path or str) – Filepath to the project_config .yaml file.

• plot (bool, default True) – If True (default), then the undistorted images are plotted.

calibration_index

Index of a calibration. Together with recording_date, it creates a unique calibration key.

Type:

int

recording_date

Date at which the calibration was done.

Type:

str

See also

CheckCalibration

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

CheckRecording

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

core.triangulation_calibration_module.CalibrationValidation

A class, in which images are triangulated based on a calibration file and the triangulated coordinates are validated based on a ground_truth.

core.filename_checker.FilenameCheckerInterface.create_calibrations

Create CheckCalibration and CheckCalibrationValidation objects for all calibration_directories added to FilenameCheckerInterface.

class core.checker_objects.CheckRecording(recording_directory: Path | str, recording_config_filepath: Path | str, project_config_filepath: Path | str, plot: bool = False)

Bases: object

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

Parameters:

• recording_directory (Path or str) – Directory, where the recording videos are stored.

• recording_config_filepath (Path or str) – Filepath to the recording_config .yaml file.

• project_config_filepath (Path or str) – Filepath to the project_config .yaml file.

• plot (bool, default True) – If True (default), then the undistorted images are plotted.

calibration_index

Index of a calibration. Together with recording_date, it creates a unique calibration key.

Type:

int

recording_date

Date at which the calibration was done.

Type:

str

mouse_id

The mouse_id as read from the filenames.

Type:

str

paradigm

The paradigm as read from the filenames.

Type:

str

See also

CheckCalibration

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

CheckCalibrationValidation

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

core.triangulation_calibration_module.TriangulationRecordings

Subclass of Triangulation, in which videos are triangulated based on a calibration file.

core.filename_checker.FilenameCheckerInterface.create_recordings

Create CheckRecording objects for all recording_directories added to FilenameCheckerInterface.

core.filename_checker module

class core.filename_checker.FilenameCheckerInterface(project_config_filepath: Path | str)

Bases: object

Interface to load all files and check filename and metadata.

Parameters:

project_config_filepath (Path or str) – Filepath to the project_config .yaml file.

objects

Dictionary of all objects added to the FilenameCheckerInterface.

Type:

dict

project_config_filepath

Filepath to the project_config .yaml file.

Type:

Path

paradigms

List of all paradigms to search for in directories.

Type:

list of str

recording_configs

List of all recording_configs added to the FilenameCheckerInterface.

Type:

list of Path

recording_dates

List of all recording_dates to search for in directories.

Type:

list of str

meta

Dictionary of metadata of all objects added to the FilenameCheckerInterface.

Type:

dict

select_recording_configs():

Open a window to select recording_config files in filedialog.

add_recording_config(filepath_to_recording_config)

Add recording_config file via method.

initialize_meta_config():

Append all directories to metadata, that match appended paradigms and recording_dates in directory name.

add_recording_manually(file, recording_day):

Adds recordings to metadata that don’t match directory name structure.

remove_recordings():

Remove recordings from analysis via user input dialog.

create_recordings():

Create CheckRecording objects for all recording_directories added to FilenameCheckerInterface.

create_calibrations(ground_truth_config_filepath):

Create CheckCalibration and CheckCalibrationValidation objects for all calibration_directories added to FilenameCheckerInterface.

See also

core.meta.MetaInterface

Interface to load all files and run analysis.

core.checker_objects.CheckRecording

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

core.checker_objects.CheckCalibration

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

core.checker_objects.CheckCalibrationValidation

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

Examples

>>> from core.filename_checker import FilenameCheckerInterface
>>> filename_checker = FilenameCheckerInterface(project_config_filepath="test_data/project_config.yaml")
>>> filename_checker.add_recording_config("test_data/Server_structure/Calibrations/220922/recording_config_220922.yaml")
>>> filename_checker.initialize_meta_config()
>>> filename_checker.create_recordings()
>>> filename_checker.create_calibrations(ground_truth_config_filepath="test_data/ground_truth_config.yaml")

add_recording_config(filepath_to_recording_config: Path | str) → None

Add recording_config via method.

Parameters:

filepath_to_recording_config (Path or str) – The path to the recording_config, that should be added to the MetaInterface.

add_recording_manually(file: Path, recording_day: str) → None

Adds recordings to metadata that don’t match directory name structure.

Parameters:

• file (Path or str) – The path to the recording directory, that should be added.

• recording_day (str) – The date of the recording.

create_calibrations(ground_truth_config_filepath: Path) → None

Create CheckCalibration and CheckCalibrationValidation objects for all calibration_directories added to FilenameCheckerInterface.

Parameters:

ground_truth_config_filepath – The path to the ground_truth config file.

create_recordings() → None

Create CheckRecording objects for all recording_directories added to FilenameCheckerInterface.

initialize_meta_config(num_recording_config_parents: int = 2) → None

Append all directories to metadata, that match appended paradigms and recording_dates in directory name.

Parameters:

num_recording_config_parents: int, default 2

The number of levels above the recording config file to look for matching recording directories.

See also

FilenameCheckerInterface.add_recording_manually

Adds recordings to metadata that don’t match directory name structure.

Notes

Demands for adding directories automatically:

• recording directory name has to start with a recording date

(YYMMDD) that is added to the FilenameCheckerInterface - recording directory name has to end with any of the paradigms (as defined in project_config)

If you want to add recording directories, that don’t match this structure, use FilenameCheckerInterface.add_recording_manually instead.

remove_recordings() → None

select_recording_configs() → None

Open a window to select recording_config files in filedialog.

Add it to recording_configs.

core.marker_detection module

class core.marker_detection.DeeplabcutInterface(object_to_analyse: Path | str, output_directory: Path | str, marker_detection_directory: Path | str | None = None)

Bases: MarkerDetection

analyze_objects(filepath: Path, filtering: bool = False, use_gpu: str = '', **kwargs) → Path

class core.marker_detection.ManualAnnotation(object_to_analyse: Path | str, output_directory: Path | str, marker_detection_directory: Path | str | None = None)

Bases: MarkerDetection

analyze_objects(filepath: Path, labels: List[str] | None = None, only_first_frame: bool = False, **kwargs) → Path

class core.marker_detection.MarkerDetection(object_to_analyse: Path | str, output_directory: Path | str, marker_detection_directory: Path | str | None = None)

Bases: ABC

Class to run marker detection using different marker detection methods.

Parameters:

• object_to_analyse (Path or str) – The path to the video to be analysed.

• output_directory (Path or str) – The directory, in which the output file will be stored.

• marker_detection_directory (Path or str, optional) – The filepath to the config file to use for marker detection. E.g., the DLC project config file. None, for manual Marker Detection.

object_to_analyse

The path to the video to be analysed.

Type:

Path

output_directory

The directory, in which the output file will be stored.

Type:

Path

marker_detection_directory

The filepath to the config file to use for marker detection. E.g., the DLC project config file. None, for manual Marker Detection.

Type:

Path

analyze_objects(filepath, labels, only_first_frame, filtering, use_gpu):

Abstract method for subclasses to be implemented using the corresponding marker detection method.

abstract analyze_objects(filepath: Path, labels: List[str] | None = None, only_first_frame: bool = False, filtering: bool = False, use_gpu: str = '') → Path

core.meta module

class core.meta.MetaInterface(project_config_filepath: Path | str, project_name: str | None = None, overwrite: bool = False)

Bases: ABC

Interface to load all files and run analysis.

Run (optimised) calibrations, triangulation of recordings, create Database and save/load the whole project to/from meta .yaml-file.

Parameters:

• project_config_filepath (Path or str) – Filepath to the project_config .yaml file.

• project_name (str, optional) – The name of the meta .yaml-file path.

• overwrite (bool, default False) – If True (default False), then the meta.yaml will be overwritten if already existing.

objects

Dictionary of all objects added to the MetaInterface.

Type:

dict

project_config_filepath

Filepath to the project_config .yaml file.

Type:

Path

paradigms

List of all paradigms to search for in directories.

Type:

list of str

recording_configs

List of all recording_configs added to the MetaInterface.

Type:

list of Path

recording_dates

List of all recording_dates to search for in directories.

Type:

list of str

meta

Dictionary of metadata of all objects added to the MetaInterface.

Type:

dict

project_name

The name of the meta .yaml-file. Default is ‘My_project’.

Type:

str

standard_yaml_filepath

The filepath to the meta .yaml-file. Stored in the same directory as the project_config.

Type:

Path

select_recording_configs():

Open a window to select recording_config files in filedialog.

add_recording_config(filepath_to_recording_config)

Add recording_config file via method.

initialize_meta_config():

Append all directories to metadata, that match appended paradigms and recording_dates in directory name.

add_recording_manually(file, recording_day):

Adds recordings to metadata that don’t match directory name structure.

remove_recordings():

Remove recordings from analysis via user input dialog.

create_recordings(recreate_undistorted_plots):

Create TriangulationRecording objects for all recording_directories added to MetaInterface.

synchronize_recordings(verbose, overwrite_DLC_analysis_and_synchro):

Run the function run_synchronization for all TriangulationRecording objects added to MetaInterface.

create_calibrations(ground_truth_config_filepath, recreate_undistorted_plots):

Create Calibration and CalibrationValidation objects and add ground_truth_config for all calibration_directories added to MetaInterface.

synchronize_calibrations(overwrite_synchronisations_and_calvin_predictions):

Run get_marker_predictions for all calibration_validation objects and run_synchronization for all calibration objects added to MetaInterface.

calibrate(p_threshold, angle_threshold, max_iters, calibrate_optimal, verbose, overwrite_calibrations):

Run the function run_calibration or calibrate_optimal for all calibration objects added to MetaInterface.

triangulate_recordings(triangulate_full_recording):

Run the function run_triangulation for all TriangulationRecording objects added to MetaInterface.

exclude_markers(all_markers_to_exclude_config_path, verbose):

Run the function exclude_marker for all TriangulationRecordings and CalibrationValidation objects added to MetaInterface.

normalize_recordings(normalization_config_path, save_dataframe):

Run the function normalize for all TriangulationRecordings objects and saves the normalisation metadata.

add_triangulated_csv_to_database(data_base_path, overwrite):

Add the 3D dataframes to a common data_base.

export_meta_to_yaml(filepath):

Store MetaInterface objects as .yaml-file.

See also

core.filename_checker.FilenameCheckerInterface

Interface to load all files and check filename and metadata.

core.triangulation_calibration_module.TriangulationRecordings

A class, in which videos are triangulated based on a calibration file.

core.triangulation_calibration_module.CalibrationValidation

A class, in which images are triangulated based on a calibration file and the triangulated coordinates are validated based on a ground_truth.

core.triangulation_calibration_module.Calibration

A class, in which videos are calibrated to each other.

Examples

>>> from core.meta import MetaInterface
>>> meta_interface = MetaInterface(
    ... project_config_filepath="test_data/project_config.yaml",
    ... project_name="test_data", overwrite=False)
>>> meta_interface.add_recording_config("test_data/Server_structure/Calibrations/220922/recording_config_220922.yaml")
>>> meta_interface.initialize_meta_config()
>>> meta_interface.create_recordings()
>>> meta_interface.synchronize_recordings(verbose=True)
>>> meta_interface.create_calibrations(ground_truth_config_filepath="test_data/ground_truth_config_only_corners.yaml")
>>> meta_interface.synchronize_calibrations()
>>> meta_interface.exclude_markers(all_markers_to_exclude_config_path = "test_data/markers_to_exclude_config.yaml", verbose=False)
>>> meta_interface.calibrate(calibrate_optimal=True, verbose=2)
>>> meta_interface.triangulate_recordings()
>>> meta_interface.normalize_recordings(normalization_config_path="test_data/normalization_config.yaml")

add_recording_config(filepath_to_recording_config: str | Path) → None

Add recording_config via method.

Parameters:

filepath_to_recording_config (Path or str) – The path to the recording_config, that should be added to the MetaInterface.

Raises:

FileNotFoundError: – If the path is not linked to a .yaml file or doesn’t exist.

add_recording_manually(file: Path | str, recording_day: str) → None

Adds recordings to metadata that don’t match directory name structure.

Parameters:

• file (Path or str) – The path to the recording directory, that should be added.

• recording_day (str) – The date of the recording.

Raises:

FileNotFoundError: – If the path is no directory or if there’s no recording_config added for the recording_day.

add_triangulated_csv_to_database(data_base_path: str | Path, overwrite: bool = True) → None

Add the 3D dataframes to a common data_base.

Parameters:

• data_base_path (str or Path) – The path to the data_base, to which the 3D df metadata will be added.

• overwrite (bool, default True) – If True (default), then metadata for recordings in the MetaInterface, that were already added to the data_base, will be overwritten.

calibrate(p_threshold: float = 0.1, angle_threshold: float = 5.0, max_iters: int = 5, calibrate_optimal: bool = True, verbose: int = 1, overwrite_calibrations: bool = False) → None

Run the function run_calibration or calibrate_optimal for all calibration objects added to MetaInterface.

Parameters:

• p_threshold (float, default 0.1) – Threshold for errors in the triangulated distances compared to ground truth (mean distances in percent). Won’t be used if calibrate_optimal is False.

• angle_threshold (float, default 5) – Threshold for errors in the triangulated angles compared to ground truth (mean angles in degrees). Won’t be used if calibrate_optimal is False.

• max_iters (int, default 5) – Number of iterations allowed to find a good calibration. Won’t be used if calibrate_optimal is False.

• calibrate_optimal (bool, default True) – If True (default), then calibrate_optimal will be run for all calibration objects added to MetaInterface. If False, then run_calibration will be run.

• verbose (int, default 1) – Show ap_lib output if > 1, calibration_validation output if > 0 or no output if < 1.

• overwrite_calibrations (bool, default False) – If True (default False), then pre-existing calibrations will be overwritten.

create_calibrations(ground_truth_config_filepath: Path, recreate_undistorted_plots: bool = True) → None

Create Calibration and CalibrationValidation objects and add ground_truth_config for all calibration_directories added to MetaInterface.

Parameters:

• ground_truth_config_filepath (Path or str) – The path to the ground_truth config file.

• recreate_undistorted_plots – If True (default), then preexisting undistorted plots will be overwritten.

create_recordings(recreate_undistorted_plots: bool = True, specify_calibration_to_use: bool = False) → None

Create TriangulationRecording objects for all recording_directories added to MetaInterface.

Parameters:

• recreate_undistorted_plots – If True (default), then preexisting undistorted plots will be overwritten.

• specify_calibration_to_use (bool, default False) – If True (default False), then you will be asked to specify the calibration index for each recording.

exclude_markers(all_markers_to_exclude_config_path: Path | str, verbose: bool = True) → None

Run the function exclude_marker for all TriangulationRecordings and CalibrationValidation objects added to MetaInterface.

Parameters:

• all_markers_to_exclude_config_path (Path or str) – Filepath to the config used for exclusion of markers.

• verbose (bool, default True) – If True (default), print if exclusion of markers worked without any abnormalities.

export_meta_to_yaml(filepath: str | Path) → None

Store MetaInterface objects as .yaml-file.

Parameters:

filepath (str or Path) – The path, where the meta .yaml-file should be saved.

initialize_meta_config(num_recording_config_parents: int = 2) → None

Append all directories to metadata, that match appended paradigms and recording_dates in directory name.

Parameters:

num_recording_config_parents: int, default 2

The number of levels above the recording config file to look for matching recording directories.

See also

MetaInterface.add_recording_manually

Adds recordings to metadata that don’t match directory name structure.

Notes

Demands for adding directories automatically:

• recording directory name has to start with a recording date (YYMMDD) that is added to the MetaInterface

• recording directory name has to end with any of the paradigms (as defined in project_config)

If you want to add recording directories, that don’t match this structure, use MetaInterface.add_recording_manually.

normalize_recordings(normalization_config_path: Path | str, save_dataframe: bool = True, verbose: bool = False) → None

Run the function normalize for all TriangulationRecordings objects and saves the normalisation metadata.

Parameters:

• normalization_config_path (Path or str) – The path to the config used for normalisation.

• save_dataframe (bool, default True) – If True (default), then the dataframe will be saved and overwrites the pre-existing one.

• verbose (bool, default False) – If True (default False), then the rotation visualization plot is shown.

remove_recordings() → None

Remove recordings from analysis via user input dialog.

select_recording_configs() → None

Open a window to select recording_config files in filedialog.

Add it to recording_configs.

synchronize_calibrations(overwrite_synchronisations_and_calvin_predictions: bool = False, verbose: bool = True) → None

Run get_marker_predictions for all calibration_validation objects and run_synchronization for all calibration objects added to MetaInterface.

Parameters:

• overwrite_synchronisations_and_calvin_predictions (bool, default False) – If True (default False), then pre-existing synchronisations and calvin predictions will be overwritten during analysis.

• verbose (bool, default True) – If True (default), then the attribute is passed to the Calibration objects.

synchronize_recordings(verbose: bool = True, overwrite_DLC_analysis_and_synchro: bool = False) → None

Run the function run_synchronization for all TriangulationRecording objects added to MetaInterface.

Parameters:

• verbose (bool, default True:) – If True (default), then the duration of a analysis is printed and the attribute is passed to the TriangulationRecordings objects.

• overwrite_DLC_analysis_and_synchro (bool, default False) – If True (default False), then pre-existing DLC files and synchronisations will be overwritten during analysis.

triangulate_recordings(triangulate_full_recording: bool = True, verbose: bool = True, use_preexisting_csvs: bool = False) → None

Run the function run_triangulation for all TriangulationRecording objects added to MetaInterface.

Parameters:

• triangulate_full_recording (bool, default True) – If False (default True), then only the first 2 frames of the recording will be triangulated and the 3D dataframe won’t be saved.

• verbose (bool, default True) – If True (default), then the recording, that is currently analysed, and the duration of an analysis will be printed.

• use_preexisting_csvs (bool, default False) – If True (default False), then a already existing file at csv_output_filepath will be read in and no triangulatin will be performed.

core.plotting module

class core.plotting.AlignmentPlotCrossvalidation(template: ndarray, led_timeseries: Dict, filename: str = '', output_directory: Path | str | None = None)

Bases: object

create_plot(plot: bool, save: bool)

class core.plotting.AlignmentPlotIndividual(template: ndarray, led_timeseries: ndarray, filename: str = '', cam_id: str = '', output_directory: Path | str | None = None, led_box_size: int | None = None, alignment_error: int | None = None)

Bases: object

create_plot(plot: bool, save: bool) → None

class core.plotting.CalibrationValidationPlot(p3d: Dict, bodyparts: List[str], output_directory: Path | str | None = None, marker_ids_to_connect: List[str] = [], filename_tag: str = '')

Bases: object

create_plot(plot: bool, save: bool) → None

class core.plotting.Intrinsics(video_filepath: Path | str, intrinsic_calibration: Dict, filename: str = '', fisheye: bool = False, output_directory: Path | str | None = None)

Bases: object

create_plot(plot: bool, save: bool) → None

class core.plotting.LEDMarkerPlot(image: ndarray, led_center_coordinates: Coordinates, box_size: int | None = None, cam_id: str = '', filename: str = '', output_directory: Path | str | None = None)

Bases: object

create_plot(plot: bool, save: bool)

class core.plotting.PredictionsPlot(image: Path, predictions: Path, cam_id: str = '', output_directory: Path | str | None = None, likelihood_threshold: float = 0.6)

Bases: object

create_plot(plot: bool, save: bool) → None

class core.plotting.RotationVisualization(rotated_markers: List, config: Dict, rotation_error: float, output_filepath: Path | str | None = None)

Bases: object

create_plot(plot: bool, save: bool) → None

class core.plotting.TriangulationVisualization(df_3D_filepath: Path, config: Dict, filename_tag: str = '', output_directory: Path | None = None)

Bases: object

create_plot(plot: bool, save: bool, idx: int, return_fig: bool = False) → ndarray | None

return_fig(idx: int) → ndarray

core.triangulation_calibration_module module

class core.triangulation_calibration_module.Calibration(calibration_directory: Path | str, project_config_filepath: Path | str, recording_config_filepath: Path | str, output_directory: Path | str | None = None, recreate_undistorted_plots: bool = True)

Bases: object

A class, in which videos are calibrated to each other.

Temporal synchronization of the videos can be performed based on a pattern. Spatial calibration is performed using aniposelib (ap_lib) with additional methods to validate the calibration based on known ground_truth.

Parameters:

• calibration_directory (Path or string) – Directory, where the calibration videos are stored.

• project_config_filepath (Path or string) – Filepath to the project_config .yaml file.

• recording_config_filepath (Path or string) – Filepath to the recording_config .yaml file.

• output_directory (Path or string, optional) – Directory, in which the files created during the analysis are saved. Per default it will be set the same as the calibration_directory.

• recreate_undistorted_plots (bool, default True) – If True (default), then preexisting undistorted plots will be overwritten.

project_config_filepath

Filepath to the project_config .yaml file.

Type:

Path

output_directory

Directory, in which the files created during the analysis are saved.

Type:

Path

camera_group

Group of ap_lib.cameras.Camera objects.

Type:

ap_lib.cameras.CameraGroup

camera_objects

List of ap_lib.cameras.Camera objects.

synchronized_charuco_videofiles

Dict of synchronized calibration video per camera.

Type:

{str: Path}

video_interfaces

Dict of VideoInterface objects for all calibration videos.

Type:

{str: VideoInterface}

metadata_from_videos

Dict of VideoMetadata objects for all calibration videos.

Type:

{str: VideoMetadata}

valid_videos

Videos, that were found in the calibration_directory and not excluded due to synchronization issues.

Type:

list of str

cams_to_exclude

Videos, that were excluded due to synchronization issues.

Type:

list of str

recording_date

Date at which the calibration was done.

Type:

str

target_fps

Fps rate, to which the videos should be synchronized.

Type:

int

led_pattern

Blinking pattern to use for temporal synchronisation.

Type:

dict

calibration_index

Index of a calibration. Together with recording_date, it creates a unique calibration key.

Type:

int

calibration_tag

Filename tag to search for in the files in calibration_directory.

Type:

str

reprojerr

Reprojection error (px) returned by ap_lib calibration.

Type:

float

report_filepath

Filepath to the report .csv for calibration optimisation.

Type:

Path

allowed_num_diverging_frames

Difference of framenumber to the framenumber median of all synchronised videos, that is allowed before a video has to be excluded.

Type:

int

synchro_metadata

Dictionary used as input for synchronizer objects.

Type:

dict

synchronization_individuals

Container for the AlignmentPlotIndividual objects of each synchronized video.

Type:

list of AlignmentPlotIndividual

led_detection_individuals

Container for the LEDMarkerPlot objects of each synchronized video.

Type:

list of LEDMarkerPlot

run_synchronization(overwrite_synchronisations, verbose)

Perform synchronization of all videos to the led_pattern and downsampling to target_fps.

run_calibration(use_own_intrinsic_calibration, charuco_calibration_board, iteration, verbose, overwrite_calibrations)

Call ap_lib calibrate function.

calibrate_optimal(calibration_validation, max_iters, p_threshold, angle_threshold, verbose, overwrite_calibrations)

Call run_calibration repeatedly and validate the quality of the resulting calibration on calibration_validation images and ground_truth.

References

[1] Karashchuk, P., Rupp, K. L., Dickinson, E. S., et al. (2021). Anipose: A toolkit for robust markerless 3D pose estimation. Cell reports, 36(13), 109730. https://doi.org/10.1016/j.celrep.2021.109730

See also

TriangulationRecordings

A class, in which videos are triangulated based on a calibration file.

CalibrationValidation

A class, in which images are triangulated based on a calibration file and the triangulated coordinates are validated based on a ground_truth.

core.checker_objects.CheckCalibration

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

core.meta.MetaInterface.create_calibrations

Create Calibration objects for all calibration_directories added to MetaInterface.

core.meta.MetaInterface.synchronize_calibrations

Run the function run_synchronization for all calibration objects added to MetaInterface.

core.meta.MetaInterface.calibrate

Run the function run_calibration or calibrate_optimal for all calibration objects added to MetaInterface.

Examples

>>> from pathlib import Path
>>> from core.triangulation_calibration_module import Calibration
>>> rec_config = Path(
... "test_data/Server_structure/Calibrations/220922/recording_config_220922.yaml"
... )
>>> calibration_object = Calibration(
... calibration_directory=rec_config.parent,
... recording_config_filepath=rec_config,
... project_config_filepath="test_data/project_config.yaml",
... output_directory=rec_config.parent,
... )
>>> calibration_object.run_synchronization()
>>> calibration_object.run_calibration(verbose=2)

calibrate_optimal(calibration_validation: CalibrationValidation, max_iters: int = 5, p_threshold: float = 0.1, angle_threshold: float = 5.0, verbose: int = 1, overwrite_calibrations: bool = True)

Call run_calibration repeatedly and validates the quality of the resulting calibration on calibration_validation images and ground_truth.

Repeat calibration until a good calibration is reached or max_iters is superceded. Check whether the triangulated data in calibration_validation matches the ground_truth data p_threshold and angle_threshold. Create a report file in which the metadata to the calibration of each iteration is specified.

Parameters:

• calibration_validation (CalibrationValidation) – Object, containing images, triangulated data and ground_truth information for calibration validation.

• max_iters (int, default 5) – Number of iterations allowed to find a good calibration.

• p_threshold (float, default 0.1) – Threshold for errors in the triangulated distances compared to ground truth (mean distances in percent).

• angle_threshold (float, default 5) – Threshold for errors in the triangulated angles compared to ground truth (mean angles in degrees).

• verbose (int, default 1) – Show ap_lib output if > 1, calibration_validation output if > 0 or no output if < 1.

• overwrite_calibrations (bool, default True) – If True (default), then pre-existing calibrations will be overwritten.

Returns:

calibration_filepath – The filepath to the optimal calibration of if no good calibration was reached during iteration, the filepath of the last calibration.

Return type:

Path

run_calibration(use_own_intrinsic_calibration: bool = True, verbose: int = 0, charuco_calibration_board: CharucoBoard | None = None, overwrite_calibrations: bool = True, iteration: int | None = None) → Path

Call ap_lib calibrate function.

Create a filename for the calibration file. Pass videos to ap_lib.cameras.camera_group.calibrate_videos function.

Parameters:

• use_own_intrinsic_calibration (bool, default True) – If True (default), then the externally created intrinsic calibrations are passed to the calibrate function. Otherwise, the ap_lib built-in intrinsic calibration is used.

• verbose (int, default 0) – Show ap_lib output if > 1 or no output if <= 1.

• charuco_calibration_board (ap_lib.boards.CharucoBoard, optional) – Specify the board, that was used in the calibration videos.

• overwrite_calibrations (bool, default True) – If True (default), then pre-existing calibrations will be overwritten.

• iteration (int, optional) – Variable to be included into the filename to make the filepath of calibration files unique for repeated calibrations.

Returns:

calibration_filepath

Return type:

Path

run_synchronization(overwrite_synchronisations: bool = False, verbose: bool = True) → None

Perform synchronization of all videos to the led_pattern and downsampling to target_fps.

Call the synchronizer via VideoInterface and save the synchronized_video_filepaths. Create a plot for crossvalidation of the synchronised LED timeseries. Exclude videos, if there are any duplicates in camera names or diverging framenumbers after synchronization.

Parameters:

• overwrite_synchronisations (bool, default False) – If True (default False), then pre-existing synchronisations will be overwritten during analysis.

• verbose (bool, default True) – If True (default), then Crossvalidation plot and synchronised number of frames for each camera are printed.

class core.triangulation_calibration_module.CalibrationValidation(project_config_filepath: Path | str, directory: Path | str, recording_config_filepath: Path | str, recreate_undistorted_plots: bool = True, output_directory: Path | str | None = None)

Bases: Triangulation

Subclass of Triangulation, in which images are triangulated based on a calibration file and the triangulated coordinates are validated based on a ground_truth.

add_ground_truth_config(ground_truth_config_filepath)

Read the metadata from ground_truth_config_filepath and create list of markers.

get_marker_predictions(overwrite_analysed_markers)

Run marker detection for all images in metadata_from_videos.

evaluate_triangulation_of_calibration_validation_markers(show_3D_plot, verbose)

Evaluate the triangulated data and return mean errors.

See also

Triangulation

Parent class, for triangulation of videos or images.

core.meta.MetaInterface.create_calibrations

Create CalibrationValidation objects and run add_ground_truth_config for all calibration_directories added to MetaInterface.

core.meta.MetaInterface.synchronize_calibrations

Run get_marker_predictions for all calibration_validation objects added to MetaInterface.

core.checker_objects.CheckCalibrationValidation

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

Calibration.triangulate_optim

Run the function run_triangulation for the CalibrationValidation object passed to triangulate_optim.

Examples

>>> from core.triangulation_calibration_module import CalibrationValidation
>>> from pathlib import Path
>>> rec_config = Path("test_data/Server_structure/Calibrations/220922/recording_config_220922.yaml")
>>> calibration_validation_object = CalibrationValidation(
    ... project_config_filepath="test_data/project_config.yaml",
    ... directory=rec_config.parent, recording_config_filepath=rec_config,
    ... recreate_undistorted_plots = True, output_directory=rec_config.parent)
>>> calibration_validation_object.add_ground_truth_config("test_data/ground_truth_config.yaml")
>>> calibration_validation_object.get_marker_predictions()
>>> calibration_validation_object.run_triangulation(
    ... calibration_toml_filepath="test_data/Server_structure/Calibrations/220922/220922_0_Bottom_Ground1_Ground2_Side1_Side2_Side3.toml",
    ... triangulate_full_recording = True)
>>> mean_dist_err_percentage, mean_angle_err, reprojerr_nonan_mean = calibration_validation_object.evaluate_triangulation_of_calibration_validation_markers()

add_ground_truth_config(ground_truth_config_filepath: Path | str) → None

Read the metadata from ground_truth_config_filepath and create list of markers.

Parameters:

ground_truth_config_filepath (str or Path) – The path to the ground_truth config file.

Notes

The ground_truth yaml file at ground_truth_config_filepath has to have the following structure:

distances: {str: {str: float}}

Dictionary with first markers as key and dictionaries as values, that have second markers as key and the known distances between first and second markers as values. Distances are floats in cm.

angles:

Dictionary with vertex markers as keys and dictionaries as values, that have the following keys:

value: float

The value of the calculated angle in degrees.

marker: list of str

The markers between that draw the triangle (if 3 markers given: 0: vertex, 1/2: ray) or a plane and a line (if 5 markers given: 1/2/3: plane, 4/5: line).

unique_ids: list of str

A list with all marker_ids to take into account for ground_truth validation and plotting.

marker_ids_to_connect_in_3D_plot: list of list of str, optional

Each element consists of a list of markers, that will be connected in the 3D calibration validation plot.

evaluate_triangulation_of_calibration_validation_markers(show_3D_plot: bool = True, verbose: bool = True) → Tuple[float64, float64, float64]

Evaluate the triangulated data and return mean errors.

Calculate the distances and angles for all references in ground truth and get the differences between triangulated and ground truth data. Print these differences and show the plot of the triangulated data. Calculate the mean of distance and angle error and reprojection error.

Parameters:

• show_3D_plot (bool, default True) – If True (default), then a plot of the triangulated calibration_validation data is shown.

• verbose (bool, default True) – If True (default), then all angles and distances compared to their ground truth will be printed.

Returns:

• mean_dist_err_percentage (np.float64) – The mean error of all triangulated distances compared to their ground truth.

• mean_angle_err (np.float64) – The mean error of all triangulated errors compared to their ground truth.

• reprojerr_nonan_mean (np.float64) – The mean reprojection error of all triangulated points.

Notes

The path directing to the ground_truth yaml file, that is saved as self.ground_truth_config, has to have the following structure:

distances: {str: {str: float}}

Dictionary with first markers as key and dictionaries as values, that have second markers as key and the known distances between first and second markers as values. Distances are floats in cm.

angles:

Dictionary with vertex markers as keys and dictionaries as values, that have the following keys:

value: float

The value of the calculated angle in degrees.

marker: list of str

The markers between that draw the triangle (if 3 markers given: 0: vertex, 1/2: ray) or a plane and a line (if 5 markers given: 1/2/3: plane, 4/5: line).

unique_ids: list of str

A list with all marker_ids to take into account for ground_truth validation and plotting.

marker_ids_to_connect_in_3D_plot: list of list of str, optional

Each element consists of a list of markers, that will be connected in the 3D calibration validation plot.

get_marker_predictions(overwrite_analysed_markers: bool = False) → None

Run marker detection for all images in metadata_from_videos.

Save predictions in triangulation_dlc_cams_filepaths, validate predictions and create predictions plots.

Parameters:

overwrite_analysed_markers (bool, default False) – If True (default False), then pre-existing files won’t be overwritten during the analysis.

class core.triangulation_calibration_module.Triangulation(project_config_filepath: Path | str, directory: Path | str, recording_config_filepath: Path | str, recreate_undistorted_plots: bool = True, output_directory: Path | str | None = None)

Bases: ABC

Parent class, for triangulation of videos or images.

Triangulation is performed using aniposelib (ap_lib).

Parameters:

• project_config_filepath (Path or string) – Filepath to the project_config .yaml file.

• directory (Path or string) – Directory, where the videos or images are stored.

• recording_config_filepath (Path or string) – Filepath to the recording_config .yaml file.

• recreate_undistorted_plots (bool, default True) – If True (default), then preexisting undistorted plots will be overwritten.

• output_directory (Path or string, optional) – Directory, in which the files created during the analysis are saved. Per default it will be set the same as the directory.

project_config_filepath

Filepath to the project_config .yaml file.

Type:

Path

output_directory

Directory, in which the files created during the analysis are saved.

Type:

Path

video_interfaces

Dict of VideoInterface objects for all videos or images.

Type:

{str: VideoInterface}

metadata_from_videos

Dict of VideoMetadata objects for all videos or images.

Type:

{str: VideoMetadata}

triangulation_dlc_cams_filepaths

Containing the filepath to the predictions for each camera.

Type:

{str: Path}

csv_output_filepath

Filepath, were the triangulated dataframe should be saved.

Type:

Path

recording_date

Date at which the calibration was done based on recording_config and as read from the filenames.

Type:

str

calibration_index

Index of a calibration. Together with recording_date, it creates a unique calibration key.

Type:

int

target_fps

Fps rate, to which the videos should be synchronized.

Type:

int

led_pattern

Blinking pattern to use for temporal synchronisation.

Type:

dict

mouse_id

Only defined in TriangulationRecordings. The mouse_id as read from the filenames.

Type:

str

paradigm

Only defined in TriangulationRecordings. The paradigm as read from the filenames.

Type:

str

cams_to_exclude

Videos, that were excluded due to synchronization issues.

Type:

list of str

all_cameras

All camera names, that are stored in the camera_group.

Type:

list of str

self.camera_group

Group of cameras loaded from calibration_toml_filepath.

Type:

ap_lib.cameras.CameraGroup

anipose_io

Containing information for ap_lib functions, such as points_flat, p3ds, n_joints, reprojerr, as well as distances and angles to validate calibration in comparison with ground truth.

Type:

dict

markers

All markers that will be triangulated.

Type:

list of str

normalised_dataframe

If True (default False), then the dataframe was normalised based on input from normalisation config.

Type:

bool, default False

markers_excluded_manually

If True (default False), then markers were excluded from predictions based on marker exclusion config.

Type:

bool, default False

rotated_filepath

Filepath, were the rotated triangulated dataframe is saved.

Type:

Path

video_plotting_config

Containing information from video_plotting config to create 3D videos.

Type:

dict

synchronization_individuals

Only defined in TriangulationRecordings. Container for the AlignmentPlotIndividual objects of each synchronized video.

Type:

list of AlignmentPlotIndividual

led_detection_individuals

Only defined in TriangulationRecordings. Container for the LEDMarkerPlot objects of each synchronized video.

Type:

list of LEDMarkerPlot

synchro_metadata

Only used in TriangulationRecordings. Dictionary used as input for synchronizer objects.

Type:

dict

allowed_num_diverging_frames

Only used in TriangulationRecordings. Difference of framenumber to the framenumber median of all synchronised videos, that is allowed before a video has to be excluded.

Type:

int

ground_truth_config

Only defined in CalibrationValidation to compare the triangulated data to ground truth data.

Type:

dict

calibration_validation_tag

Only used in CalibrationValidation. Filename tag to search for in the filenames in directory.

Type:

str

_allowed_filetypes

Abstract property, specify what file endings to look for in directory.

Type:

list of str

triangulation_visualization

Object to create plots for triangulation video.

Type:

TriangulationVisualization

video_start_s

The second, in which the triangulation video starts.

Type:

int

run_triangulation(calibration_toml_filepath, triangulate_full_recording):

Load and validate the calibration, triangulate and create 3D df.

exclude_marker(all_markers_to_exclude_config_path, verbose):

Exclude markers in prediction based on markers_to_exclude config.

_get_dataframe_of_triangulated_points(anipose_io):

Combine the triangulated data from anipose_io to a 3D df.

References

[1] Karashchuk, P., Rupp, K. L., Dickinson, E. S., et al. (2021). Anipose: A toolkit for robust markerless 3D pose estimation. Cell reports, 36(13), 109730. https://doi.org/10.1016/j.celrep.2021.109730

See also

TriangulationRecordings

Subclass of Triangulation, in which videos are triangulated based on a calibration file.

CalibrationValidation

Subclass of Triangulation, in which images are triangulated based on a calibration file and the triangulated coordinates are validated based on a ground_truth.

Calibration

A class, in which videos are calibrated to each other.

core.meta.MetaInterface.triangulate_recordings

Run the function run_triangulation for all TriangulationRecording objects added to MetaInterface.

Calibration.triangulate_optim

Run the function run_triangulation for the CalibrationValidation object passed to triangulate_optim.

exclude_markers(all_markers_to_exclude_config_path: Path | str, verbose: bool = True)

Exclude markers in prediction based on markers_to_exclude config.

Parameters:

• all_markers_to_exclude_config_path (Path or str) – Filepath to the config used for exclusion of markers.

• verbose (bool, default True) – If True (default), print if exclusion of markers worked without any abnormalities.

Notes

The all_markers_to_exclude_config_path has to be a path to a yaml file representing a dictionary with the camera names as keys and a list of the markers, that should be excluded as value.

run_triangulation(calibration_toml_filepath: Path | str, triangulate_full_recording: bool = True, use_preexisting_csvs: bool = False) → None

Load and validate the calibration, triangulate and create 3D df.

Validate, that the camera names in camera_group match triangulation_dlc_cams_filepaths and drop or add empty files if they don’t. Triangulate using different options as defined in the project_config via ap_lib functions.

Parameters:

• calibration_toml_filepath (Path or str) – Filepath to the calibration, that should be used for triangulation.

• triangulate_full_recording (bool, default True) – If False (default True), then only the first 2 frames of the recording will be triangulated and the 3D dataframe won’t be saved.

• use_preexisting_csvs (bool, default False) – If True (default False), then a already existing file at csv_output_filepath will be read in and no triangulatin will be performed.

class core.triangulation_calibration_module.TriangulationRecordings(project_config_filepath: Path | str, directory: Path | str, recording_config_filepath: Path | str, recreate_undistorted_plots: bool = True, output_directory: Path | str | None = None)

Bases: Triangulation

Subclass of Triangulation, in which videos are triangulated based on a calibration file.

Temporal synchronization of the videos can be performed based on a pattern. The triangulated dataframe can be normalised (rotated and translated). For visalization, a triangulated video can be created.

run_synchronization(overwrite_DLC_analysis_and_synchro, verbose):

Perform analysis of all videos using DLC or other methods, synchronization to the led_pattern and downsampling to target_fps.

create_triangulated_video(filename, config_path):

Create video of the triangulated data.

normalize(normalization_config_path, save_dataframe):

Rotate and translate the triangulated dataframe.

See also

Triangulation

Parent class, for triangulation of videos or images.

core.meta.MetaInterface.create_recordings

Create TriangulationRecording objects for all recording_directories added to MetaInterface.

core.meta.MetaInterface.synchronize_recordings

Run the function run_synchronization for all TriangulationRecording objects added to MetaInterface.

core.meta.MetaInterface.triangulate_recordings

Run the function run_triangulation for all TriangulationRecording objects added to MetaInterface.

core.checker_objects.CheckRecording

A class, that checks the metadata and filenames of videos in a given folder and allows for filename changing via user input.

Examples

>>> from core.triangulation_calibration_module import TriangulationRecordings
>>> rec_config = "test_data/Server_structure/Calibrations/220922/recording_config_220922.yaml"
>>> directory = "test_data/Server_structure/VGlut2-flp/September2022/206_F2-63/220922_OTE/"
>>> triangulation_object = TriangulationRecordings(
    ... directory=directory,
    ... recording_config_filepath=rec_config,
    ... project_config_filepath="test_data/project_config.yaml",
    ... recreate_undistorted_plots = True,
    ... output_directory=directory
    ... )
>>> triangulation_object.run_synchronization()
>>> triangulation_object.exclude_markers(
    ... all_markers_to_exclude_config_path="test_data/markers_to_exclude_config.yaml",
    ... verbose=False,
    ... )
>>> triangulation_object.run_triangulation(
    ... calibration_toml_filepath="test_data/Server_structure/Calibrations/220922/220922_0_Bottom_Ground1_Ground2_Side1_Side2_Side3.toml"
    ... )
>>> normalised_path, normalisation_error = triangulation_object.normalize(
    ... normalization_config_path="test_data/normalization_config.yaml"
    ... )

create_triangulated_video(filename: str, config_path: Path | str, start_s: int, end_s: int) → None

Create video of the triangulated data.

Parameters:

• filename (str) – The filename, where the video should be saved.

• config_path (Path or str) – The path to the config used to create triangulated videos.

• start_s – The second in the recording to start video creation.

• end_s – The second in the recording to end video creation.

Notes

The yaml file at config_path has to have the following keys:

body_marker_size, body_label_size: int, int body_marker_color, body_label_color: str, str

matplotlib color

body_marker_alpha, body_label_alpha: float 0 < 1, float 0 < 1 markers_to_exclude: list of str

Markers that should not be plotted. Recommended is, giving at least “M_”, “center_”, “fnum” and “Unnamed” to it, since those labels are created from aniposelib and can not be plotted.

markers_to_connect: list of list of str, optional

Each element consists of a list of markers, that will be connected in the video.

markers_to_fill: list of dict, optional

Each element consists of a dict, that will be filled in the video. The keys of the dict have to be:

markers: list of str

The markers, to create a polygon in between.

color: str

matplotlib color, to fill the polygon.

alpha: float 0 < 1

additional_markers_to_plot: list of dict, optional

Markers to plot in addition to the triangulated points. The elements of the lists are dictionaries containing the following key-value pairs:

name: str x, y, z: int, int, int alpha: float 0 < 1 size: int color: str

matplotlib color

Can be used to set the axis aspect equal, since this feature is not established for 3D axes in matplotlib. All markers can be used to be connected or filled.

normalize(normalization_config_path: Path | str, save_dataframe: bool = True, verbose: bool = False) → Tuple[Path, float]

Rotate and translate the triangulated dataframe.

Find frame, in which all markers given in the config are defined. Translate all points to center. Convert into cm. Rotate dataframe using scipy.transform.Rotation.align_vectors to align triangulated vectors and ground truth vectors. Create a plot for Visualization of Rotation.

Parameters:

• normalization_config_path (Path or str) – The path to the config used for normalisation.

• save_dataframe (bool, default True) – If True (default), then the dataframe will be saved and overwrites the pre-existing one.

Returns:

• self.rotated_filepath (Path) – Filepath, were the rotated triangulated dataframe is saved.

• rotation_error (flot) – Error returned by scipy.transform.Rotation.align_vectors, representing whether the alignment worked well.

• verbose (bool, default False) – If True (default False), then the rotation visualization plot is shown.

Notes

The normalization_config_path is a path to a yaml file, representing a dictionary with the following key-value pairs:

CENTER: str

The marker, at which to set (0, 0, 0).

REFERENCE_LENGTH_CM: int

The reference length in cm.

REFERENCE_LENGTH_MARKERS: list

The two markers for defining the reference length. The distance between those markers in px will be set to ReferenceLengthCm.

REFERENCE_ROTATION_COORDS: list of list of int

List of reference rotation markers (at least 3), to align the real world space and the triangulated space. Each element is a list of int, defining their x, y and z coordinate.

REFERENCE_ROTATION_MARKERS: list of str

List of triangulated markers, that should be aligned with ReferenceRotationCoords. Their lengths have to be equal!

INVISIBLE_MARKERS: {str: list of int}

Keys are ‘x’, ‘y’ and ‘z’, values are lists of ints. All lists have to match in length. The length is equal to the number of points to be plotted. Markers to plot in rotation visualization plot invisiblly. Can be used to set the axis aspect equal, since this feature is not established for 3D axes in matplotlib.

run_synchronization(overwrite_DLC_analysis_and_synchro: bool = False, verbose: bool = True) → None

Perform analysis of all videos using DLC or other methods, synchronization to the led_pattern and downsampling to target_fps.

Call the synchronizer via VideoInterface and save the prediction file in triangulation_dlc_cams_filepaths. Create a plot for crossvalidation of the synchronised LED timeseries. Define self.markers as unique markers found in all prediction files. Exclude videos, if there are diverging framenumbers after synchronization.

Parameters:

• overwrite_DLC_analysis_and_synchro (bool, default False) – If True (default False), then pre-existing DLC files and synchronisations will be overwritten during analysis.

• verbose (bool, default True) – If True (default), then Crossvalidation plot and synchronised number of frames for each camera are printed.

core.user_specific_rules module

core.user_specific_rules.user_specific_rules_on_triangulation_calibration_videos(videometadata: VideoMetadata)

core.user_specific_rules.user_specific_rules_on_videometadata(videometadata: VideoMetadata)

core.utils module

class core.utils.Coordinates(y_or_row: int, x_or_column: int, z: int | None = None)

Bases: object

Class to store xyz coordinates.

Parameters:

• y_or_row (int) –

• x_or_column (int) –

• z (int) –

x

Represent x coordinates in a system and columns in an image.

Type:

int, column: int

y, row

Represent y coordinates in a system and rows in an image.

Type:

int

z

Represent z coordinates in a system.

Type:

int

core.utils.KEYS_TO_CHECK_CAMERA_RECORDING = ['fps', 'offset_row_idx', 'offset_col_idx', 'flip_h', 'flip_v', 'fisheye']

int

The framerate of the camera.

offset_row_idx: int, default 0

If cropping was performed, specify the row or y index initial here.

offset_col_idx: int, default 0

If cropping was performed, specify the col or x index initial here.

flip_h: bool, default False

Based on the size from uncropped intrinsic calibration videos, the row or y index end is calculated and used instead of offset_row_idx.

flip_v: bool, default False

Based on the size from uncropped intrinsic calibration videos, the col or x index end is calculated and used instead of offset_col_idx.

fisheye: bool, default False

If a fisheye lens was used, set True.

Type:

fps

core.utils.KEYS_TO_CHECK_PROJECT = ['valid_cam_ids', 'paradigms', 'animal_lines', 'led_extraction_type', 'led_extraction_filepath', 'max_calibration_frames', 'max_cpu_cores_to_pool', 'max_ram_digestible_frames', 'rapid_aligner_path', 'use_gpu', 'load_calibration', 'calibration_tag', 'calibration_validation_tag', 'allowed_num_diverging_frames', 'handle_synchro_fails', 'default_offset_ms', 'start_pattern_match_ms', 'end_pattern_match_ms', 'synchro_error_threshold', 'synchro_marker', 'led_box_size', 'use_2D_filter', 'score_threshold', 'num_frames_to_pick', 'triangulation_type']

Keys

calibration_evaluation_filepath: {str: str}

Dictionary with keys for all valid cam_ids, defining the filepath to the file to use for marker detection for calibration_validation per cam.

calibration_evaluation_type: {str: str}

Dictionary with keys for all valid cam_ids, defining the type of marker detection to use per cam for calibration_evaluation files. Values: DLC, manual

processing_filepath: {str: str}

Dictionary with keys for all valid cam_ids, defining the filepath to the file to use for marker detection for recordings per cam.

processing_type: {str: str}

Dictionary with keys for all valid cam_ids, defining the type of marker detection to use per cam for recordings. Values: DLC, manual, exclude

led_extraction_filepath: {str: str}

Dictionary with keys for all valid cam_ids, defining the filepath to the file to use for marker detection for synchronization per cam.

led_extraction_type: {str: str}

Dictionary with keys for all valid cam_ids, defining the type of marker detection to use per cam for synchronization. Values: DLC, manual

animal_lines: list of str

List of all animal_lines to search for in recording filenames. If the lines are numbers, you need to set the lines as str.

paradigms: list of str

List of all paradigms to search for in filenames and directories.

valid_cam_ids: list of str

List of all cam_ids to search for in filenames.

calibration_tag: str

Filename tag to search for in the calibration files. “Calvin” is an invalid value.

calibration_validation_tag: str

Filename tag to search for in the calibration_validation files. “Calvin” is an invalid value.

use_gpu: str, default “”

Whether to restrict the usage of GPU for DLC analyses. Values: “”, “prevent”, “low”, “full” “prevent” disables GPU usage “low” restricts GPU memory for synchronization “full” uses entire GPU capacity, equivalent to “”

intrinsic_calibration_directory: str

The directory, in which the intrinsic calibration pickle .p files or the intrinsic calibration checkerboard videos are stored. Intrinsic calibration videos have to have “checkerboard” and a valid cam_id in their filename. They have to be recorded in same resolution as the recording/calibration videos without cropping, using a 6x6 checkerboard.

load_calibration: bool

If True, then use this package to set the intrinsics of the cameras instead of using the aniposelib function to calibrate intrinsics. Requires checkerboard videos or .p pickle files for all cameras in intrinsic_calibration_directory.

triangulation_type: str, default “triangulate”

Specify the method of aniposelib triangulation to use. Values: “triangulate”, “triangulate_optim_ransac_True”, “triangulate_optim_ransac_False”

allowed_num_diverging_frames: int

Specify how many frames a synchronized file can differ from the median of all synchronized files before it will be excluded from analysis.

handle_synchro_fails: str, default error

How to proceed if the first synchronisation try exceeds synchro_error_threshold. Values: “repeat”, “default”, “manual”, “error” “repeat” run the same method of synchronisation again. “default” using default value default_offset_ms as synchro offset. “manual” using manual marker detection for synchronisation “error” raises an error and breaks the analysis

default_offset_ms: int

Synchro offset to use if first synchro try fails and handle_synchro_fails is “default”. In milliseconds.

start_pattern_match_ms: int, default 0

Start of time range in which to search for matching synchro pattern. In milliseconds.

end_pattern_match_ms: int

End of time range in which to search for matching synchro pattern. In milliseconds.

synchro_error_threshold: int, default 100

Below this threshold, a synchro patter alignment error will be considered as good synchro, above, as failed synchro.

synchro_marker: str

The marker to use for synchronisation. Has to be detectable by the led_extraction method.

led_box_size: int

Pixel range around predicted synchro marker position to calculate mean pixel intensity for blinking pattern from.

use_2D_filter: bool, default True

Whether to use filtering on 2D marker detection predictions in recordings. At the moment only available for DLC.

score_threshold: float, default 0.9

Only predictions with likelihood above this threshold will be taken into account for triangulation.

num_frames_to_pick: int, default 5

The number of frames to use to find optimal prediction for the synchro_marker.

max_ram_digestible_frames: int, default 3000

Maximum number of frames to keep in RAM during writing of synchronised videos. Increase to speed up analysis, reduce to adapt to available RAM.

max_cpu_cores_to_pool: int, default 0

If 0, then no multiprocessing will be used for writing of synchronised videos. Set to as many CPU cores, as you would like to use for multiprocessing.

max_calibration_frames: int

Number of frames to take into account for intrinsic calibration. 300 works well, depending on CPU speed, it can be necessary to reduce.

rapid_aligner_path: str, default “”

If “”, then no rapid_aligner (GPU based pattern alignment) will be used. Insert path to locally installed clone of the rapid_aligner package to use GPU for pattern synchronisation.

core.utils.KEYS_TO_CHECK_RECORDING = ['led_pattern', 'target_fps', 'calibration_index', 'recording_date']

dict

Blinking pattern to use for temporal synchronisation.

target_fps: int

Fps rate, to which the videos should be synchronized.

calibration_index: int, default 0

Index of a calibration. Together with recording_date, it creates a unique calibration key.

recording_date: str

Date at which the calibration was done.

Type:

led_pattern

core.utils.check_keys(dictionary: Dict, list_of_keys: List[str]) → List

Check, whether list_of_keys are in a dictionary and returns a list of keys missing in the dictionary.

core.utils.construct_dlc_output_style_df_from_dictionary(marker_predictions: Dict) → DataFrame

Create a DataFrame from dictionary with DLC-like multi-index.

Parameters:

marker_predictions ({str: {str: list of int}}) – Dictionary containing markers as keys and dictionaries as values with x, y and z as keys and lists of int as values. The length of the lists is equivalent to the number of frames annotated.

Returns:

df – The dataframe with DLC-like multiindex and data from input dictionary.

Return type:

pd.DataFrame

core.utils.convert_to_path(attribute: str | Path) → Path

Convert strings to Path and returns them.

core.utils.create_calibration_key(videos: List[str], recording_date: str, calibration_index: int, iteration: int | None = None) → str

core.utils.get_3D_array(df: DataFrame, key: str, index: int | None = None) → array

Construct a ndarray of shape (3, N) representing one marker in a 3 dimensional DataFrame with N number of indices.

core.utils.get_3D_df_keys(key: str) → Tuple[str, str, str]

Construct a tuple of strings representing keys in a aniposelib created 3 dimensional DataFrame.

core.utils.get_multi_index(markers: List) → MultiIndex

core.utils.load_image(filepath: Path, idx: int = 0) → ndarray

core.utils.load_single_frame_of_video(filepath: Path, frame_idx: int = 0) → ndarray

core.utils.plot_image(filepath: Path, idx: int = 0, plot_size: Tuple[int, int] = (9, 6)) → None

core.utils.plot_single_frame_of_video(filepath: Path, frame_idx: int = 0, plot_size: Tuple[int, int] = (9, 6)) → None

core.utils.read_config(path: Path) → Dict

Read structured config file defining a project.

core.video_interface module

class core.video_interface.VideoInterface(video_metadata: VideoMetadata, output_dir: Path, recreate_undistorted_plots: bool = True)

Bases: object

export_for_aniposelib() → Union

inspect_intrinsic_calibration() → None

run_synchronizer(synchronizer: Type, output_directory: Path, synchronize_only: bool, overwrite_DLC_analysis_and_synchro: bool, synchro_metadata: Dict, verbose: bool = True) → None

core.video_metadata module

class core.video_metadata.VideoMetadata(video_filepath: str | Path, recording_config_dict: Dict, project_config_dict: Dict, tag: str)

Bases: ABC

Class to store metadata for videos.

Metadata is read from the filepath and the recording and project config dictionaries.

Parameters:

• video_filepath (str or Path) – The path to the video, which metadata will be stored here.

• recording_config_dict (Dict) – The recording config dictionary as read from the .yaml file.

• project_config_dict (Dict) – The project config dictionary as read from the .yaml file.

• tag (str) – Depending on the type of video. Values: “calvin” for calibration validation, “recording” for recording and “calibration” for calibration videos.

self.intrinsic_calibration

The intrinsic calibration assigned to the camera, consisting of camera matrix and distortion coefficient.

self.intrinsic_calibration_filepath

The filepath to the intrinisc calibration saved as .p pickle file.

self.calvin

Boolean argument, whether VideoMetadata belongs to calibration validation.

self.recording

Boolean argument, whether VideoMetadata belongs to recording video.

self.calibration

Boolean argument, whether VideoMetadata belongs to calibration video.

self.exclusion_state

“Valid”, if video has not to be excluded for any reasons, “exclude”, if it has to be excluded.

self.filepath

The path to the video, which metadata will be stored here.

self.recording_date

Date at which the calibration was done based on recording_config and as read from the filenames.

self.mouse_id

The mouse_id as read from the filename.

self.mouse_line

The mouse line as read from the filename.

self.mouse_number

The mouse number as read from the filename.

self.cam_id

The cam_id as read from the filename.

self.paradigm

The paradigm as read from the filename.

See also

core.utils.KEYS_TO_CHECK_CAMERA_RECORDING, core.utils.KEYS_TO_CHECK_RECORDING, core.utils.KEYS_TO_CHECK_PROJECT

Notes

The Attributes described in the docstring are only those ones, that are not keys from recording or project config. For explanation of these attributes, we refer to KEYS_TO_CHECK_CAMERA_RECORDING, KEYS_TO_CHECK_RECORDING and KEYS_TO_CHECK_PROJECT.

class core.video_metadata.VideoMetadataChecker(video_filepath: str | Path, recording_config_dict: Dict, project_config_dict: Dict, tag: str)

Bases: VideoMetadata

Class to verify metadata for videos and rename filenames if necessary.

core.video_synchronization module

class core.video_synchronization.CharucoVideoSynchronizer(video_metadata: VideoMetadata, output_directory: Path, synchro_metadata: Dict)

Bases: Synchronizer

class core.video_synchronization.MotifTemplate(led_on_time_in_ms: int, on_off_period_length_in_ms: int, motif_duration_in_ms: int)

Bases: TimeseriesTemplate

Class to store single motif.

led_on_time_in_ms

Duration of one on-peak.

Type:

int

on_off_period_length_in_ms

Period between two on-peaks.

Type:

int

motif_duration_in_ms

Total duration of the motif.

Type:

int

template_timeseries

Array of binary 0 / 1 values representing a single motif with a resolution of 1 ms.

Type:

np.ndarray

adjust_template_timeseries_to_fps()

Adjust template to framerate.

property template_attribute_string: str

class core.video_synchronization.MultiMotifTemplate

Bases: TimeseriesTemplate

Class to combine single motives and store the resulting multi_motif.

multi_motif_template

Array of binary 0 / 1 values representing a multi_motif_template with a resolution of 1 ms.

Type:

np.ndarray

motif_templates

List of single motifs.

Type:

list of MotifTemplates

add_motif_template(motif_template)

Append a motif to the already existing multi_motif.

adjust_template_timeseries_to_fps()

Adjust template to framerate.

See also

construct_template_motif

Construct template from dictionary.

add_motif_template(motif_template: MotifTemplate) → None

Append a motif to the already existing multi_motif.

property template_attribute_string: str

class core.video_synchronization.RecordingVideoDownSynchronizer(video_metadata: VideoMetadata, output_directory: Path, synchro_metadata: Dict)

Bases: RecordingVideoSynchronizer

class core.video_synchronization.RecordingVideoSynchronizer(video_metadata: VideoMetadata, output_directory: Path, synchro_metadata: Dict)

Bases: Synchronizer

class core.video_synchronization.RecordingVideoUpSynchronizer(video_metadata: VideoMetadata, output_directory: Path, synchro_metadata: Dict)

Bases: RecordingVideoSynchronizer

class core.video_synchronization.Synchronizer(video_metadata: VideoMetadata, output_directory: Path, synchro_metadata: Dict)

Bases: ABC

Class to synchronize videos.

Run alignment between constructed MotifTemplate and detected led blinking pattern. Select frames based on timestemps to match framerate, potentially interpolate and add frames. Start marker detection or write video and save the adjusted files.

Parameters:

• video_metadata (VideoMetadata) – Metadata from video.

• output_directory (Path or str) – Directory, in which the files created during the synchronisation are saved.

• synchro_metadata (Dict) – Metadata for synchronisation containing SYNCHRO_METADATA_KEYS.

video_metadata

Metadata from video.

Type:

VideoMetadata

use_rapid_aligner

If True, synchro pattern alignment will be based on GPU, if False, on CPU.

Type:

bool

rapid_aligner_path

Path to locally installed clone of the rapid_aligner package to use GPU for pattern synchronisation.

Type:

Path

use_gpu

Whether to restrict the usage of GPU for DLC analyses.

Type:

str

led_box_size

Pixel range around predicted synchro marker position to calculate mean pixel intensity for blinking pattern from.

Type:

int

output_directory

Directory, in which the files created during the synchronisation are saved.

Type:

Path or str

synchro_metadata

Metadata for synchronisation containing SYNCHRO_METADATA_KEYS.

Type:

Dict

led_timeseries_for_cross_video_validation

Mean pixel intensity at predicted synchro marker position adjusted to target fps.

Type:

np.ndarray

led_detection

Plot to verify correct prediction of LED marker.

Type:

LEDMarkerPlot

led_timeseries

Extracted mean pixel intensity at predicted synchro marker position.

Type:

np.ndarray

template_blinking_motif

Constructed template from blinking pattern metadata.

Type:

MotifTemplate or MultiMotifTemplate

run_synchronization(synchronize_only, overwrite_DLC_analysis_and_synchro, verbose)

Run alignment between template and detected led blinking pattern. Select frames to adjust framerate. Start marker detection or write video.

Examples

>>> from core.video_synchronization import RecordingVideoDownSynchronizer
>>> from core.video_metadata import VideoMetadata
>>> from core.utils import read_config, SYNCHRO_METADATA_KEYS
>>> from pathlib import Path
>>> project_config = read_config("test_data/project_config.yaml")
>>> video_filepath = Path("test_data/Server_structure/VGlut2-flp/September2022/206_F2-63/220922_OTE/220922_206_F2-63_OTE_Side2.mp4")
>>> recording_config_dict = read_config("test_data/Server_structure/Calibrations/220922/recording_config_220922.yaml")
>>> synchro_metadata = {key:project_config[key] for key in SYNCHRO_METADATA_KEYS})
>>> video = VideoMetadata(video_filepath=video_filepath, recording_config_dict=recording_config_dict,
    ... project_config_dict=project_config, tag = "recording")
>>> synchronizer_object = RecordingVideoDownSynchronizer(video_metadata=video,
    ... output_directory=video_filepath.parent, synchro_metadata=synchro_metadata)
>>> marker_detection_filepath, _ = synchronizer_object.run_synchronization()

run_synchronization(synchronize_only: bool = False, overwrite_DLC_analysis_and_synchro: bool = False, verbose: bool = True) → Tuple[Path | None, Path | None]

Run alignment between template and detected led blinking pattern. Select frames to adjust framerate. Start marker detection or write video.

Parameters:

• synchronize_only (bool, default False) – To be used by calibration videos only. If True, then only synchronized videos are created and no marker detection is run.

• overwrite_DLC_analysis_and_synchro (bool, default False) – If True (default False), then pre-existing DLC files and synchronisations will be overwritten during analysis.

• verbose (bool, default True) – If True (default), then the number of synchronized frames is printed.

Returns:

• marker_detection_filepath (Path, optional) – The path to the synchronised marker prediction file.

• synchronized_video_filepath (Path, optional) – The path to the synchronised video file.

property target_fps: int

class core.video_synchronization.TimeseriesTemplate

Bases: ABC

adjust_template_timeseries_to_fps(fps: int) → List[Tuple[ndarray, int]]

Adjust template to framerate.

Parameters:

fps (int) – The framerate, to which the template will be adjusted.

Returns:

fps_adjusted_templates – List of all possible timeseries that could be observed, given the framerate and the resolution of the template (1 ms) as tuple of template as np.ndarray and offset in ms.

Return type:

list of tuple[np.ndarray, int]

abstract property template_attribute_string: str

core.video_synchronization.construct_template_motif(blinking_patterns_metadata: Dict) → MotifTemplate | MultiMotifTemplate

Construct template from dictionary.

Parameters:

blinking_patterns_metadata – Dictionaries with patterns as values, that specify arguments for MotifTemplate as key-value pairs: “led_on_time_in_ms”, “on_off_period_length_in_ms”, “motif_duration_in_ms”.

Returns:

template_motif – The constructed template.

Return type:

MotifTemplate or MultiMotifTemplate

See also

MultiMotifTemplate

Class to combine single motives and store the resulting multi_motif.

MotifTemplate

Class to store single motif.

Examples

>>> from core.video_synchronization import construct_template_motif
>>> led_pattern =     ... {0: {'led_on_time_in_ms': 50,
... 'on_off_period_length_in_ms': 100,
... 'motif_duration_in_ms': 3000},
... 1: {'led_on_time_in_ms': 50,
... 'on_off_period_length_in_ms': 1000,
... 'motif_duration_in_ms': 2000}}
>>> template_motif = construct_template_motif(led_pattern)

Module contents