let_it_be_3D

With let_it_be_3D we want to extend the functions of aniposelib and bring them into a pipeline structure. Our goals are having as less manual steps required as possible and standardized quality assurance/collection of metadata. We provide additional methods for video synchronisation, adjustment for different framerates, validation of anipose calibration, adjustment of intrinsic calibrations to croppings, manual marker detection, checking and correcting filenames and normalisation of the 3D triangulated dataframe.

Pipeline explained Step-by-Step!

1) Load videos and metadata

  • read video metadata from filename and recording config file

  • intrinsic calibrations

    • use anipose intrinsic calibration

    • run or load intrinsic calibration based on uncropped checkerboard videos adjust intrinsic calibration for video cropping

2) Video processing

  • synchronize videos temporally based on a blinking signal

    Example

  • run marker detection on videos manually or using DeepLabCut networks

    Example

  • write videos and marker detection files to the same framerate

3) Calibration

  • run extrinsic Anipose camera calibration

  • validate calibration based on known distances and angles (ground truth) between calibration validation markers

    Example This calibration validation shows the triangulated representation of a tracked rectangle, that has 90° angles at the corners.

4) Triangulation

  • triangulate recordings

    Example

  • rotate dataframe, translate to origin, normalize to centimeter

    Example The blue vectors were aligned to the yellow vectors succesfully.

  • add metadata to database

How to use

Installation

# Clone this repository
$ git clone https://github.com/retune-commons/let_it_be_3D.git

# Go to the folder in which you cloned the repository
$ cd let_it_be_3D

# Install dependencies
# first, install deeplabcut into a new environment as described here: (https://deeplabcut.github.io/DeepLabCut/docs/installation.html)
$ conda env update --file env.yml 

# Open Walkthrough.ipynb in jupyter lab
$ jupyter lab

# Update project_config.yaml to your needs and you're good to go!

Required filestructure

Video filename

  • calibration:

    • has to be a video [“.AVI”, “.avi”, “.mov”, “.mp4”]

    • including recording_date (YYMMDD), calibration_tag (as defined in project_config) and cam_id (element of valid_cam_ids in project_config)

    • recording_date and calibration_tag have to be separated by an underscore (“_”)

    • f”{recording_date}{calibration_tag}{cam_id}” = Example: “220922_charuco_Front.mp4”

  • calibration_validation:

    • has to be a video or image [“.bmp”, “.tiff”, “.png”, “.jpg”, “.AVI”, “.avi”, “.mp4”]

    • including recording_date (YYMMDD), calibration_validation_tag (as defined in project_config) and cam_id (element of valid_cam_ids in project_config)

    • recording_date and calibration_validation_tag have to be separated by an underscore (“_”)

    • calibration_validation_tag mustn’t be “calvin”

    • f”{recording_date}_{calibration_validation_tag}” = Example: “220922_position_Top.jpg”

  • recording:

    • has to be a video [“.AVI”, “.avi”, “.mov”, “.mp4”]

    • including recording_date (YYMMDD), cam_id (element of valid_cam_ids in project_config), mouse_line (element of animal_lines in project_config), animal_id (beginning with F, split by “-” and followed by a number) and paradigm (element of paradigms in project_config)

    • recording_date, cam_id, mouse_line, animal_id and paradigm have to be separated by an underscore (“_”)

    • f”{recording_date}{cam_id}{mouse_line}{animal_id}{paradigm}.mp4” = Example: “220922_Side_206_F2-12_OTT.mp4”

Folder structure

  • A folder, in which a recordings is stored should match the followed structure to be detected automatically:

    • has to start with the recording_date (YYMMDD)

    • has to end with any of the paradigms (as defined in project_config)

    • recording date and paradigm have to be separated by an underscore (“_”)

    • f”{recording_date}_{paradigm}” = Example: “230427_OF”

License

GNU General Public License v3.0

Contributers

This is a Defense Circuits Lab project. The pipeline was designed by Konstantin Kobel, Dennis Segebarth and Michael Schellenberger. At the Sfb-Retune Hackathon 2022, Elisa Garulli, Robert Peach and Veronika Selzam joined the taskforce to push the project towards completion.

Sfb-Retune DefenseCircuitsLab

Contact

If you want to help with writing this pipeline, please get in touch.