# 3D

sleap-io supports multi-camera setups for 3D pose reconstruction. A `RecordingSession` ties together camera calibrations, synchronized videos, and cross-camera instance matching, making it straightforward to work with multi-view data in a single unified data model.

### How the types fit together

- **`Camera`** stores calibration data for a single viewpoint: intrinsic matrix, distortion coefficients, and extrinsic pose (rotation + translation).
- **`CameraGroup`** groups the cameras that are used together in a multi-view rig.
- **`RecordingSession`** links each camera to its video and holds frame groups for cross-view annotations.
- **`FrameGroup`** matches the labeled frames from each camera at the same time point.
- **`InstanceGroup`** matches instances of the same animal across cameras for 3D triangulation.

The main relationships are: `RecordingSession` → `CameraGroup` → `Camera` for calibration, and `RecordingSession` → `FrameGroup` → `InstanceGroup` for cross-view correspondences. Sessions are stored on [`Labels.sessions`](labels.md) so that multi-view data travels alongside the rest of the annotations.

---

## Camera

A `Camera` stores everything needed to map between 3D world coordinates and 2D pixel coordinates for a single viewpoint:

- **Intrinsic parameters**: the 3x3 camera matrix encoding focal length and principal point, plus radial-tangential distortion coefficients.
- **Extrinsic parameters**: the camera's position and orientation in world coordinates, stored as a rotation vector (`rvec`, axis-angle representation) and translation vector (`tvec`). An extrinsic 4x4 matrix is derived automatically and stays in sync when either vector is updated.

```pycon
>>> import sleap_io as sio
>>> import numpy as np
>>> cam = sio.Camera(
...     matrix=np.array([[500, 0, 320], [0, 500, 240], [0, 0, 1]], dtype="float64"),
...     dist=np.zeros(5),
...     size=(640, 480),
...     rvec=np.zeros(3),
...     tvec=np.array([0, 0, 1], dtype="float64"),
...     name="cam_top",
... )
>>> print(cam.name)
>>> print(cam.size)
>>> print(cam.matrix)

```

The `matrix` field follows the standard pinhole camera model:

$$
K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}
$$

where $f_x, f_y$ are focal lengths in pixels and $(c_x, c_y)$ is the principal point.

The five distortion coefficients follow OpenCV ordering: $[k_1, k_2, p_1, p_2, k_3]$.

---

## Camera group

A `CameraGroup` collects the set of cameras that were used together in a multi-view rig. This is primarily an organizational container: it holds a list of `Camera` objects and optional metadata.

```pycon
>>> import sleap_io as sio
>>> import numpy as np
>>> cam = sio.Camera(
...     matrix=np.array([[500, 0, 320], [0, 500, 240], [0, 0, 1]], dtype="float64"),
...     dist=np.zeros(5),
...     size=(640, 480),
...     rvec=np.zeros(3),
...     tvec=np.array([0, 0, 1], dtype="float64"),
...     name="cam_top",
... )
>>> cam2 = sio.Camera(
...     matrix=np.eye(3) * 500,
...     dist=np.zeros(5),
...     size=(640, 480),
...     rvec=np.array([0, 0.5, 0]),
...     tvec=np.array([1, 0, 1], dtype="float64"),
...     name="cam_side",
... )
>>> cg = sio.CameraGroup(cameras=[cam, cam2])
>>> print(len(cg.cameras))

```

---

## Recording session

A `RecordingSession` represents a synchronized multi-camera recording. It links each `Camera` in a `CameraGroup` to the [`Video`](video.md) captured from that viewpoint, and it stores `FrameGroup`s that contain the cross-view annotations.

```pycon
>>> import sleap_io as sio
>>> import numpy as np
>>> cam = sio.Camera(
...     matrix=np.array([[500, 0, 320], [0, 500, 240], [0, 0, 1]], dtype="float64"),
...     dist=np.zeros(5), size=(640, 480), rvec=np.zeros(3),
...     tvec=np.array([0, 0, 1], dtype="float64"), name="cam_top",
... )
>>> cam2 = sio.Camera(
...     matrix=np.eye(3) * 500, dist=np.zeros(5), size=(640, 480),
...     rvec=np.array([0, 0.5, 0]),
...     tvec=np.array([1, 0, 1], dtype="float64"), name="cam_side",
... )
>>> cg = sio.CameraGroup(cameras=[cam, cam2])
>>> vid_top = sio.Video("top_view.mp4", open_backend=False)
>>> vid_side = sio.Video("side_view.mp4", open_backend=False)
>>> session = sio.RecordingSession(
...     camera_group=cg,
...     video_by_camera={cam: vid_top, cam2: vid_side},
...     camera_by_video={vid_top: cam, vid_side: cam2},
... )
>>> print(session.cameras)
>>> print(session.videos)
>>> print(session.get_camera(vid_top).name)

```

You can also add or remove video-camera mappings after construction:

```python
# Adding a new camera to an existing session
cam3 = sio.Camera(name="cam_rear")
cg.cameras.append(cam3)
vid_rear = sio.Video("rear_view.mp4", open_backend=False)
session.add_video(vid_rear, cam3)
len(session.videos)  # 3
```

!!! note "See also"

    `RecordingSession` objects are stored on the top-level dataset container at [`Labels.sessions`](labels.md). This is how multi-view calibration and correspondence data travels alongside the rest of the pose annotations.

---

## Frame group

A `FrameGroup` represents a single time point across all camera views. It groups the [`LabeledFrame`](labels.md)s from each camera at the same frame index and contains one or more `InstanceGroup`s that define cross-camera correspondences.

Key properties:

- `frame_idx` — the synchronized frame index shared across views.
- `labeled_frames` — the per-camera `LabeledFrame` objects at this time point.
- `instance_groups` — cross-camera matches for individual animals (see below).
- `get_frame(camera)` — look up the `LabeledFrame` for a specific camera.

---

## Instance group

An `InstanceGroup` links together 2D [`Instance`](poses.md) annotations of the **same animal** as seen from different cameras at the same frame. This cross-camera correspondence is the key input for 3D triangulation.

Key properties:

- `instance_by_camera` — dictionary mapping each `Camera` to the corresponding `Instance`.
- `instances` — flat list of all matched `Instance` objects.
- `cameras` — list of cameras that have a matched instance.
- `instance_3d` — optional `Instance3D` with triangulated 3D keypoints.
- `points` — convenience accessor for `instance_3d.points` (backwards compatible).
- `score` — optional confidence score for the match.
- `identity` — optional `Identity` for this group (which animal).

```pycon
>>> import numpy as np
>>> import sleap_io as sio
>>> skel = sio.Skeleton(nodes=["head", "tail"], edges=[("head", "tail")])
>>> cam1, cam2 = sio.Camera(name="cam1"), sio.Camera(name="cam2")
>>> inst_2d_cam1 = sio.Instance.from_numpy(np.array([[10, 20], [30, 40]]), skeleton=skel)
>>> inst_2d_cam2 = sio.Instance.from_numpy(np.array([[12, 22], [32, 42]]), skeleton=skel)
>>> inst_3d = sio.Instance3D(points=np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]), skeleton=skel)
>>> mouse_a = sio.Identity(name="mouse_A")
>>> group = sio.InstanceGroup(
...     instance_by_camera={cam1: inst_2d_cam1, cam2: inst_2d_cam2},
...     instance_3d=inst_3d,
...     identity=mouse_a,
... )
>>> print(len(group.instances), "matched instances")
>>> print(group.identity.name)

```

---

## Identity

An `Identity` represents a ground-truth animal that can be recognized across videos, sessions, and experiments. Unlike [`Track`](poses.md) (an ephemeral, video-local temporal trajectory), `Identity` is the **global** re-identification key, persistent across recordings, sessions, and multi-view setups.

Key properties:

- `name` — human-readable name (e.g., `"mouse_A"`). Not required to be unique, but `name` is how identities are matched across separately-loaded files and merges.
- `metadata` — arbitrary string-keyed, string-valued metadata dictionary (e.g. `{"color": "#e6194b", "strain": "C57BL/6"}`). A visualization color, if wanted, is just a conventional `metadata["color"]` entry.

Identities attach at three levels:

- **Per instance** — [`Instance.identity`](poses.md) + `Instance.identity_score` assign a global identity to a detection (distinct from the short-term `track`/`tracking_score`).
- **Per multi-view group** — `InstanceGroup.identity` binds the triangulated detections of one animal across cameras; the many per-camera `Track`s of an animal map to a single `Identity`.
- **Top-level catalog** — [`Labels.identities`](labels.md) holds the canonical identity objects, auto-collected from instances.

```pycon
>>> import sleap_io as sio
>>> mouse_a = sio.Identity(name="mouse_A")
>>> mouse_b = sio.Identity(name="mouse_B")
>>> labels = sio.Labels(identities=[mouse_a, mouse_b])
>>> print([ident.name for ident in labels.identities])

```

### Matching identities

Compare identities with `matches()` (default `method="name"`), or configure an [`IdentityMatcher`](../merging.md) for merges. Because `Identity` uses object-identity equality (like `Track`), two `Identity` objects with the same `name` are distinct objects but still match by name — the key that survives serialization and cross-file merges. Pass `method="identity"` to instead require the same Python object:

```pycon
>>> import sleap_io as sio
>>> a1 = sio.Identity(name="mouse_A")
>>> a2 = sio.Identity(name="mouse_A")
>>> print(a1.matches(a2))  # same name -> same animal
>>> print(a1.matches(a2, method="identity"))  # distinct objects -> no match

```

!!! note "SLP persistence"
    The identity catalog and the per-detection identity links (`Instance.identity`/`identity_score`) persist in the `/identity` group at **format 2.5+**; the appearance vectors (`Instance.identity_embedding`) persist in the `/embeddings` group, also at **format 2.5+**. All are additive, so older readers ignore them and older files round-trip unchanged. See [Embeddings](embedding.md) and [Formats → SLP](../formats/slp.md).

---

## Instance3D

An `Instance3D` stores triangulated 3D keypoints in world coordinates. It is associated with an `InstanceGroup` that contains the source 2D instances used for triangulation.

Key properties:

- `points` — `(N, 3)` float64 array of 3D coordinates. `NaN` values indicate missing keypoints.
- `skeleton` — the `Skeleton` defining keypoint semantics.
- `score` — optional instance-level confidence score.
- `n_visible` — number of non-NaN keypoints.
- `is_empty` — whether all keypoints are missing.

`PredictedInstance3D` extends `Instance3D` with per-keypoint confidence scores in the `point_scores` field.

```pycon
>>> import numpy as np
>>> import sleap_io as sio
>>> skel = sio.Skeleton(nodes=["head", "tail"], edges=[("head", "tail")])
>>> pts = np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
>>> inst_3d = sio.Instance3D(points=pts, skeleton=skel)
>>> print(inst_3d.n_visible, inst_3d.is_empty)
>>> pred_3d = sio.PredictedInstance3D(
...     points=pts, skeleton=skel, point_scores=np.array([0.9, 0.8]),
... )
>>> print(pred_3d.point_scores)

```

Both `Instance3D` and `PredictedInstance3D` serialize into SLP v1.9+.

---

## Class relationships

The following diagram shows how the 3D and multi-view classes relate to each other and to the core data model.

``` mermaid
classDiagram
    direction LR

    class CameraGroup:::threed {
        +cameras
        +dict metadata
    }
    class Camera:::threed {
        +ndarray matrix
        +ndarray dist
        +tuple size
        +str name
    }
    class RecordingSession:::threed {
        +CameraGroup camera_group
        +frame_groups
        +videos
    }
    class FrameGroup:::threed {
        +int frame_idx
        +instance_groups
        +labeled_frames
    }
    class InstanceGroup:::threed {
        +instance_by_camera
        +Instance3D instance_3d
        +Identity identity
        +float score
    }
    class Video:::core {
        +str filename
    }
    class LabeledFrame:::core {
        +Video video
        +int frame_idx
    }
    class Instance:::core {
        +PointsArray points
        +Skeleton skeleton
        +Identity identity
        +float identity_score
        +Embedding identity_embedding
    }
    class Identity:::threed {
        +str name
        +dict metadata
    }
    class Instance3D:::threed {
        +ndarray points
        +Skeleton skeleton
        +float score
    }

    CameraGroup "1" *-- "0..*" Camera
    RecordingSession --> CameraGroup : uses
    RecordingSession "1" *-- "0..*" FrameGroup
    RecordingSession --> Video : links

    FrameGroup "1" *-- "0..*" InstanceGroup
    FrameGroup --> LabeledFrame : references

    InstanceGroup --> Instance : links
    InstanceGroup --> Camera : indexed by
    InstanceGroup --> Instance3D : has
    InstanceGroup --> Identity : references

    classDef threed fill:#7b1fa2,stroke:#6a1b9a,color:#fff
    classDef core fill:#546e7a,stroke:#37474f,color:#fff
```

---

## API reference

::: sleap_io.Camera

::: sleap_io.CameraGroup

::: sleap_io.RecordingSession

::: sleap_io.FrameGroup

::: sleap_io.InstanceGroup

::: sleap_io.Identity

::: sleap_io.Instance3D

::: sleap_io.PredictedInstance3D