SLP File Format

The .slp file format is SLEAP's native format for storing pose tracking data. It is built on top of HDF5, a hierarchical data format designed for storing and organizing large amounts of scientific data.

SLP files can contain:

• Video metadata and references to source video files (Video)
• Embedded images with configurable encoding (PNG, JPEG, or raw arrays)
• Skeleton definitions with nodes, edges, and symmetries (Skeleton)
• Labeled frames with user annotations and model predictions (LabeledFrame)
• Tracks for identity tracking across frames (Track)
• Suggestions for frames to label (SuggestionFrame)
• Recording sessions for multi-camera setups (RecordingSession)
• Bounding boxes for object detection and tracking (BoundingBox)
• Centroids for point tracking and TrackMate-style spot detection (Centroid)
• Regions of interest (ROIs) with vector geometry (ROI)
• Segmentation masks with run-length encoding (SegmentationMask)
• Label images for dense per-pixel instance segmentation (LabelImage)

In v0.7.0 the annotation types (ROI, SegmentationMask, BoundingBox, Centroid, and LabelImage) are abstract base classes; instances are always one of the User* or Predicted* subclasses. The on-disk dtype includes an is_predicted flag (and a score field for predicted variants).

HDF5 Layout

SLP files have the following hierarchical structure:

file.slp
├── /metadata                    # Group: Format and skeleton metadata
│   ├── @format_id               # Attribute: Format version (float, e.g., 1.4)
│   └── @json                    # Attribute: JSON metadata string (skeletons, nodes; ≤64 KB)
│
├── /videos_json                 # Dataset: Video metadata (variable-length bytes)
├── /tracks_json                 # Dataset: Track metadata (variable-length bytes)
├── /suggestions_json            # Dataset: Suggestions (variable-length bytes, optional)
├── /sessions_json               # Dataset: Recording sessions (calibration + video map + fg range; variable-length bytes, optional)
├── /session_data/               # Group: columnar session frame-group data (Format 2.8+, optional)
│   ├── frame_groups             # Dataset: struct (frame_idx, ig_start, ig_end)
│   ├── instance_groups          # Dataset: struct (identity_idx, score, instance_3d_score, pts3d_start/end, pts3d_predicted, member_start/end)
│   ├── instance_group_members   # Dataset: struct (camera, lf, inst) — columnarized camcorder_to_lf_and_inst_idx_map
│   ├── points_3d                # Dataset: float64 (N, 3) triangulated Instance3D points (chunked+gzip)
│   ├── pred_points_3d           # Dataset: float64 (N, 4) [x,y,z,score] PredictedInstance3D (chunked+gzip)
│   ├── frame_group_meta         # Dataset: per-frame-group JSON metadata blobs (optional)
│   └── instance_group_meta      # Dataset: per-instance-group JSON metadata blobs (optional)
├── /provenance_json             # Dataset: Provenance metadata (JSON bytes, optional)
│
├── /frames                      # Dataset: Labeled frame metadata (structured array)
├── /instances                   # Dataset: Instance metadata (structured array)
├── /points                      # Dataset: User-labeled points (structured array)
├── /pred_points                 # Dataset: Predicted points (structured array)
├── /negative_frames             # Dataset: Negative frame markers (optional)
│
├── /bboxes/                     # Group: Columnar bounding box storage (Format 2.0+)
│   ├── x1                       # Dataset: float64 top-left x
│   ├── y1                       # Dataset: float64 top-left y
│   ├── x2                       # Dataset: float64 bottom-right x
│   ├── y2                       # Dataset: float64 bottom-right y
│   ├── angle                    # Dataset: float64 rotation angle (radians)
│   ├── video                    # Dataset: int32 video index
│   ├── frame_idx                # Dataset: int64 frame index
│   ├── track                    # Dataset: int32 track index
│   ├── instance                 # Dataset: int32 instance index
│   ├── is_predicted             # Dataset: uint8 (0=user, 1=predicted)
│   ├── score                    # Dataset: float32 confidence score
│   ├── tracking_score           # Dataset: float32 tracking link confidence
│   ├── category                 # Dataset: vlen str category labels
│   ├── name                     # Dataset: vlen str name labels
│   └── source                   # Dataset: vlen str source labels
│
├── /centroids/                  # Group: Columnar centroid storage (v0.7.0+)
│   ├── x                        # Dataset: float64 x-coordinate
│   ├── y                        # Dataset: float64 y-coordinate
│   ├── z                        # Dataset: float64 z-coordinate (NaN for 2D)
│   ├── video                    # Dataset: int32 video index
│   ├── frame_idx                # Dataset: int64 frame index
│   ├── track                    # Dataset: int32 track index
│   ├── instance                 # Dataset: int32 instance index
│   ├── is_predicted             # Dataset: uint8 (0=user, 1=predicted)
│   ├── score                    # Dataset: float32 confidence score
│   ├── tracking_score           # Dataset: float32 tracking link confidence
│   ├── category                 # Dataset: vlen str category labels
│   ├── name                     # Dataset: vlen str name labels
│   └── source                   # Dataset: vlen str source labels
│
├── /rois                        # Dataset: ROI metadata (structured array, optional)
│   ├── @categories              # Attribute: JSON array (legacy fallback)
│   ├── @names                   # Attribute: JSON array (legacy fallback)
│   └── @sources                 # Attribute: JSON array (legacy fallback)
├── /roi_wkb                     # Dataset: Packed WKB geometry bytes (uint8 array, gzip-compressed)
├── /roi_categories              # Dataset: vlen string, one per ROI (Format 1.9+)
├── /roi_names                   # Dataset: vlen string, one per ROI (Format 1.9+)
├── /roi_sources                 # Dataset: vlen string, one per ROI (Format 1.9+)
│
├── /masks                       # Dataset: Mask metadata (structured array, optional)
│   ├── @categories              # Attribute: JSON array (legacy fallback)
│   ├── @names                   # Attribute: JSON array (legacy fallback)
│   └── @sources                 # Attribute: JSON array (legacy fallback)
├── /mask_rle                    # Dataset: Packed RLE bytes (uint8 array, gzip-compressed)
├── /mask_categories             # Dataset: vlen string, one per mask (Format 1.9+)
├── /mask_names                  # Dataset: vlen string, one per mask (Format 1.9+)
├── /mask_sources                # Dataset: vlen string, one per mask (Format 1.9+)
│
├── /mask_score_map_index        # Dataset: Score map index for masks (Format 1.9+)
├── /mask_score_maps             # Dataset: Packed score map data for masks (Format 1.9+)
├── /label_image_score_map_index # Dataset: Score map index for label images (Format 1.9+)
├── /label_image_score_maps      # Dataset: Packed score map data for label images (Format 1.9+)
│
├── /label_images                # Dataset: Label image metadata (Format 1.8+)
├── /label_image_objects         # Dataset: Per-object metadata (Format 1.8+)
├── /label_image_data            # Dataset: Pixel data (Format 1.8+, see below)
├── /label_image_sources         # Dataset: vlen string, one per label image (Format 1.9+)
├── /label_image_obj_categories  # Dataset: vlen string, one per object (Format 1.9+)
├── /label_image_obj_names       # Dataset: vlen string, one per object (Format 1.9+)
│
├── /video_crops                 # Dataset: JSON, virtual on-read crops (Format 2.3+, optional)
│
├── /identity/                   # Group: re-ID identity catalog (Format 2.5+, optional)
│   ├── name                     # Dataset: vlen utf-8 str, one per identity (catalog order)
│   ├── meta_owner               # Dataset: int32 identity index (EAV metadata; all three omitted if none)
│   ├── meta_key                 # Dataset: vlen utf-8 str metadata key
│   ├── meta_val                 # Dataset: vlen utf-8 str metadata value
│   └── links                    # Dataset: structured (owner_type, owner_id, identity_idx, identity_score)
│                                 #   one row per detection that has an identity
│
├── /categories/                 # Group: class/category catalog (Format 2.7+, optional)
│   ├── name                     # Dataset: vlen utf-8 str, one per category (catalog order)
│   ├── meta_owner               # Dataset: int32 category index (EAV metadata; all three omitted if none)
│   ├── meta_key                 # Dataset: vlen utf-8 str metadata key
│   ├── meta_val                 # Dataset: vlen utf-8 str metadata value
│   └── links                    # Dataset: structured (owner_type, owner_id, category_idx, category_score)
│                                 #   one row per detection that has a category
│
├── /embeddings/                 # Group: per-detection re-ID embeddings (Format 2.5+, optional)
│   ├── vectors                  # Dataset: float (N, D), chunked + gzip (all rows share D)
│   ├── owner_type               # Dataset: uint8 (0=instance, 2=centroid, 3=mask, 4=bbox, 5=roi)
│   ├── owner_id                 # Dataset: int64 (global instance_id or per-modality list index)
│   ├── category_vectors         # Dataset: float (M, D'), parallel category embeddings (Format 2.7+, optional)
│   ├── category_owner_type      # Dataset: uint8 category-embedding join (Format 2.7+, optional)
│   └── category_owner_id        # Dataset: int64 category-embedding join (Format 2.7+, optional)
│
├── /event_types/                # Group: frame-spanning event catalog (Format 2.6+, optional)
│   ├── name                     # Dataset: vlen utf-8 str, one per event type (catalog order)
│   ├── description              # Dataset: vlen utf-8 str (omitted if all descriptions empty)
│   ├── meta_owner               # Dataset: int32 event-type index (EAV metadata; all three omitted if none)
│   ├── meta_key                 # Dataset: vlen utf-8 str metadata key
│   └── meta_val                 # Dataset: vlen utf-8 str metadata value
│
├── /events/                     # Group: frame-spanning event annotations (Format 2.6+, optional)
│   ├── video                    # Dataset: int64 video index (-1 = none)
│   ├── start_frame / end_frame  # Dataset: int64 inclusive interval bounds
│   ├── type                     # Dataset: int64 index into /event_types (-1 = none)
│   ├── subject_kind/target_kind # Dataset: int8 (0=none/self, 1=track, 2=identity)
│   ├── subject_idx/target_idx   # Dataset: int64 index into tracks or identities (-1 = none)
│   ├── is_predicted             # Dataset: bool (user vs predicted)
│   ├── score                    # Dataset: float64 scalar confidence, NaN=unset (omitted if none set)
│   ├── name / source            # Dataset: vlen utf-8 str free-text
│   ├── scores                   # Dataset: float32 flat framewise traces, chunked+gzip (CSR; omitted if none)
│   ├── score_offsets            # Dataset: int64 (n_events+1) CSR offsets into scores
│   └── meta_owner/meta_key/meta_val  # Dataset: per-event EAV metadata (omitted if none)
│
└── /video{N}/                   # Group: Per-video embedded data (one per video)
    ├── /video                   # Dataset: Embedded image data
    │   ├── @format              # Attribute: "png", "jpg", or "hdf5"
    │   ├── @channel_order       # Attribute: "RGB" or "BGR"
    │   ├── @frames              # Attribute: Total frames in source video
    │   ├── @height              # Attribute: Frame height
    │   ├── @width               # Attribute: Frame width
    │   ├── @channels            # Attribute: Number of channels
    │   └── @fps                 # Attribute: Frames per second (optional)
    ├── /frame_numbers           # Dataset: Embedded frame indices (int array)
    └── /source_video/           # Group: Source video metadata
        ├── @json                # Attribute: JSON with source video info (≤64 KB)
        └── /json                # Dataset: JSON fallback when info exceeds 64 KB (optional)

Source video metadata over 64 KB

Source video metadata is normally stored in the source_video/@json attribute. If it would exceed HDF5's 64 KB attribute limit (e.g. an image sequence with many thousands of filenames), it is written to a source_video/json dataset instead and a warning is emitted; readers prefer the dataset when present.

Embedded image storage and format

For encoded formats (png/jpg), the /video dataset is stored uncompressed and contiguous: the bytes are already entropy-coded, so gzip gained almost nothing while its chunked storage made row-by-row writes pathologically slow. Only the raw-array hdf5 format is gzip-compressed.

When the source is an image sequence (ImageVideo) of PNG/JPEG files, the original file bytes are copied verbatim — no decode/re-encode cycle — and the stored @format follows the source (e.g. jpg) rather than the requested default. This is faster, lossless (no added JPEG artifacts), and typically much smaller than re-encoding to PNG.

Core Datasets

Dataset	Shape	Dtype	Description
points	(N,)	structured	User-labeled point coordinates
pred_points	(N,)	structured	Predicted point coordinates with scores
instances	(N,)	structured	Instance metadata linking to points
frames	(N,)	structured	Frame metadata linking to instances

Metadata Datasets

Dataset	Type	Description
videos_json	bytes[]	JSON array of video metadata
tracks_json	bytes[]	JSON array of track definitions
suggestions_json	bytes[]	JSON array of suggested frames (optional)
sessions_json	bytes[]	JSON array of recording sessions: calibration + camera→video map + session metadata + (Format 2.8+) an fg_start/fg_end range into session_data/frame_groups; ≤2.7 files instead inline frame_group_dicts (optional)
session_data/	group	Columnar session frame-group data: frame_groups / instance_groups / instance_group_members struct tables, points_3d (N,3) / pred_points_3d (N,4) chunked float matrices, and optional per-row frame_group_meta / instance_group_meta JSON blobs (Format 2.8+, present only when a session has frame groups)
identity/	group	re-ID identity catalog: name dataset + optional meta_owner/meta_key/meta_val EAV metadata + per-detection links (Format 2.5+, optional)
categories/	group	class/category catalog: name dataset + optional meta_owner/meta_key/meta_val EAV metadata + per-detection links (Format 2.7+, optional)
embeddings/	group	Per-detection re-ID embeddings: vectors (N, D) + owner_type/owner_id join columns; plus optional parallel category_vectors (M, D') + category_owner_type/category_owner_id for category embeddings (Format 2.5+; category datasets 2.7+, optional)
event_types/	group	Frame-spanning event catalog: name dataset + optional description + meta_* EAV metadata (Format 2.6+, optional)
events/	group	Frame-spanning event annotations: columnar per-field datasets + ragged CSR scores/score_offsets framewise traces (Format 2.6+, optional)
provenance_json	bytes	JSON object of provenance metadata (optional)

Provenance storage and the 64 KB metadata limit

The skeletons and nodes live in the metadata/@json HDF5 attribute. HDF5 caps any single attribute at 64 KB (65,536 bytes), so provenance — which can grow without bound (e.g. merge_history accrues a record on every Labels merge()) — is stored in its own /provenance_json dataset, which has no such limit.

Reading is backward compatible: provenance is read from /provenance_json when present, otherwise from the provenance key inside the metadata/@json attribute (the legacy layout). When provenance is small it is also mirrored into the attribute so that older readers, which only look at the attribute, still see it.

As a final safeguard, if the metadata/@json blob would still exceed 64 KB after relocating provenance, its largest droppable top-level keys are removed (largest-first) until it fits and a UserWarning names the dropped keys. This is lossless — provenance is already in /provenance_json and the other droppable keys are empty placeholders whose data lives in their own datasets — so saving never fails on oversized metadata.

Forward compatibility: preserving unknown datasets

Saving a .slp truncates the file and rebuilds it from the in-memory model, so any top-level dataset/group the writer does not recognize is dropped. This means an older sleap-io version round-tripping a file written by a newer version would silently lose the newer version's additions.

To guard against this, pass preserve_unknown=True to save_slp():

labels = sio.load_slp("from_newer_version.slp")
sio.save_slp(labels, "out.slp", preserve_unknown=True)

Top-level members in the source file (labels.provenance["filename"]) that are not part of the known schema are copied verbatim into the saved file, after the known sections are written (regenerated data always wins on a name clash). It is opt-in (default False) and best-effort: it requires the source file to still exist and be readable HDF5. Per-video embedded groups (video0, video1, ...) and all datasets listed above are part of the known schema and are regenerated, not carried over.

Videos

Video metadata is stored in the /videos_json dataset as an array of JSON strings. Each video entry contains:

{
    "filename": "path/to/video.mp4",
    "backend": {
        "type": "MediaVideo",
        "shape": [1000, 480, 640, 3],
        "filename": "path/to/video.mp4",
        "grayscale": false,
        "bgr": true,
        "fps": 30.0
    },
    "source_video": null
}

Backend Types

Type	Description	Key Fields
MediaVideo	Standard video files (mp4, avi, mov, etc.)	filename, bgr, fps
HDF5Video	Embedded frames in HDF5	dataset, input_format, has_embedded_images
ImageVideo	Image sequences	filename, filenames (list)
TiffVideo	TIFF stacks	filename, keep_open

Source Video Lineage

Videos can have a source_video field that tracks the original video when frames are embedded:

{
    "filename": ".",
    "backend": {
        "type": "HDF5Video",
        "dataset": "video0/video"
    },
    "source_video": {
        "filename": "original.mp4",
        "backend": { ... }
    }
}

This creates a chain of provenance, allowing the original video to be restored when extracting embedded data.

Virtual Crops (Format 2.3+)

A virtually-cropped video stores its crop rect in a dedicated top-level /video_crops dataset (a single JSON string), written only when at least one video is cropped. Its /videos_json entry describes the uncropped source backend, and its source_video is the uncropped original — so a reader that does not understand /video_crops simply loads the full-frame source (a graceful, lossy degrade).

[
  {"video": 2, "crop": [128, 96, 384, 352], "fill": 0},
  {"video": 5, "crop": [0, 0, 256, 256], "fill": 0}
]

• video — integer index into the /videos_json video list.
• crop — [x1, y1, x2, y2] in source pixel coordinates, x2/y2 exclusive.
• fill — out-of-bounds fill value (regions outside the source are padded, not clamped).

The presence of any crop bumps format_id to 2.3. Files with no crops are byte-identical to earlier versions (no /video_crops dataset, no version bump).

Embedded Images

Frames can be embedded directly in SLP files for portability. Embedded frames are stored in /video{N}/ groups.

Encoding Formats

Format	Storage	Compression	Notes
png	int8[]	Lossless PNG	Default, best quality
jpg	int8[]	Lossy JPEG	Smaller files, some quality loss
hdf5	Raw array	Optional gzip	No encoding overhead, large files

Frame Selection Modes

When saving with embedded frames, the embed parameter controls which frames to include:

Mode	Description
None	Re-embed existing embedded frames
True / "all"	All labeled frames and suggestions
"user"	Only user-labeled frames
"suggestions"	Only suggested frames
"user+suggestions"	Both user and suggested frames
"source"	No embedding, restore source video
list[(Video, int)]	Custom list of (video, frame_idx) pairs

Channel Order

The channel_order attribute (introduced in format 1.4) tracks the color channel ordering:

• OpenCV encoding: BGR ("BGR")
• imageio encoding: RGB ("RGB")
• Raw HDF5 arrays: RGB ("RGB")

This ensures correct color reproduction when reading embedded frames.

Skeletons

Skeleton definitions are stored in the /metadata group's json attribute. The metadata JSON contains both a global node list and skeleton definitions that reference Nodes by index.

JSON Format (SLP Files)

{
    "version": "2.0.0",
    "skeletons": [
        {
            "directed": true,
            "graph": {"name": "Skeleton-0", "num_edges_inserted": 5},
            "links": [
                {"source": 0, "target": 1, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}},
                {"source": 1, "target": 2, "type": {"py/id": 1}}
            ],
            "nodes": [{"id": 0}, {"id": 1}, {"id": 2}]
        }
    ],
    "nodes": [
        {"name": "head", "weight": 1.0},
        {"name": "thorax", "weight": 1.0},
        {"name": "abdomen", "weight": 1.0}
    ]
}

Edge Types

Edges are encoded with a type field using Python's pickle-style encoding:

Type ID	Meaning	Description
1	Regular edge	Skeletal connection between nodes
2	Symmetry edge	Bilateral symmetry relationship

The first occurrence of each type uses the full py/reduce encoding; subsequent occurrences use py/id references.

Symmetries

Symmetry relationships define bilateral pairings (e.g., left/right body parts). They are stored as edges with type 2:

{
    "source": 3,
    "target": 4,
    "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [2]}]}
}

Symmetry Deduplication

Legacy SLEAP files may store symmetries bidirectionally. The decoder automatically deduplicates them.

YAML Format

In addition to the JSON format stored in SLP files, sleap-io supports a simplified YAML format for Skeleton definitions. This format is more human-readable and easier to edit manually.

Preferred Format

The YAML format is the preferred format for skeleton definitions in sleap-nn and new tooling. The JSON format will be maintained for backwards compatibility with existing SLP files.

Structure

skeleton_name:
  nodes:
    - name: head
    - name: thorax
    - name: abdomen
  edges:
    - source:
        name: head
      destination:
        name: thorax
    - source:
        name: thorax
      destination:
        name: abdomen
  symmetries:
    - - name: left_wing
      - name: right_wing

Fields

Field	Type	Description
nodes	list[dict]	List of Node definitions with name key
edges	list[dict]	List of Edge definitions with source and destination
symmetries	list[list]	List of Symmetry pairs, each as a list of two node references

Multiple Skeletons

The YAML format supports multiple skeletons in a single file, with skeleton names as top-level keys:

fly:
  nodes:
    - name: head
    - name: thorax
  edges:
    - source: { name: head }
      destination: { name: thorax }
  symmetries: []

mouse:
  nodes:
    - name: nose
    - name: spine
  edges:
    - source: { name: nose }
      destination: { name: spine }
  symmetries: []

API Functions

Use the following functions to work with YAML skeletons:

• encode_yaml_skeleton - Encode skeleton(s) to YAML string
• decode_yaml_skeleton - Decode skeleton(s) from YAML string or file

Metadata

The /metadata group stores format information and serialized metadata about the labels.

Attributes

Attribute	Type	Description
format_id	float	Format version (e.g., 1.4)
json	bytes	JSON-encoded metadata string

JSON Structure

The json attribute contains a JSON object with the following fields:

{
    "version": "2.0.0",
    "skeletons": [...],
    "nodes": [...],
    "videos": [],
    "tracks": [],
    "suggestions": [],
    "negative_anchors": {},
    "provenance": {
        "sleap_version": "1.3.4",
        "filename": "labels.slp"
    }
}

Field	Type	Description
version	string	SLEAP software version
skeletons	array	Skeleton graph definitions (see Skeletons)
nodes	array	Node definitions with names and weights
videos	array	Empty (stored in /videos_json dataset)
tracks	array	Empty (stored in /tracks_json dataset)
suggestions	array	Empty (stored in /suggestions_json dataset)
negative_anchors	object	Negative sample anchors for training
provenance	object	File origin and creation metadata

Provenance

The provenance field tracks the origin and history of the labels file:

Key	Type	Set by	Description
sleap_version	string	SLEAP	SLEAP version that created the file
filename	string	load_slp	Original filename (set on load)
source_labels	string	split / extract	Path to parent labels file
merge_history	array	merge	Records of merge operations (timestamp, source, strategy)
custom	any	user	Additional user-defined provenance data

Custom Provenance

The provenance dictionary can contain arbitrary key-value pairs for tracking custom metadata. Values must be JSON-serializable. Path objects are auto-converted to strings on save.

Recording segmentation model parameters

When using segmentation tools like Cellpose, record the model parameters in provenance for reproducibility:

import sleap_io as sio

labels = sio.Labels(label_images=label_images)
labels.provenance["segmentation_model"] = "cellpose"
labels.provenance["cellpose_model_type"] = "cyto3"
labels.provenance["cellpose_diameter"] = 30
labels.provenance["cellpose_cellprob_threshold"] = 0.0
labels.save("segmentation.slp")

# Later, verify parameters:
loaded = sio.load_slp("segmentation.slp")
print(loaded.provenance["cellpose_diameter"])  # 30

Tracks

Tracks enable identity tracking of individual animals across frames. Track metadata is stored in the /tracks_json dataset as an array of JSON strings.

JSON Structure

Each track is stored as a two-element JSON array:

[0, "track_name"]

Index	Type	Description
0	int	Spawned frame index (reserved, currently always 0)
1	string	Track name for identification

Example

[0, "mouse_1"]
[0, "mouse_2"]
[0, "female"]

Instance Linking

Instances reference tracks by index in the /instances dataset:

• track = 0 → First track in /tracks_json
• track = 1 → Second track in /tracks_json
• track = -1 → Untracked instance

Track Identity

Tracks are compared by object identity, not name. Two tracks with the same name are considered different unless they are the same object. This allows multiple tracks to share a name if needed.

Identities

Identity objects represent ground-truth animal identities that persist across sessions and videos. An Identity has exactly two fields: a human-readable name (a string, not required to be unique) and a free-form metadata mapping of string keys to string values. Identities are matched by name by default (see Identity.matches), so two detections labeled "mouse_A" refer to the same animal across separately loaded files and merges.

Labels.identities is the catalog of these objects — a list, like Labels.tracks — auto-collected in first-seen order from the detections and instance groups. Identities are stored in the optional /identity group (Format 2.5+).

No dedicated color

There is no color field on an Identity. If a visualization color is desired, store it as a conventional metadata entry such as metadata["color"] = "#e6194b"; it persists like any other metadata key. Coloring by identity uses the palette index into Labels.identities order (identical to color-by-track), not a per-identity color.

Catalog (/identity/name)

The name dataset holds one variable-length UTF-8 string per identity, in catalog order:

f["/identity/name"][:]  # ["mouse_A", "mouse_B"]

Metadata (/identity/meta_owner, /identity/meta_key, /identity/meta_val)

Free-form per-identity metadata is stored as an entity-attribute-value (EAV) table: three parallel datasets with one row per (identity, key, value) triple.

Dataset	Type	Description
meta_owner	int32	Index of the owning identity in /identity/name
meta_key	vlen str	Metadata key
meta_val	vlen str	Metadata value

All three datasets are omitted entirely when no identity carries any metadata. A conventional key such as metadata["color"] = "#e6194b" rides here like any other entry.

Per-Detection Linking (/identity/links)

Per-detection identity assignments are stored in the /identity/links structured dataset, with one row per detection that carries an identity. Each row joins a detection — identified by an owner_type/owner_id pair (the same join scheme as the /embeddings group) — to an entry in the identity catalog:

Column	Type	Description
owner_type	uint8	Detection modality code (shared OWNER_* codes: 0=instance, 2=centroid, 3=mask, 4=bbox, 5=roi)
owner_id	int64	Per-owner-type positional id: the global instance id for instance owners, or the global per-modality list index for centroid / mask / bbox / ROI owners (each enumeration order over frames then the modality, matching write_lfs / write_masks / write_centroids / write_bboxes / write_rois; static ROIs follow frame ROIs)
identity_idx	int32	Index into /identity/name
identity_score	float32	Identity-assignment score (NaN if unrecorded)

This is additive: old readers ignore the group, and detections without an identity simply have no row. Each detection carries a single identity_embedding slot; those vectors, when present, live in the /embeddings group — see Embeddings.

Instance Group Linking

InstanceGroups reference identities by index via the identity_idx field in the session JSON:

{
    "camcorder_to_lf_and_inst_idx_map": {...},
    "identity_idx": 0
}

The index corresponds to the position in /identity/name.

Optional Group

The /identity group is only written when the Labels object contains identities. On read, a missing group defaults to an empty catalog.

Categories

Category objects name the class a detection belongs to (e.g. "female_fly", "fur_shaved"), assigned by classification or re-ID. A Category has exactly two fields: a human-readable name (a string, not required to be unique) and a free-form metadata mapping of string keys to string values. Categories are matched by name by default (see Category.matches), so two detections labeled "female_fly" refer to the same class across separately loaded files and merges.

Labels.categories is the catalog of these objects — a list, like Labels.identities — auto-collected in first-seen order from the detections and instance groups on save. The /categories group is a fully self-contained mirror of /identity (its own catalog, EAV metadata, and per-detection links), introduced at Format 2.7 (additive; older readers ignore it and category-free files are byte-identical).

Legacy category string

Older files stored a free-form class label on ROI / bounding-box / centroid / mask annotations (the roi_categories / mask_categories / bbox / centroid category datasets). Those remain unchanged; the new /categories catalog is the first-class promotion of that concept and is written separately. See Categories.

Catalog (/categories/name)

The name dataset holds one variable-length UTF-8 string per category, in catalog order:

f["/categories/name"][:]  # ["female_fly", "male_fly"]

Metadata (/categories/meta_owner, /categories/meta_key, /categories/meta_val)

Free-form per-category metadata is stored as an entity-attribute-value (EAV) table, exactly like /identity: three parallel datasets with one row per (category, key, value) triple.

Dataset	Type	Description
meta_owner	int32	Index of the owning category in /categories/name
meta_key	vlen str	Metadata key
meta_val	vlen str	Metadata value

All three datasets are omitted entirely when no category carries any metadata.

Per-Detection Linking (/categories/links)

Per-detection category assignments are stored in the /categories/links structured dataset, with one row per detection that carries a category. Each row joins a detection — identified by an owner_type/owner_id pair (the same join scheme as /identity/links and /embeddings) — to an entry in the category catalog:

Column	Type	Description
owner_type	uint8	Detection modality code (shared OWNER_* codes: 0=instance, 2=centroid, 3=mask, 4=bbox, 5=roi)
owner_id	int64	Per-owner-type positional id (same enumeration as /identity/links: global instance id for instance owners, or global per-modality list index for centroid / mask / bbox / ROI owners; static ROIs follow frame ROIs)
category_idx	int32	Index into /categories/name
category_score	float32	Category-assignment score (NaN if unrecorded)

This is additive: old readers ignore the group, and detections without a category simply have no row. Each detection also carries a single category_embedding slot; those vectors, when present, live alongside identity vectors in the /embeddings group as parallel category_* datasets — see Embeddings.

Optional Group

The /categories group is only written when the Labels object contains categories. On read, a missing group defaults to an empty catalog.

Embeddings

Per-detection appearance / re-identification Embedding vectors are stored in the optional /embeddings group (Format 2.5+). An Embedding is a bare value object wrapping a single 1-D feature vector of shape (D,) (its dim property returns D); each detection — an instance, Centroid, BoundingBox, SegmentationMask, or ROI — carries at most one, in its identity_embedding slot.

The group is a single columnar struct-of-arrays. Row i of all three datasets describes the same detection:

Dataset	Shape	Dtype	Description
vectors	(N, D)	float	Stacked embedding vectors, one row per detection. Chunked so whole rows stay within a chunk (a single-detection read touches exactly one chunk) and gzip-compressed. The floating dtype is preserved from the source vectors (typically float32)
owner_type	(N,)	uint8	Detection modality code (shared OWNER_* codes: 0=instance, 2=centroid, 3=mask, 4=bbox, 5=roi)
owner_id	(N,)	int64	Per-owner-type positional id: the global instance id for instance owners, or the global per-modality list index for centroid / mask / bbox / ROI owners (matching write_lfs / write_masks / write_centroids / write_bboxes / write_rois; static ROIs follow frame ROIs)

All identity vectors in a file share a single dimensionality D; a mix of dimensionalities is rejected at write time. The owner_type/owner_id join is identical to the one used by /identity/links, so a detection's identity link and its embedding are matched by the same pair.

Category embeddings (parallel datasets, Format 2.7+)

Each detection can also carry a category_embedding (the appearance vector its class was predicted from). These are stored in the same /embeddings group as a set of parallel, independent datasets, added at Format 2.7:

Dataset	Shape	Dtype	Description
category_vectors	(M, D')	float	Stacked category embedding vectors, chunked + gzip-compressed
category_owner_type	(M,)	uint8	Detection modality code (same shared OWNER_* codes)
category_owner_id	(M,)	int64	Same per-owner-type positional id join as the identity columns

Category vectors are kept independent of the identity vectors/owner_type/owner_id datasets rather than sharing one table, so the two embedding kinds need not share dimensionality (D may differ from D'). A file with only identity embeddings has no category_* datasets and stays byte-identical to a pre-2.7 file. On read, absent category_* datasets simply mean no detection receives a category embedding.

Optional Group

The /embeddings group is only written when at least one detection carries an identity_embedding or a category_embedding and embedding vectors are being persisted. Passing save_slp(labels, path, save_embedding_vectors=False) writes the identity and category links but skips the /embeddings group entirely, so the (potentially large) vectors are omitted while the identity/category assignments are kept. On read, a missing group means no detection receives an embedding.

Events

Frame-spanning Event annotations — behavior bouts, stimulus epochs, review flags, or any labeled time range — and their EventType catalog are stored in the optional /event_types and /events groups (Format 2.6+). Unlike every other annotation, an event has a temporal extent (an inclusive [start_frame, end_frame] interval) and lives on Labels.events rather than on any LabeledFrame. Bumps format_id to 2.6; both groups are additive, so old readers ignore them and event-free files are byte-identical.

Catalog (/event_types)

Mirrors /identity: a name string dataset (one per type, catalog order), an optional description string dataset (omitted when every description is empty), and an optional meta_owner / meta_key / meta_val EAV metadata table (omitted when no type carries metadata). Labels.event_types is the catalog, auto-collected and deduped by name from labels.events on save.

Annotations (/events)

A columnar struct-of-arrays (one dataset per field, like /bboxes); row i of every dataset describes event i:

Dataset	Dtype	Description
video	int64	Index into the videos list (-1 = none)
start_frame / end_frame	int64	Inclusive interval bounds (stored int64, read defensively)
type	int64	Index into /event_types (-1 = none)
subject_kind / target_kind	int8	Participant kind: 0=none/self, 1=track, 2=identity
subject_idx / target_idx	int64	Index into tracks (kind 1) or identities (kind 2) (-1 = none)
is_predicted	bool	User vs. predicted event
score	float64	Scalar event-level confidence; NaN = unset. Omitted when no event sets one
name / source	vlen str	Free-text
meta_owner / meta_key / meta_val	mixed	Per-event EAV metadata (omitted when none)

Framewise PredictedEvent.scores traces are variable-length, so they are stored as a ragged CSR pair (the same idiom as mask RLE / ROI WKB): a flat scores (float32, chunked + gzip) dataset plus an score_offsets (int64, length n_events + 1) dataset. Event i's trace is scores[score_offsets[i]:score_offsets[i+1]]; an event with no trace gets a zero-length slice. Both trace datasets are omitted entirely when no event has a framewise trace. The scalar score and the framewise scores are independent — a predictor may set either, both, or neither.

Optional Groups

Both groups are only written when the Labels object has events / event types; every column is presence-guarded so unused features cost zero bytes. On read, missing groups default to empty events / event_types lists.

Suggestions

Suggestions indicate frames that should be labeled, typically generated by active learning algorithms or manual selection. Suggestion metadata is stored in the /suggestions_json dataset.

JSON Structure

Each suggestion is stored as a JSON object:

{
    "video": "0",
    "frame_idx": 42,
    "group": 0
}

Field	Type	Description
video	string	Video index (as string)
frame_idx	int	Frame index within the video
group	int	Suggestion group ID (default: 0)

Groups

The group field enables organizing suggestions into batches, useful for:

• Separating suggestions by generation method
• Tracking labeling progress across multiple sessions
• Grouping frames by difficulty or priority

Optional Dataset

The /suggestions_json dataset is optional. Files without suggestions will not contain this dataset, and the reader returns an empty list when it's missing.

Sessions

Recording sessions store multi-camera calibration data and synchronized frame groups. Small, bounded session data (calibration + the camera→video map + session-level metadata) lives in the /sessions_json dataset; the unbounded per-frame payload (frame groups, instance groups, 3D points, and their metadata) lives in the columnar /session_data group (Format 2.8+).

Older files (≤ 2.7) store the entire session — including every frame group and its inline 3D points — as one JSON string per session inside /sessions_json. This did not scale: a single multi-view project could produce a hundreds-of-MB string dominated by 3D point text, which also exceeds the ~0.45 GB limit for reading an HDF5 variable-length string in JS/WASM consumers. Format 2.8 moves that numeric payload into typed, chunked datasets referenced by row range (mirroring how 2D /points are referenced from /instances). The reader accepts both layouts (see Backward Compatibility).

Slim sessions_json (Format 2.8+)

Each session is stored as a compact JSON object holding only bounded, O(cameras) data plus a range into /session_data/frame_groups:

{
    "calibration": {
        "cam_0": {
            "name": "Camera 1",
            "size": [1080, 1920],
            "matrix": [[...], [...], [...]],
            "distortions": [...],
            "rotation": [...],
            "translation": [...]
        },
        "cam_1": {...},
        "metadata": {}
    },
    "camcorder_to_video_idx_map": {
        "0": 0,
        "1": 1
    },
    "fg_start": 0,
    "fg_end": 120000
}

fg_start/fg_end are a half-open range into the /session_data/frame_groups dataset. Session-level metadata is merged in at the top level (as before).

The /session_data group (Format 2.8+)

All O(frames × instances × nodes) session data is stored columnar, so it streams and compresses like the rest of the file:

Dataset	Kind	Columns / shape
frame_groups	compound	frame_idx (i8), ig_start (u8), ig_end (u8) — range into instance_groups
instance_groups	compound	identity_idx (i4, -1 if none), score (f8, NaN if none), instance_3d_score (f8, NaN if none), pts3d_start/pts3d_end (i8, -1 if no 3D), pts3d_predicted (u1), member_start/member_end (u8) — range into instance_group_members
instance_group_members	compound	camera (u4), lf (i8), inst (u4) — the columnarized camcorder_to_lf_and_inst_idx_map
points_3d	float64 (N, 3)	Triangulated Instance3D coordinates, chunked + gzip, sliced by pts3d_start:pts3d_end when pts3d_predicted == 0
pred_points_3d	float64 (N, 4)	[x, y, z, score] for PredictedInstance3D, chunked + gzip, sliced when pts3d_predicted == 1
frame_group_meta	str[]	One json.dumps(metadata) blob per frame group (presence-guarded; omitted when all empty)
instance_group_meta	str[]	One json.dumps(metadata) blob per instance group (presence-guarded)

NaN rows in points_3d/pred_points_3d denote missing/unresolved keypoints and round-trip natively. The group (and the 2.8 version bump) is only written when a session actually has frame groups, so single-view and session-free files are byte-identical to before.

Legacy inline structure (≤ 2.7)

Files written before 2.8 carry an extra frame_group_dicts key inside each /sessions_json blob and have no /session_data group:

{
    "calibration": {...},
    "camcorder_to_video_idx_map": {...},
    "frame_group_dicts": [...]
}

Each frame-group dict nests instance_groups, each with camcorder_to_lf_and_inst_idx_map and inline 3D (points, instance_3d_score, instance_3d_point_scores).

Calibration

Camera calibration data is stored per camera with these fields:

Field	Type	Shape	Description
name	string	-	Camera identifier
size	int[]	(2,)	Image dimensions [height, width]
matrix	float[][]	(3, 3)	Intrinsic camera matrix
distortions	float[]	(5,)	Radial-tangential distortion coefficients [k1, k2, p1, p2, k3]
rotation	float[]	(3,)	Rotation vector (axis-angle representation)
translation	float[]	(3,)	Translation vector

Camera-Video Mapping

The camcorder_to_video_idx_map object maps camera indices to video indices in /videos_json:

{
    "0": 0,
    "1": 1
}

This links each camera in the calibration to its corresponding video.

Frame Groups

Frame groups synchronize labeled frames across multiple cameras at the same time point. Each frame group contains:

• A frame index identifying the synchronized time point
• Instance groups linking instances across camera views
• References to LabeledFrame objects by index

Instance groups may also contain 3D reconstruction data. In Format 2.8+ this is columnar (instance_groups.pts3d_* → points_3d/pred_points_3d, instance_groups.identity_idx, instance_group_meta); in legacy files it is inline on each instance-group dict (points, instance_3d_score, instance_3d_point_scores, identity_idx).

Sessions backward compatibility

The reader dispatches on the presence of the /session_data group:

• /session_data present (Format 2.8+): frame groups are reconstructed from the columnar tables using each session's fg_start/fg_end range.
• absent (≤ 2.7): frame groups are reconstructed from the inline frame_group_dicts (including inline 3D points).

Forward compatibility: a reader older than 2.8 opening a 2.8 file sees the slim sessions_json (calibration + video map only) and ignores /session_data, so it loads the session's calibration but not its frame groups/3D — data is never corrupted, only gracefully absent.

Lazy loads capture the raw sessions_json bytes and /session_data arrays and copy them verbatim on save (no frame materialization), so a lazy re-save preserves frame groups and 3D losslessly as long as the video list is unchanged.

Optional Dataset

The /sessions_json dataset is optional (always written when sessions exist, even without frame groups). The /session_data group is only present for sessions that have frame groups (Format 2.8+). Files without multi-camera sessions contain neither.

Instances

Instances represent individual animals or objects in a frame. They are stored in the /instances dataset as a structured array.

Instance Dtype

instance_dtype = np.dtype([
    ("instance_id", "i8"),        # Unique instance identifier
    ("instance_type", "u1"),      # 0=USER, 1=PREDICTED
    ("frame_id", "u8"),           # Index into frames dataset
    ("skeleton", "u4"),           # Index into skeletons list
    ("track", "i4"),              # Index into tracks list (-1 if untracked)
    ("from_predicted", "i8"),     # Parent prediction ID (-1 if none)
    ("score", "f4"),              # Prediction score (0.0 for user)
    ("point_id_start", "u8"),     # Start index in points array
    ("point_id_end", "u8"),       # End index (exclusive)
    ("tracking_score", "f4"),     # Tracking confidence (format >= 1.2)
])

Instance Types

Type	Value	Data Model Class	Description
USER	0	Instance	User-labeled annotation
PREDICTED	1	PredictedInstance	Model prediction

Instance Linking

The from_predicted field links user Instances to their source PredictedInstances, enabling tracking of corrections made to model outputs.

Points

Point coordinates are stored in separate datasets for user-labeled and predicted instances.

User Points (/points)

point_dtype = np.dtype([
    ("x", "f8"),        # X coordinate
    ("y", "f8"),        # Y coordinate
    ("visible", "?"),   # Is point visible
    ("complete", "?"),  # Is point marked complete
])

Predicted Points (/pred_points)

predicted_point_dtype = np.dtype([
    ("x", "f8"),        # X coordinate
    ("y", "f8"),        # Y coordinate
    ("visible", "?"),   # Is point visible
    ("complete", "?"),  # Is point marked complete
    ("score", "f8"),    # Prediction confidence
])

Coordinate System

Coordinate System Change

Format 1.1 changed the coordinate system from pixel corner to pixel center.

Format	Origin	Notes
< 1.1	Top-left corner of pixel at (0, 0)	Legacy
>= 1.1	Center of pixel at (0, 0)	Current

When reading format < 1.1 files, the reader applies a -0.5 offset to convert coordinates.

Labeled Frames

LabeledFrames are stored in the /frames dataset, linking video frames to their instances.

Frame Dtype

frame_dtype = np.dtype([
    ("frame_id", "u8"),           # Unique frame identifier
    ("video", "u4"),              # Video index or sparse video ID
    ("frame_idx", "u8"),          # Frame index within video
    ("instance_id_start", "u8"),  # Start index in instances array
    ("instance_id_end", "u8"),    # End index (exclusive)
])

Video ID Mapping

Modern SLP files use sequential video indices (0, 1, 2, ...), but legacy files may contain sparse video IDs derived from the embedded video group names (e.g., 0, 15, 29). The reader handles both cases transparently.

Negative Frames

Negative frames are frames explicitly marked as containing no instances (pure background). They are valuable for training, helping models learn what backgrounds look like without any animals present. Negative frames are distinct from "empty frames" which had instances that were deleted.

Storage

Negative frames are stored in two places:

1. /frames dataset: Like all labeled frames, negative frames have a row in the /frames dataset. They have instance_id_start == instance_id_end (empty instance range).

2. /negative_frames dataset: A sidecar dataset that marks which empty frames are intentionally negative vs accidentally empty.

Negative Frames Dtype

negative_frames_dtype = np.dtype([
    ("video_id", "u4"),   # Sparse video ID (same as in /frames)
    ("frame_idx", "u8"),  # Frame index within video
])

Example

# A file with two negative frames
negative_frames = [
    (0, 42),   # Video 0, frame 42 is negative
    (0, 100),  # Video 0, frame 100 is negative
]

Data Model Integration

When loading SLP files, the is_negative attribute is set on LabeledFrame objects:

import sleap_io as sio

labels = sio.load_slp("labels.slp")

# Access negative frames
negative = labels.negative_frames  # List of LabeledFrames with is_negative=True

# Check if a frame is negative
for lf in labels:
    if lf.is_negative:
        print(f"Frame {lf.frame_idx} is a negative frame")

# Negative frames are included in user_labeled_frames for training export
user_frames = labels.user_labeled_frames  # Includes negative frames

clean() Behavior

When calling Labels.clean(frames=True), negative frames are preserved even though they have no instances. Only non-negative empty frames are removed. Frames that contain spatial annotations (centroids, bounding boxes, masks, ROIs, or label images) but no instances are also preserved.

When tracks=True (the default), clean() also removes spatial annotations within frames that reference tracks no longer in the dataset. This prevents orphaned references after track cleanup.

Optional Dataset

The /negative_frames dataset is optional. Files without negative frames will not contain this dataset, and all frames will have is_negative=False.

Lazy Loading

For large SLP files with hundreds of thousands of frames, sleap-io provides a lazy loading mode that defers Labels object creation until needed.

Streaming from a URL

.slp/.pkg.slp files can be opened straight from http/https, cloud, or Google Drive URLs via load_slp with lazy range-based reads (embedded pkg.slp frames reopen the remote file on demand). See Loading from URLs.

Architecture

Labels (lazy mode)
├── LazyDataStore
│   ├── frames_data (numpy array)
│   ├── instances_data (numpy array)
│   ├── points_data (numpy array)
│   └── pred_points_data (numpy array)
└── LazyFrameList
    └── Materializes frames on-demand

Performance Benefits

Operation	Eager	Lazy	Speedup
Load file	~0.5s	~0.005s	~100x
Load + numpy()	~0.9s	~0.4s	~2x
Full iteration	~0.0002s	~0.4s	Eager faster

Benchmarks on 18,000 frames with ~40,000 instances.

When to Use Lazy Loading

Recommended for:

• Converting to NumPy arrays (labels.numpy())
• Saving to another file without modifications
• Accessing a small subset of frames
• Quick metadata inspection

Not recommended for:

• Iterating over all frames (eager is faster)
• Modifying data (must materialize first)
• Multiple passes over the data

Fast Paths

Lazy labels support optimized code paths:

1. numpy() conversion: Builds arrays directly from raw HDF5 data without creating Python objects
2. Saving: Copies raw arrays directly without materialization (when embed is None, False, or "source")
3. Metadata queries: Properties like n_user_instances, n_pred_instances use O(1) array operations

Usage

import sleap_io as sio

# Load lazily
labels = sio.load_slp("predictions.slp", lazy=True)

# Check lazy state
print(labels.is_lazy)  # True

# Fast numpy conversion
poses = labels.numpy()

# Materialize when modifications needed
labels = labels.materialize()
labels.append(new_frame)  # Now works

Bounding Boxes

BoundingBox annotations store axis-aligned or oriented bounding boxes for object detection and tracking workflows. Bounding box support was introduced in format 1.7. Format 2.0+ uses columnar storage under the /bboxes/ HDF5 group.

Columnar Datasets (Format 2.0+)

Bounding box data is stored as individual datasets within the /bboxes/ HDF5 group:

Dataset	Dtype	Description
x1	float64	Top-left x-coordinate in pixels
y1	float64	Top-left y-coordinate in pixels
x2	float64	Bottom-right x-coordinate in pixels
y2	float64	Bottom-right y-coordinate in pixels
angle	float64	Rotation angle in radians (0 = axis-aligned)
video	int32	Video index (-1 if none)
frame_idx	int64	Frame index (-1 if none)
track	int32	Track index (-1 if none)
instance	int32	Instance index (-1 if none)
is_predicted	uint8	0 = UserBoundingBox, 1 = PredictedBoundingBox
score	float32	Confidence score (NaN for user bboxes)
tracking_score	float32	Tracking link confidence (NaN if unset)
category	vlen str	Category label per bounding box
name	vlen str	Name label per bounding box
source	vlen str	Source label per bounding box

User vs Predicted

• is_predicted = 0: UserBoundingBox -- human-annotated
• is_predicted = 1: PredictedBoundingBox -- model-predicted, score contains the confidence value

String Metadata

String metadata (category, name, source) is stored as vlen string datasets within the /bboxes/ group, one entry per bounding box.

Optional Dataset

The /bboxes/ group is only written when the Labels object contains bounding boxes. On read, a missing group defaults to an empty list.

Legacy format (1.7--1.9)

Older SLP files store bounding boxes as a single structured array dataset (/bboxes) with x_center, y_center, width, height columns and JSON-encoded string attributes. The reader auto-detects whether /bboxes is a group (format 2.0+) or a dataset (legacy) and handles both transparently.

Migration from Format 1.5/1.6

When reading older files without a /bboxes dataset or group, any ROIs with axis-aligned rectangular geometry (is_bbox = True) are automatically migrated to UserBoundingBox objects in Labels.bboxes. The migrated ROIs are removed from Labels.rois.

Centroids

Centroid annotations store lightweight 2D or 3D point detections used for point tracking and TrackMate-style spot workflows. Centroid storage was added in v0.7.0 and uses a columnar HDF5 group, mirroring the bounding box layout.

Columnar Datasets

Centroid data is stored as individual datasets within the /centroids/ HDF5 group:

Dataset	Dtype	Description
x	float64	x-coordinate in pixels
y	float64	y-coordinate in pixels
z	float64	z-coordinate (NaN for 2D centroids)
video	int32	Video index (-1 if none)
frame_idx	int64	Frame index (-1 if none)
track	int32	Track index (-1 if none)
instance	int32	Instance index (-1 if none)
is_predicted	uint8	0 = UserCentroid, 1 = PredictedCentroid
score	float32	Confidence score (NaN for user centroids)
tracking_score	float32	Tracking link confidence (NaN if unset)
category	vlen str	Category label per centroid
name	vlen str	Name label per centroid
source	vlen str	Source label (e.g., "trackmate")

The z dataset is always written but stores NaN for 2D centroids. This makes the columnar layout uniform between 2D point tracking and 3D triangulated workflows without requiring a separate dtype.

User vs Predicted

• is_predicted = 0: UserCentroid -- human-annotated
• is_predicted = 1: PredictedCentroid -- model- or tracker-detected, score contains the detection confidence

String Metadata

Per-centroid category, name, and source strings are stored as vlen string datasets within the /centroids/ group. The source field is the canonical place for tracker provenance — for example, sio.load_trackmate(...) sets source="trackmate".

Optional Group

The /centroids/ group is only written when the Labels object contains centroids. On read, a missing group defaults to an empty list.

Format version

Centroid presence alone does not bump the SLP format version — the /centroids/ group rides alongside whatever other state drives the file's format_id (see Version History below). A file containing only centroids and pose instances may still be written at format 1.4.

Regions of Interest (ROIs)

ROIs store vector geometry annotations such as polygons and other shapes. ROI support was introduced in format 1.5.

ROI Datasets

ROI data is stored across two datasets:

• /rois: Structured array containing ROI metadata and byte offsets into the geometry data
• /roi_wkb: Packed uint8 array of WKB (Well-Known Binary) geometry bytes, gzip-compressed (compression_opts=1)

Each ROI's geometry is stored as a WKB blob in /roi_wkb, with /rois providing the byte range via wkb_start and wkb_end offsets.

/roi_wkb is stored with the HDF5 gzip filter (compression="gzip", compression_opts=1), a transparent on-disk storage detail: readers decompress it automatically, byte ranges are unchanged, and no format_id bump is required. This is lossless and typically shrinks /roi_wkb ~2x on polygon-heavy ROI sets. Browser/h5wasm readers must support the deflate filter, which is already required for the format's gzip-filtered /mask_rle and embedded video frames (and the v2.2 chunked /label_image_data).

ROI Dtype

roi_dtype = np.dtype([
    ("annotation_type", "u1"),  # Legacy field, always written as 0
    ("video", "i4"),            # Video index (-1 if none)
    ("frame_idx", "i8"),        # Frame index (-1 for static ROIs)
    ("track", "i4"),            # Track index (-1 if none)
    ("is_predicted", "u1"),     # 0 = UserROI, 1 = PredictedROI (Format 1.9+)
    ("score", "f4"),            # Confidence score (NaN for user ROIs) (Format 1.9+)
    ("tracking_score", "f4"),   # Tracking link confidence (NaN if unset)
    ("wkb_start", "u8"),       # Start byte offset into /roi_wkb
    ("wkb_end", "u8"),         # End byte offset into /roi_wkb
    ("instance", "i4"),        # Instance index (-1 if none) (Format 1.6+)
])

Legacy and predicted fields

The annotation_type column is retained in the on-disk dtype for backward compatibility with older readers but is no longer used. Writers always set annotation_type = 0. Use the category string metadata for semantic classification and BoundingBox for detection annotations.

The is_predicted field distinguishes UserROI (0) from PredictedROI (1). The score field stores the confidence value for predicted ROIs (NaN for user ROIs).

String Metadata

Format 1.9+ stores ROI string metadata as vlen HDF5 string datasets at the root level:

• /roi_categories: One category string per ROI
• /roi_names: One name string per ROI
• /roi_sources: One source string per ROI

The reader checks for these datasets first. For pre-1.9 files, it falls back to JSON-encoded HDF5 attributes on the /rois dataset (@categories, @names, @sources).

# Format 1.9+ (vlen string datasets)
f["/roi_categories"]  # ["arena", "nest"]
f["/roi_names"]       # ["arena_boundary", "nest_region"]
f["/roi_sources"]     # ["manual", "model_v2"]

# Pre-1.9 legacy (JSON attributes on /rois dataset)
rois_dataset.attrs["categories"]  # '["arena", "nest"]'
rois_dataset.attrs["names"]       # '["arena_boundary", "nest_region"]'
rois_dataset.attrs["sources"]     # '["manual", "model_v2"]'

Static vs Temporal ROIs

• Static ROIs: frame_idx = -1. Apply globally (e.g., arena boundaries).
• Temporal ROIs: frame_idx >= 0. Associated with a specific frame in a video.

Optional Dataset

The /rois and /roi_wkb datasets are only written when the Labels object contains ROIs. On read, missing datasets default to empty lists.

Segmentation Masks

SegmentationMasks store raster binary masks using run-length encoding (RLE). Mask support was introduced in format 1.5.

Mask Datasets

Mask data is stored across two datasets:

• /masks: Structured array containing mask metadata and byte offsets into the RLE data
• /mask_rle: Packed uint8 array of RLE-encoded mask bytes, gzip-compressed (compression_opts=1)

The RLE encoding stores uint32 run-length counts packed as little-endian uint8 bytes. Each mask's RLE data is located in /mask_rle at the byte range specified by rle_start and rle_end.

/mask_rle is stored with the HDF5 gzip filter (compression="gzip", compression_opts=1), a transparent on-disk storage detail: readers decompress it automatically, byte ranges are unchanged, and no format_id bump is required. RLE counts are run-lengths whose packed bytes are highly redundant, so gzip is lossless and typically shrinks /mask_rle ~8x (and the whole file ~5x on fragmented segmentation masks). Degenerate empty-RLE masks are written uncompressed (nothing to compress).

Mask Dtype

mask_dtype = np.dtype([
    ("height", "u4"),           # Mask height in pixels
    ("width", "u4"),            # Mask width in pixels
    ("annotation_type", "u1"),  # Legacy field, always written as 2
    ("video", "i4"),            # Video index (-1 if none)
    ("frame_idx", "i8"),        # Frame index (-1 for static masks)
    ("track", "i4"),            # Track index (-1 if none)
    ("instance", "i4"),         # Instance index (-1 if none) (Format 1.9+)
    ("from_predicted", "i4"),   # Source prediction index into this mask list, -1 if none (Format 2.4+)
    ("is_predicted", "u1"),     # 0 = UserSegmentationMask, 1 = Predicted (Format 1.9+)
    ("score", "f4"),            # Confidence score (NaN for user masks) (Format 1.9+)
    ("tracking_score", "f4"),   # Tracking link confidence (NaN if unset)
    ("rle_start", "u8"),       # Start byte offset into /mask_rle
    ("rle_end", "u8"),         # End byte offset into /mask_rle
    ("scale_x", "f4"),          # Spatial scale x (1.0 = native res) (Format 2.1+)
    ("scale_y", "f4"),          # Spatial scale y (1.0 = native res) (Format 2.1+)
    ("offset_x", "f4"),         # Spatial offset x in pixels (Format 2.1+)
    ("offset_y", "f4"),         # Spatial offset y in pixels (Format 2.1+)
])

Legacy and predicted fields

The annotation_type column is retained for backward compatibility but ignored on read. Writers always set annotation_type = 2 (SEGMENTATION).

The is_predicted field distinguishes UserSegmentationMask (0) from PredictedSegmentationMask (1). The score field stores the confidence value for predicted masks (NaN for user masks).

String Metadata

Format 1.9+ stores mask string metadata as vlen HDF5 string datasets at the root level:

• /mask_categories: One category string per mask
• /mask_names: One name string per mask
• /mask_sources: One source string per mask

The reader checks for these datasets first. For pre-1.9 files, it falls back to JSON-encoded HDF5 attributes on the /masks dataset (@categories, @names, @sources).

Optional Dataset

The /masks and /mask_rle datasets are only written when the Labels object contains masks. On read, missing datasets default to empty lists.

The UserSegmentationMask.from_predicted provenance link (set by PredictedSegmentationMask.to_user()) is persisted (Format 2.4+) in the from_predicted column as an index into the flat mask list, mirroring instance from_predicted (see the Instances → Instance Linking section above). On write, the linked PredictedSegmentationMask is resolved to its global mask index in a deferred pass; a None link or a source that is no longer in labels.masks is written as -1. On read, the index is resolved back to the mask object in a deferred pass after all masks are constructed; -1 and out-of-range indices load as None. The column is gated on presence at read time, so files written before Format 2.4 (which lack the column) load from_predicted as None.

Score Map Datasets

Score maps store per-pixel confidence values for predicted segmentation masks and label images. These datasets are only written when PredictedSegmentationMask or PredictedLabelImage objects have a score_map set.

Mask Score Maps

Mask score maps are stored across two datasets:

• /mask_score_map_index: Structured array indexing into the packed data
• /mask_score_maps: Packed uint8 array of zlib-compressed float32 score map data

Index Dtype

mask_score_map_index_dtype = np.dtype([
    ("mask_idx", "u4"),     # Index into /masks dataset
    ("data_start", "u8"),   # Start byte offset into /mask_score_maps
    ("data_end", "u8"),     # End byte offset into /mask_score_maps
    ("height", "u4"),       # Score map height in pixels
    ("width", "u4"),        # Score map width in pixels
    ("scale_x", "f4"),      # Score map spatial scale x (Format 2.1+)
    ("scale_y", "f4"),      # Score map spatial scale y (Format 2.1+)
    ("offset_x", "f4"),     # Score map spatial offset x in pixels (Format 2.1+)
    ("offset_y", "f4"),     # Score map spatial offset y in pixels (Format 2.1+)
])

Label Image Score Maps

Label image score maps follow the same structure:

• /label_image_score_map_index: Structured array indexing into the packed data
• /label_image_score_maps: Packed uint8 array of zlib-compressed float32 score map data

Index Dtype

label_image_score_map_index_dtype = np.dtype([
    ("li_idx", "u4"),       # Index into /label_images dataset
    ("data_start", "u8"),   # Start byte offset into /label_image_score_maps
    ("data_end", "u8"),     # End byte offset into /label_image_score_maps
    ("height", "u4"),       # Score map height in pixels
    ("width", "u4"),        # Score map width in pixels
    ("scale_x", "f4"),      # Score map spatial scale x (Format 2.1+)
    ("scale_y", "f4"),      # Score map spatial scale y (Format 2.1+)
    ("offset_x", "f4"),     # Score map spatial offset x in pixels (Format 2.1+)
    ("offset_y", "f4"),     # Score map spatial offset y in pixels (Format 2.1+)
])

Data Format

Score map pixel data is stored as float32 arrays, compressed with zlib, and packed into a single uint8 byte array. Each score map's compressed bytes are located at the byte range [data_start, data_end) in the corresponding packed dataset.

Optional Datasets

The score map datasets are only written when at least one predicted mask or label image has a score_map set. On read, missing datasets are silently skipped.

Label Images

LabelImages store dense per-pixel instance segmentation data, where each pixel is assigned an integer label corresponding to an object. Label image support was introduced in format 1.8.

Label Image Datasets

Label image data is stored across three datasets:

• /label_images: Structured array containing label image metadata
• /label_image_objects: Structured array containing per-object metadata
• /label_image_data: Pixel data in one of two formats:
  • Blob format (v1.8-v2.1): Flat uint8 array of zlib-compressed bytes, indexed by (data_start, data_end) byte offsets
  • Chunked format (v2.2+): (T, H, W) int32 dataset with per-frame gzip chunks, written via write_direct_chunk for maximum throughput

Label Image Dtype

label_image_dtype = np.dtype([
    ("video", "i4"),          # Video index (-1 if none)
    ("frame_idx", "i8"),      # Frame index
    ("height", "u4"),         # Image height in pixels
    ("width", "u4"),          # Image width in pixels
    ("n_objects", "u4"),      # Number of objects in this label image
    ("objects_start", "u4"),  # Start index into /label_image_objects
    ("data_start", "u8"),     # Start byte offset into /label_image_data
    ("data_end", "u8"),       # End byte offset into /label_image_data
    ("is_predicted", "u1"),   # 0 = UserLabelImage, 1 = Predicted (Format 1.9+)
    ("score", "f4"),          # Confidence score (NaN for user) (Format 1.9+)
    ("scale_x", "f4"),        # Spatial scale x (1.0 = native res) (Format 2.1+)
    ("scale_y", "f4"),        # Spatial scale y (1.0 = native res) (Format 2.1+)
    ("offset_x", "f4"),       # Spatial offset x in pixels (Format 2.1+)
    ("offset_y", "f4"),       # Spatial offset y in pixels (Format 2.1+)
])

Objects Dtype

Each object within a label image is described by a row in /label_image_objects:

label_image_object_dtype = np.dtype([
    ("label_id", "i4"),       # Pixel label value in the image data
    ("track", "i4"),          # Track index (-1 if none)
    ("instance", "i4"),       # Instance index (-1 if none)
    ("score", "f4"),          # Per-object confidence score (Format 1.9+)
    ("tracking_score", "f4"), # Tracking link confidence (NaN if unset)
])

Pixel Data

Label image pixel data is stored as int32 arrays (0 = background, positive values = object IDs) in one of two formats:

Blob format (v1.8-v2.1): Each frame is individually zlib-compressed and packed into /label_image_data as a flat uint8 byte array. The data_start and data_end fields in the index table give the byte range [data_start, data_end) for each frame.

Chunked format (v2.2+): When all frames share the same (H, W) dimensions, pixel data is stored as a 3D (T, H, W) int32 dataset with chunk shape (1, H, W) and gzip-1 compression. Data is written via write_direct_chunk (pre-compressed with zlib.compress(level=1)) for ~43x throughput improvement over standard h5py writes. The data_start and data_end fields are unused (set to 0). The reader auto-detects the format by checking label_image_data.ndim (3 = chunked, 1 = blob).

When frame sizes are mixed, the writer falls back to blob format automatically.

String Metadata

Format 1.9+ stores label image string metadata as vlen HDF5 string datasets:

• /label_image_sources: One source string per label image
• /label_image_obj_categories: One category string per object
• /label_image_obj_names: One name string per object

Optional Datasets

The label image datasets are only written when the Labels object contains label images. On read, missing datasets default to empty lists.

Storage Representation Matrix

Different data structures use different on-disk representations, chosen per subsystem (fixed-width structural rows, bulk numeric data that must stream, and open-ended metadata each get a different form). This section consolidates, for every data structure at the current format (2.8), its representation, the format version that introduced it, and its read/write support.

Representation kinds

Kind	Meaning
compound	A single HDF5 dataset with a compound (structured) dtype — multiple named fields per row, usually row-range–sliced by an index table.
columnar group	An HDF5 group with one plain 1-D dataset per field (struct-of-arrays), all parallel-indexed.
EAV	A columnar entity-attribute-value triple (meta_owner / meta_key / meta_val) encoding arbitrary per-entity metadata dicts.
plain matrix	A single numeric dataset (2-D or 3-D), no named fields.
ragged CSR	A flat concatenated values dataset sliced by an offset/index array.
string	A variable-length UTF-8 string dataset, one string per row.
JSON	A variable-length UTF-8 string dataset whose values are JSON documents.
attribute	Stored as an HDF5 attribute, not a dataset.

Write is single-method; read is broader

• Write (the default, and only, method): the Python library writes exactly one native representation per data structure — there is no toggle. In particular it always writes genuine HDF5 compound datasets for the compound rows; it never writes the h5wasm flat-2D substitute.
• Read: the compound readers also accept the flat-2D + field_names encoding that sleap-io.js writes (h5wasm cannot create compound datasets), routed through utils._read_dataset_from_open_file. Every other kind (columnar / matrix / string / JSON) is read directly and is already h5wasm-native, so no conversion is needed. compound is therefore the only kind with a browser-interop dependency (see Browser-side compatibility).

Matrix

Reads JS flat-2D = the read path accepts the sleap-io.js flat-2D + field_names encoding (only meaningful, and only needed, for compound datasets).

Data structure	HDF5 path(s)	Representation	Ver	Reads JS flat-2D
Labeled frames / instances / 2D points	/frames, /instances, /points, /pred_points	compound	1.0	✅
Negative frames	/negative_frames	compound	1.x	✅
Segmentation masks (table)	/masks	compound	1.5	✅
Mask RLE geometry	/mask_rle	ragged CSR	1.5	✅
ROIs (table)	/rois	compound	1.5	✅
ROI geometry (WKB)	/roi_wkb	ragged CSR	1.5	✅
Bounding boxes / centroids	/bboxes, /centroids	columnar group	2.0 †	n/a
Label images (table)	/label_images, /label_image_objects	compound	1.8	✅
Label-image pixel maps	/label_image_data (T,H,W)	plain matrix	2.2	n/a
Dense score-map index tables	/mask_score_map_index, /label_image_score_map_index	compound	1.9 / 2.1	⚠️ no
Dense score-map blobs	/mask_score_maps, /label_image_score_maps	ragged CSR	1.9	n/a
Identity / category catalog names	/identity/name, /categories/name	string	2.5 / 2.7	n/a
Identity / category / event metadata	…/meta_owner, …/meta_key, …/meta_val	EAV	2.5 / 2.7 / 2.6	n/a
Per-detection identity / category links	/identity/links, /categories/links	compound	2.5 / 2.7	✅
Re-ID / category embeddings + join cols	/embeddings/vectors, owner_type, owner_id, category_*	plain matrix	2.5 / 2.7	n/a
Event types catalog	/event_types/name, /description	string	2.6	n/a
Frame-spanning events	/events/* (video, start_frame, …)	columnar group	2.6	n/a
Event framewise score traces	/events/scores + /events/score_offsets	ragged CSR	2.6	n/a
Sessions — calibration + video map + frame-group range	sessions_json	JSON	2.8	n/a
Session frame groups / instance groups / members	/session_data/frame_groups, /instance_groups, /instance_group_members	compound	2.8	✅
3D points (row-range sliced)	/session_data/points_3d (N,3), /pred_points_3d (N,4)	plain matrix	2.8	n/a
Per-group session metadata	/session_data/frame_group_meta, /instance_group_meta	JSON (per row)	2.8	n/a
Videos / tracks / suggestions / provenance	videos_json, tracks_json, suggestions_json, provenance_json	JSON	1.0 / 2.x	n/a
Virtual video crops	/video_crops	JSON	2.3	n/a
Per-detection string metadata	/mask_*, /roi_*, /label_image_* name/category/source	string	1.9	n/a
Skeletons + provenance + format_id	/metadata attrs (json, format_id)	attribute	1.0	n/a

† Centroid presence alone does not bump format_id (see Centroids); both /bboxes and /centroids use the same columnar-group mechanism introduced at 2.0.

Notes

• The 2.8 sessions subsystem deliberately mixes three representations — compound (the small structural index tables), plain matrix (the bulk 3D coordinates, chunked + gzip so they stream), and per-row JSON (arbitrary group metadata) — matching each payload to the right storage (see Sessions).
• Only compound has a browser-interop dependency. Two compound tables — the dense score-map index tables (⚠️) — are read without the flat-2D fallback, so a sleap-io.js-written score-map file would not load through them; score maps are a rare prediction-only feature outside the currently-in-flight port.
• Lazy loads copy the entire /session_data group + sessions_json verbatim on save (no re-encoding), preserving the same on-disk representation without materializing frames.

Version History

The SLP format has evolved through several versions, tracked by the format_id attribute in /metadata.

Format 1.0

Initial release format.

Format 1.1

Coordinate system change: Changed from top-left pixel corner at (0, 0) to pixel center at (0, 0).

• Reading: Applies -0.5 offset to coordinates from older files
• Writing: Always uses new coordinate system

Format 1.2

Added tracking_score field to instance dtype.

• Instance dtype expanded from 9 to 10 fields
• tracking_score stores tracking confidence for multi-animal workflows
• Reading: Defaults to 0.0 for older files

Format 1.3

Minor handling improvements for tracking_score (no schema change from 1.2).

Format 1.4

Added channel_order attribute to embedded video datasets.

• Tracks RGB vs BGR channel ordering for embedded images
• Ensures correct color reproduction across different encoding backends
• Reading: Defaults to RGB if attribute missing

Format 1.5

Added ROI and segmentation mask support.

• New datasets: /rois, /roi_wkb for vector geometry (WKB-encoded)
• New datasets: /masks, /mask_rle for binary masks (RLE-encoded)
• String metadata stored as JSON HDF5 attributes (categories, names, sources)
• Backward compatible: datasets only written when non-empty, missing datasets default to empty lists on read
• Requires shapely>=2.0 for geometry operations

Format 1.6

Added ROI-instance association.

• Added instance field (i4) to the /rois dtype for linking ROIs to specific instances
• ROI instance associations are persisted via instance index

Format 1.7

Added bounding box support.

• New dataset: /bboxes for first-class bounding box annotations
• Supports axis-aligned and oriented (rotated) bounding boxes
• User/predicted distinction via is_predicted flag and score field
• Migration on read: rectangular ROIs from older files are automatically converted to BoundingBox objects
• annotation_type and score fields on /rois and /masks are now legacy (always written as constants)

Format 1.8

Added label image support.

• New datasets: /label_images, /label_image_objects, /label_image_data for per-pixel segmentation annotations
• First-class LabelImage type for instance segmentation workflows

Format 1.9

Added Instance3D and predicted variant support.

• Extended InstanceGroup serialization with instance_3d_score and instance_3d_point_scores fields
• Instance3D and PredictedInstance3D provide structured 3D keypoint storage
• Added is_predicted (u1) and updated score fields to ROI, mask, and label image dtypes for predicted variant support (PredictedROI, PredictedSegmentationMask, PredictedLabelImage)
• Added instance (i4) field to mask dtype for mask-instance association
• Migrated ROI and mask string metadata from JSON attributes to vlen HDF5 string datasets (/roi_categories, /roi_names, /roi_sources, /mask_categories, /mask_names, /mask_sources)
• Added label image string metadata datasets (/label_image_sources, /label_image_obj_categories, /label_image_obj_names)
• Added score map datasets (/mask_score_map_index, /mask_score_maps, /label_image_score_map_index, /label_image_score_maps)
• Backward compatible: new fields are optional, old readers skip unknown keys via metadata pass-through

Format 2.0

Columnar bounding box storage.

• /bboxes changed from a structured array dataset to an HDF5 group with columnar datasets
• Bounding box coordinates use x1/y1/x2/y2 (top-left/bottom-right) representation instead of x_center/y_center/width/height
• String metadata (category, name, source) stored as vlen string datasets within the group
• The reader auto-detects whether /bboxes is a group (format 2.0+) or a dataset (legacy 1.7--1.9) and handles both transparently

Format 2.1

Multi-resolution spatial metadata for dense annotations.

• Added scale_x, scale_y, offset_x, offset_y (f4) fields to mask_dtype, label_image_dtype, mask_score_map_index_dtype, and label_image_score_map_index_dtype
• Lets a SegmentationMask or LabelImage describe pixel data at half resolution, quarter resolution, or any spatial offset/scale relative to the source video
• A (scale, offset) of ((1.0, 1.0), (0.0, 0.0)) means native resolution; e.g., stride=2 constructors set scale=(0.5, 0.5) for half-resolution masks
• Triggered automatically when any mask or label image has non-trivial spatial transform; otherwise the format remains 1.x or 2.0
• Backward compatible: older readers ignore the new fields; writers always emit them but set defaults for unset cases

Format 2.2

Chunked label image storage and lazy loading.

• /label_image_data can now be a 3D (T, H, W) int32 dataset with per-frame gzip chunks instead of the legacy flat byte blob
• Written via write_direct_chunk with zlib.compress(level=1) for ~43x faster writes than standard h5py
• Pixel data is loaded lazily: the HDF5 file stays open and each frame is decompressed on first .data access
• Format auto-detected on read by label_image_data.ndim (3 = chunked, 1 = blob)
• Falls back to blob format when frame sizes are not uniform
• New LabelImageWriter enables streaming writes with constant memory
• New merge_label_images() copies raw compressed chunks between files (zero decompression for chunked sources)
• Backward compatible: old files (v1.8-v2.1) remain fully readable; data_start/data_end fields are unused (set to 0) in chunked format

Format 2.3

Virtual on-read video crops.

• Optional /video_crops dataset stores per-video crop rectangles applied virtually on read (see Virtual Crops above)
• Triggered only when a video carries a crop; uncropped files stay byte-identical and at format_id <= 2.2
• Purely additive: does not cross the only legacy threshold (< 1.4)

Format 2.4

Persisted mask from_predicted provenance.

• Added a from_predicted (i4) column to mask_dtype storing the index of the source PredictedSegmentationMask in the flat mask list (-1 if none), mirroring instance from_predicted
• Lets UserSegmentationMask.from_predicted (set by PredictedSegmentationMask.to_user()) survive a save/load round-trip when the source prediction is also saved
• Triggered automatically when any mask records a from_predicted link; otherwise the format stays at whatever other state drives format_id
• Backward compatible: the column is always written but reads are gated on its presence, so files written before 2.4 (which lack the column) load from_predicted as None

Format 2.5

Re-ID identity subsystem: identity catalog, per-detection links, and appearance embeddings.

• New optional /identity group (see Identities):
  • name — one variable-length UTF-8 string per Identity, in catalog order
  • meta_owner / meta_key / meta_val — an entity-attribute-value table of per-identity metadata (one row per (identity, key, value) triple), all three omitted when no identity carries metadata
  • links — a structured dataset (owner_type, owner_id, identity_idx, identity_score) with one row per detection that carries an identity, joining it to a catalog entry. The owner_type column uses the shared OWNER_* codes (0=instance, 2=centroid, 3=mask, 4=bbox, 5=roi), the same scheme as the /embeddings join
• New optional /embeddings group holding per-detection appearance / re-ID vectors as a single columnar struct-of-arrays: vectors (N, D) float (chunked so whole rows stay within a chunk, gzip-compressed; all vectors share one D) plus the owner_type/owner_id join columns (see Embeddings). Persists the single identity_embedding slot on each instance, SegmentationMask, Centroid, BoundingBox, or ROI; large float vectors live in gzipped numeric datasets, never in JSON
• Triggered automatically when any detection carries an Identity or an embedding; identity- and embedding-free files stay at format_id <= 2.4. Pass save_slp(..., save_embedding_vectors=False) to write the identity links but skip the (large) /embeddings group
• Backward compatible: reads are gated on group presence, so older readers ignore both groups

Format 2.6

Frame-spanning events: event catalog and interval annotations.

• New optional /event_types group (a name string dataset + optional description + meta_* EAV metadata) and /events group (a columnar struct-of-arrays plus a ragged CSR scores/score_offsets pair for optional framewise PredictedEvent.scores) — see Events
• An Event has a temporal extent (an inclusive [start_frame, end_frame] interval) and lives on Labels.events rather than a LabeledFrame; participants are stored as (kind, idx) pairs referencing tracks or identities
• Triggered automatically when the file has any events / event types; event-free files stay at format_id <= 2.5
• Backward compatible: reads are gated on group presence, so older readers ignore both groups

Format 2.7

Class/category subsystem: category catalog, per-detection links, and category appearance embeddings.

• New optional /categories group (see Categories), a fully self-contained mirror of /identity:
  • name — one variable-length UTF-8 string per Category, in catalog order
  • meta_owner / meta_key / meta_val — an entity-attribute-value table of per-category metadata, all three omitted when no category carries metadata
  • links — a structured dataset (owner_type, owner_id, category_idx, category_score) with one row per detection that carries a category, using the same shared OWNER_* join codes as /identity/links
• Category appearance vectors (the category_embedding slot on each instance, SegmentationMask, Centroid, BoundingBox, or ROI) live in the same /embeddings group as parallel, independent datasets — category_vectors (M, D') plus category_owner_type / category_owner_id — kept separate from the identity vectors datasets so the two embedding kinds need not share dimensionality (see Embeddings)
• Triggered automatically when any detection carries a Category or a category embedding; category-free files stay at format_id <= 2.6. Pass save_slp(..., save_embedding_vectors=False) to write the category links but skip the (large) category vectors
• Backward compatible and byte-identical for unused features: the /categories group and the category_* embedding datasets are only written when present, so identity-only files stay unchanged

Format 2.8 (Current)

Columnar RecordingSession storage: frame groups and 3D points moved out of the sessions_json string.

• Previously an entire RecordingSession — every frame group with its inline 3D points — was serialized as one JSON string per session in /sessions_json. For real multi-view projects this string grew to hundreds of MB (dominated by 3D point text) and became unreadable in JS/WASM consumers past the ~0.45 GB variable-length-string limit
• /sessions_json now holds only the small, bounded per-session data — calibration, camcorder_to_video_idx_map, session-level metadata, and an fg_start/fg_end range into session_data/frame_groups
• New optional /session_data group (see Sessions) stores the unbounded per-frame payload columnar: frame_groups / instance_groups / instance_group_members struct tables (the last columnarizing the old camcorder_to_lf_and_inst_idx_map), points_3d (N, 3) and pred_points_3d (N, 4 = xyz+score) chunked+gzip float matrices sliced by row range, plus presence-guarded per-row frame_group_meta / instance_group_meta JSON blobs. NaN denotes missing keypoints and round-trips natively
• Triggered automatically only when a session has frame groups; session-free and single-view files stay at format_id <= 2.7 and are byte-identical
• Backward compatible: the reader dispatches on /session_data presence and still parses the legacy inline frame_group_dicts (with inline points) of ≤ 2.7 files. Lazy loads copy the raw sessions_json + /session_data verbatim on save, so a lazy re-save preserves frame groups and 3D losslessly (previously the lazy path dropped them)
• Coordinated cross-port change: the sleap-io.js browser port adopts the same /session_data layout in a companion release. Because h5wasm cannot create HDF5 compound datasets, it writes the frame_groups / instance_groups / instance_group_members struct tables as flat 2D arrays with a field_names attribute (exactly as it already does for points / instances / frames); the Python reader converts them on read. The points_3d / pred_points_3d float matrices are plain 2D arrays and need no conversion in either direction

Browser-side compatibility (h5wasm / sleap-io.js)

sleap-io.js is the browser port of this library and writes SLP files via h5wasm. h5wasm cannot create HDF5 compound (structured) datasets, so it stores compound datasets — points, pred_points, instances, frames, and (Format 2.8+) the /session_data struct tables (frame_groups, instance_groups, instance_group_members) — as flat 2D arrays with the same per-row field layout as the structured dtype, tagged with a field_names attribute. The Python reader detects this representation (via shape and the field_names attribute) and auto-converts on the fly, so files written by sleap-io.js round-trip cleanly through the Python library without a manual conversion step. Multiple HDF5 string encodings are also tolerated (PR #378).

The /mask_rle dataset is stored with the HDF5 gzip (deflate) filter. Readers must support deflate to read masks — this is already required for the SLP format's embedded video frames (and the v2.2 chunked /label_image_data), and h5wasm bundles zlib, so no additional reader capability is introduced.

API

High-Level Functions

sleap_io.io.main.load_slp(filename, open_videos=True, lazy=False, *, headers=None, stream_mode='auto', cache_storage=None, cache_expiry=None, block_size=1048576, max_blocks=32, retries=3, _file_like=None)

Load a SLEAP dataset from a local path or HTTP/cloud URL.

For local paths, all URL-specific keyword arguments are ignored.

Parameters:

Name	Type	Description	Default
filename	str | PathLike	Path to a SLEAP labels file (.slp), or a URL. Supported URL schemes: http, https, s3, gs, gcs, az, abfs. Cloud schemes require pip install 'sleap-io[cloud]'. Google Drive share links (https://drive.google.com/file/d/<ID>/view) are also supported and resolved to a direct download automatically (the file is fully downloaded into memory; folder links are not supported).	required
open_videos	bool	If True (the default), attempt to open the video backend for I/O. If False, the backend will not be opened (useful for reading metadata when the video files are not available).	True
lazy	bool	If True, defer instance materialization for faster loading. Lazy-loaded Labels support read operations and fast numpy/save. To modify, call labels.materialize() first. Default is False.	False
headers	dict[str, str] | None	HTTP headers (e.g. {"Authorization": "Bearer ..."}) forwarded to fsspec for URL loads. Stripped on cross-origin redirect. Ignored for local paths.	None
stream_mode	str	Remote streaming strategy (ignored for local paths). One of: "auto" (default; uses fsspec blockcache for lazy range reads), "blockcache", "cache" (full download via simplecache), "filecache" (download with ETag revalidation), or "download" (ephemeral full download into memory).	'auto'
cache_storage	str | PathLike | None	Override fsspec's cache directory for cache/filecache modes. Ignored for local paths.	None
cache_expiry	float | None	TTL (seconds) for filecache revalidation. Defaults to 3600 (1h) when not given. Ignored for other modes and local paths.	None
block_size	int	Range block size in bytes for blockcache mode. Default: 1 MiB. Ignored for local paths.	1048576
max_blocks	int	Max blocks kept in the in-memory LRU per open file. Default: 32 (32 MiB cap per open file). Ignored for local paths.	32
retries	int	Retry count for transient HTTP errors. Default: 3. Ignored for local paths.	3

Returns:

Type	Description
Labels	The dataset as a Labels object.

Raises:

Type	Description
RemoteIOError	For HTTP errors against URLs (404, 416, 5xx after retries, connection failures).
ImportError	For cloud schemes when the corresponding extra is not installed.
ValueError	For an unrecognized stream_mode.

See Also

Labels.is_lazy: Check if Labels is lazy-loaded. Labels.materialize: Convert lazy Labels to eager.

Source code in sleap_io/io/main.py

def load_slp(
    filename: str | os.PathLike,
    open_videos: bool = True,
    lazy: bool = False,
    *,
    headers: dict[str, str] | None = None,
    stream_mode: str = "auto",
    cache_storage: str | os.PathLike | None = None,
    cache_expiry: float | None = None,
    block_size: int = 1 << 20,
    max_blocks: int = 32,
    retries: int = 3,
    _file_like: Any | None = None,
) -> Labels:
    """Load a SLEAP dataset from a local path or HTTP/cloud URL.

    For local paths, all URL-specific keyword arguments are ignored.

    Args:
        filename: Path to a SLEAP labels file (`.slp`), or a URL. Supported URL
            schemes: `http`, `https`, `s3`, `gs`, `gcs`, `az`, `abfs`. Cloud
            schemes require `pip install 'sleap-io[cloud]'`. Google Drive share
            links (`https://drive.google.com/file/d/<ID>/view`) are also
            supported and resolved to a direct download automatically (the file
            is fully downloaded into memory; folder links are not supported).
        open_videos: If `True` (the default), attempt to open the video backend for
            I/O. If `False`, the backend will not be opened (useful for reading metadata
            when the video files are not available).
        lazy: If `True`, defer instance materialization for faster loading.
            Lazy-loaded Labels support read operations and fast numpy/save.
            To modify, call `labels.materialize()` first. Default is `False`.
        headers: HTTP headers (e.g. `{"Authorization": "Bearer ..."}`) forwarded
            to fsspec for URL loads. Stripped on cross-origin redirect. Ignored
            for local paths.
        stream_mode: Remote streaming strategy (ignored for local paths). One of:
            `"auto"` (default; uses fsspec `blockcache` for lazy range reads),
            `"blockcache"`, `"cache"` (full download via `simplecache`),
            `"filecache"` (download with ETag revalidation), or `"download"`
            (ephemeral full download into memory).
        cache_storage: Override fsspec's cache directory for `cache`/`filecache`
            modes. Ignored for local paths.
        cache_expiry: TTL (seconds) for `filecache` revalidation. Defaults to
            3600 (1h) when not given. Ignored for other modes and local paths.
        block_size: Range block size in bytes for `blockcache` mode. Default:
            1 MiB. Ignored for local paths.
        max_blocks: Max blocks kept in the in-memory LRU per open file. Default:
            32 (32 MiB cap per open file). Ignored for local paths.
        retries: Retry count for transient HTTP errors. Default: 3. Ignored for
            local paths.

    Returns:
        The dataset as a `Labels` object.

    Raises:
        RemoteIOError: For HTTP errors against URLs (404, 416, 5xx after
            retries, connection failures).
        ImportError: For cloud schemes when the corresponding extra is not
            installed.
        ValueError: For an unrecognized `stream_mode`.

    See Also:
        Labels.is_lazy: Check if Labels is lazy-loaded.
        Labels.materialize: Convert lazy Labels to eager.
    """
    import h5py

    from sleap_io.io import _remote, slp

    if _remote._is_url(filename):
        url = os.fspath(filename) if isinstance(filename, os.PathLike) else filename
        # ``_file_like`` lets a caller hand in an already-resolved file-like
        # (private; used by the Google Drive auto-detect path to reuse the bytes
        # it had to download to sniff the format, rather than re-resolving the
        # link a second time against Drive's per-file download quota). When
        # provided, the caller owns closing it.
        owns_file_like = _file_like is None
        file_like = (
            _remote.open_url(
                url,
                headers=headers,
                stream_mode=stream_mode,
                cache_storage=cache_storage,
                cache_expiry=cache_expiry,
                block_size=block_size,
                max_blocks=max_blocks,
                retries=retries,
            )
            if owns_file_like
            else _file_like
        )
        resolved_mode = "blockcache" if stream_mode == "auto" else stream_mode

        # Google Drive resolves to a full in-memory BytesIO (no range support).
        # Capture its bytes once so the long-lived label-image reopen reuses them
        # instead of re-resolving (and re-downloading) the Drive link.
        from sleap_io.io._gdrive import _is_gdrive_url

        url_bytes = None
        if _is_gdrive_url(url) and hasattr(file_like, "getvalue"):
            url_bytes = file_like.getvalue()

        try:
            with h5py.File(file_like, "r") as f:
                reader = (
                    slp._read_labels_lazy_from_open_file
                    if lazy
                    else slp._read_labels_from_open_file
                )
                labels = reader(
                    url,
                    f,
                    open_videos=open_videos,
                    _url_headers=headers,
                    _url_stream_mode=resolved_mode,
                    _url_bytes=url_bytes,
                )
        finally:
            if owns_file_like:
                file_like.close()

        # The URL auth context (headers/resolved_mode) is threaded into each
        # video backend at construction time and persisted on the Video by
        # `make_video` (via `_read_labels_*_from_open_file` -> `read_videos`), so
        # the embedded HDF5Video probe is authenticated and later frame reads /
        # existence probes / reopens stay authenticated. No post-hoc backfill.
        return labels

    # Local path - UNCHANGED behaviour; URL-specific kwargs are no-ops.
    if lazy:
        return slp._read_labels_lazy(filename, open_videos=open_videos)
    return slp.read_labels(filename, open_videos=open_videos)

sleap_io.io.main.save_slp(labels, filename, embed=False, restore_original_videos=True, embed_inplace=False, verbose=True, plugin=None, progress_callback=None, prefer_metadata=True, preserve_unknown=False, save_embedding_vectors=False)

Save a SLEAP dataset to a .slp file.

Parameters:

Name	Type	Description	Default
labels	Labels	A SLEAP Labels object (see load_slp).	required
filename	str	Path to save labels to ending with .slp.	required
embed	bool | str | list[tuple[Video, int]] | None	Frames to embed in the saved labels file. One of None, True, "all", "user", "suggestions", "user+suggestions", "source" or list of tuples of (video, frame_idx). If False is specified (the default), the source video will be restored if available, otherwise the embedded frames will be re-saved. If True or "all", all labeled frames and suggested frames will be embedded. If "source" is specified, no images will be embedded and the source video will be restored if available. This argument is only valid for the SLP backend.	False
restore_original_videos	bool	If True (default) and embed=False, use original video files. If False and embed=False, keep references to source .pkg.slp files. Only applies when embed=False.	True
embed_inplace	bool	If False (default), a copy of the labels is made before embedding to avoid modifying the in-memory labels. If True, the labels will be modified in-place to point to the embedded videos, which is faster but mutates the input. Only applies when embedding.	False
verbose	bool	If True (the default), display a progress bar when embedding frames.	True
plugin	str | None	Image plugin to use for encoding embedded frames. One of "opencv" or "imageio". If None, uses the global default from get_default_image_plugin(). If no global default is set, auto-detects based on available packages (opencv preferred, then imageio).	None
progress_callback	Callable[[int, int, str], bool] | None	Optional callback function called during embedding with (current, total, phase) arguments, where phase is "embed" or "write". If it returns False, the operation is cancelled and ExportCancelled is raised. When provided, tqdm progress bars are disabled in favor of the callback. The phase argument is a breaking change from the previous (current, total) signature.	None
prefer_metadata	bool	If True (the default), serialize each uncropped video's shape/grayscale/fps from its backend_metadata when recorded there instead of querying the live backend. For an open MediaVideo this avoids decoding a frame (and leaving a resident decoder) just to recompute already-known metadata. Set to False to always read shape/grayscale/fps through the live backend.	True
preserve_unknown	bool	If True, top-level HDF5 datasets/groups in the source file that sleap-io does not recognize are carried over into the saved file. This preserves additions from a newer sleap-io version across a load/save cycle. Default False. Best-effort (requires the source file to still exist and be readable HDF5). See write_labels.	False
save_embedding_vectors	bool	If False (the default), skip the /embeddings group entirely -- appearance vectors are large on disk, so only the identity links are persisted by default (the vectors stay in memory, e.g. to build identity prototypes). This mirrors embed, which is also off by default for video frames. Set True to also write the /embeddings group. Identity links (/identity/links) are written regardless.	False

Source code in sleap_io/io/main.py

def save_slp(
    labels: Labels,
    filename: str,
    embed: bool | str | list[tuple[Video, int]] | None = False,
    restore_original_videos: bool = True,
    embed_inplace: bool = False,
    verbose: bool = True,
    plugin: str | None = None,
    progress_callback: Callable[[int, int, str], bool] | None = None,
    prefer_metadata: bool = True,
    preserve_unknown: bool = False,
    save_embedding_vectors: bool = False,
):
    """Save a SLEAP dataset to a `.slp` file.

    Args:
        labels: A SLEAP `Labels` object (see `load_slp`).
        filename: Path to save labels to ending with `.slp`.
        embed: Frames to embed in the saved labels file. One of `None`, `True`,
            `"all"`, `"user"`, `"suggestions"`, `"user+suggestions"`, `"source"` or list
            of tuples of `(video, frame_idx)`.

            If `False` is specified (the default), the source video will be restored
            if available, otherwise the embedded frames will be re-saved.

            If `True` or `"all"`, all labeled frames and suggested frames will be
            embedded.

            If `"source"` is specified, no images will be embedded and the source video
            will be restored if available.

            This argument is only valid for the SLP backend.
        restore_original_videos: If `True` (default) and `embed=False`, use original
            video files. If `False` and `embed=False`, keep references to source
            `.pkg.slp` files. Only applies when `embed=False`.
        embed_inplace: If `False` (default), a copy of the labels is made before
            embedding to avoid modifying the in-memory labels. If `True`, the
            labels will be modified in-place to point to the embedded videos,
            which is faster but mutates the input. Only applies when embedding.
        verbose: If `True` (the default), display a progress bar when embedding frames.
        plugin: Image plugin to use for encoding embedded frames. One of "opencv"
            or "imageio". If None, uses the global default from
            `get_default_image_plugin()`. If no global default is set, auto-detects
            based on available packages (opencv preferred, then imageio).
        progress_callback: Optional callback function called during embedding with
            `(current, total, phase)` arguments, where ``phase`` is ``"embed"`` or
            ``"write"``. If it returns `False`, the operation is cancelled and
            `ExportCancelled` is raised. When provided, tqdm progress bars are
            disabled in favor of the callback. The ``phase`` argument is a breaking
            change from the previous ``(current, total)`` signature.
        prefer_metadata: If `True` (the default), serialize each uncropped video's
            shape/grayscale/fps from its `backend_metadata` when recorded there
            instead of querying the live backend. For an open `MediaVideo` this
            avoids decoding a frame (and leaving a resident decoder) just to recompute
            already-known metadata. Set to `False` to always read shape/grayscale/fps
            through the live backend.
        preserve_unknown: If `True`, top-level HDF5 datasets/groups in the source
            file that sleap-io does not recognize are carried over into the saved
            file. This preserves additions from a newer sleap-io version across a
            load/save cycle. Default `False`. Best-effort (requires the source file
            to still exist and be readable HDF5). See `write_labels`.
        save_embedding_vectors: If `False` (the default), skip the `/embeddings`
            group entirely -- appearance vectors are large on disk, so only the
            identity *links* are persisted by default (the vectors stay in memory,
            e.g. to build identity prototypes). This mirrors `embed`, which is also
            off by default for video frames. Set `True` to also write the
            `/embeddings` group. Identity links (`/identity/links`) are written
            regardless.
    """
    from sleap_io.io import slp

    return slp.write_labels(
        filename,
        labels,
        embed=embed,
        restore_original_videos=restore_original_videos,
        embed_inplace=embed_inplace,
        verbose=verbose,
        plugin=plugin,
        progress_callback=progress_callback,
        prefer_metadata=prefer_metadata,
        preserve_unknown=preserve_unknown,
        save_embedding_vectors=save_embedding_vectors,
    )

Core Module

sleap_io.io.slp.read_labels(labels_path, open_videos=True)

Read a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file.	required
open_videos	bool	If True (the default), attempt to open the video backend for I/O. If False, the backend will not be opened (useful for reading metadata when the video files are not available).	True

Returns:

Type	Description
Labels	The processed Labels object.

Source code in sleap_io/io/slp.py

def read_labels(labels_path: str, open_videos: bool = True) -> Labels:
    """Read a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file.
        open_videos: If `True` (the default), attempt to open the video backend for
            I/O. If `False`, the backend will not be opened (useful for reading metadata
            when the video files are not available).

    Returns:
        The processed `Labels` object.
    """
    with h5py.File(labels_path, "r") as f:
        return _read_labels_from_open_file(labels_path, f, open_videos=open_videos)

sleap_io.io.slp.write_labels(labels_path, labels, embed=None, restore_original_videos=True, embed_inplace=False, verbose=True, plugin=None, embed_all_videos=True, progress_callback=None, prefer_metadata=True, preserve_unknown=False, save_embedding_vectors=False)

Write a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file to save.	required
labels	Labels	A Labels object to save.	required
embed	bool | str | list[tuple[Video, int]] | None	Frames to embed in the saved labels file. One of None, True, "all", "user", "suggestions", "user+suggestions", "source" or list of tuples of (video, frame_idx). If None is specified (the default) and the labels contains embedded frames, those embedded frames will be re-saved to the new file. If True or "all", all labeled frames and suggested frames will be embedded. If "source" is specified, no images will be embedded and the source video will be restored if available. This argument is only valid for the SLP backend.	None
restore_original_videos	bool	If True (default) and embed=False, use original video files. If False and embed=False, keep references to source .pkg.slp files. Only applies when embed=False.	True
embed_inplace	bool	If False (default), a copy of the labels is made before embedding to avoid modifying the in-memory labels. If True, the labels will be modified in-place to point to the embedded videos, which is faster but mutates the input. Only applies when embedding.	False
verbose	bool	If True (the default), display a progress bar when embedding frames.	True
plugin	str | None	Image plugin to use for encoding embedded frames. One of "opencv" or "imageio". If None, uses the global default from get_default_image_plugin(). If no global default is set, auto-detects based on available packages.	None
embed_all_videos	bool	If True (the default), all videos in the labels will be converted to embedded references, even if they have no frames to embed. This ensures package files are portable. If False, only videos with frames to embed are converted.	True
progress_callback	Callable[[int, int, str], bool] | None	Optional callback function called during embedding with (current, total, phase) arguments, where phase is "embed" (frames loaded/encoded/byte-copied) or "write" (bytes flushed to the HDF5 file). If it returns False, the operation is cancelled and ExportCancelled is raised. The phase argument is a breaking change from the previous (current, total) signature.	None
prefer_metadata	bool	If True (the default), serialize each uncropped video's shape/grayscale/fps from its backend_metadata when recorded there instead of querying the live backend, avoiding frame decoding when the metadata is already known (e.g. saving copies of labels loaded from a .slp). Set to False to always read through the live backend. See video_to_dict.	True
preserve_unknown	bool	If True, top-level HDF5 datasets/groups present in the source file (labels.provenance["filename"]) that sleap-io does not recognize are copied into the saved file. This preserves additions made by a newer sleap-io version across a load/save cycle with an older version (which would otherwise drop them, since saving rebuilds the file from the in-memory model). Default False. Best-effort: requires the source file to still exist and be readable HDF5.	False
save_embedding_vectors	bool	If False (the default), skip the /embeddings group -- appearance vectors are large on disk, so only the identity links (/identity/links, always written) are persisted by default, keeping the vectors in memory. Set True to also write the attached re-ID appearance embeddings. Off by default, mirroring embed (which embeds video frames).	False

Source code in sleap_io/io/slp.py

def write_labels(
    labels_path: str,
    labels: Labels,
    embed: bool | str | list[tuple[Video, int]] | None = None,
    restore_original_videos: bool = True,
    embed_inplace: bool = False,
    verbose: bool = True,
    plugin: str | None = None,
    embed_all_videos: bool = True,
    progress_callback: Callable[[int, int, str], bool] | None = None,
    prefer_metadata: bool = True,
    preserve_unknown: bool = False,
    save_embedding_vectors: bool = False,
):
    """Write a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file to save.
        labels: A `Labels` object to save.
        embed: Frames to embed in the saved labels file. One of `None`, `True`,
            `"all"`, `"user"`, `"suggestions"`, `"user+suggestions"`, `"source"` or list
            of tuples of `(video, frame_idx)`.

            If `None` is specified (the default) and the labels contains embedded
            frames, those embedded frames will be re-saved to the new file.

            If `True` or `"all"`, all labeled frames and suggested frames will be
            embedded.

            If `"source"` is specified, no images will be embedded and the source video
            will be restored if available.

            This argument is only valid for the SLP backend.
        restore_original_videos: If `True` (default) and `embed=False`, use original
            video files. If `False` and `embed=False`, keep references to source
            `.pkg.slp` files. Only applies when `embed=False`.
        embed_inplace: If `False` (default), a copy of the labels is made before
            embedding to avoid modifying the in-memory labels. If `True`, the
            labels will be modified in-place to point to the embedded videos,
            which is faster but mutates the input. Only applies when embedding.
        verbose: If `True` (the default), display a progress bar when embedding frames.
        plugin: Image plugin to use for encoding embedded frames. One of "opencv"
            or "imageio". If None, uses the global default from
            `get_default_image_plugin()`. If no global default is set, auto-detects
            based on available packages.
        embed_all_videos: If `True` (the default), all videos in the labels will be
            converted to embedded references, even if they have no frames to embed.
            This ensures package files are portable. If `False`, only videos with
            frames to embed are converted.
        progress_callback: Optional callback function called during embedding with
            `(current, total, phase)` arguments, where ``phase`` is ``"embed"``
            (frames loaded/encoded/byte-copied) or ``"write"`` (bytes flushed to the
            HDF5 file). If it returns `False`, the operation is cancelled and
            `ExportCancelled` is raised. The ``phase`` argument is a breaking change
            from the previous ``(current, total)`` signature.
        prefer_metadata: If `True` (the default), serialize each uncropped video's
            shape/grayscale/fps from its `backend_metadata` when recorded there
            instead of querying the live backend, avoiding frame decoding when the
            metadata is already known (e.g. saving copies of labels loaded from a
            `.slp`). Set to `False` to always read through the live backend. See
            `video_to_dict`.
        preserve_unknown: If `True`, top-level HDF5 datasets/groups present in the
            source file (``labels.provenance["filename"]``) that sleap-io does not
            recognize are copied into the saved file. This preserves additions made
            by a newer sleap-io version across a load/save cycle with an older
            version (which would otherwise drop them, since saving rebuilds the file
            from the in-memory model). Default `False`. Best-effort: requires the
            source file to still exist and be readable HDF5.
        save_embedding_vectors: If `False` (the default), skip the `/embeddings`
            group -- appearance vectors are large on disk, so only the identity
            *links* (`/identity/links`, always written) are persisted by default,
            keeping the vectors in memory. Set `True` to also write the attached
            re-ID appearance embeddings. Off by default, mirroring `embed` (which
            embeds *video frames*).
    """
    # Fast path for lazy labels (avoids materializing frames/instances)
    # Supported for simple embed modes: None, False, "source"
    if labels.is_lazy:
        # Check if embed mode requires materialization
        needs_materialization = (
            embed is True
            or embed
            in (
                "all",
                "user",
                "suggestions",
                "user+suggestions",
            )
            or isinstance(embed, list)
        )

        if needs_materialization:
            # Materialize to support embedding
            labels = labels.materialize()
        else:
            # Use fast path - copy raw arrays directly
            _write_labels_lazy(
                labels_path,
                labels,
                embed=embed,
                restore_original_videos=restore_original_videos,
                verbose=verbose,
                prefer_metadata=prefer_metadata,
                preserve_unknown=preserve_unknown,
                save_embedding_vectors=save_embedding_vectors,
            )
            return

    # Capture unknown top-level members from the source file before it is
    # truncated, so additions made by a newer sleap-io version survive a
    # load/save cycle through this (potentially older) writer.
    unknown_stash = (
        _stash_unknown_hdf5(labels.provenance.get("filename"))
        if preserve_unknown
        else None
    )

    if Path(labels_path).exists():
        Path(labels_path).unlink()

    # Make a copy to avoid mutating the input labels when embedding
    if embed and not embed_inplace:
        original_labels = labels
        labels = labels.copy(open_videos=True)

        # If embed is a list of (video, frame_idx) tuples, remap videos to the copy
        if isinstance(embed, list):
            # Create mapping from original videos to copied videos
            video_map = {
                orig: copied
                for orig, copied in zip(original_labels.videos, labels.videos)
            }
            # Remap the embed list to use copied video objects
            embed = [
                (video_map.get(video, video), frame_idx) for video, frame_idx in embed
            ]

    # Auto-collect event catalog entries + participants (event types into
    # labels.event_types, subject/target tracks/identities into labels.tracks/
    # labels.identities, and the event's own video into labels.videos) BEFORE the
    # original-videos snapshot and embedding, so an event-only video is embedded and
    # remapped like any other and every event reference is persisted (a post-hoc
    # `labels.events.append(...)` is not dropped). Mutates labels (eager path).
    labels._collect_events()

    # Store original videos before embedding modifies them
    # We need to make a copy of the actual video objects, not just the list
    original_videos = [v for v in labels.videos] if embed else None

    if embed:
        embed_videos(
            labels_path,
            labels,
            embed,
            verbose=verbose,
            plugin=plugin,
            embed_all_videos=embed_all_videos,
            progress_callback=progress_callback,
        )

    # Determine reference mode based on parameters
    if embed == "source" or (embed is False and restore_original_videos):
        reference_mode = VideoReferenceMode.RESTORE_ORIGINAL
    elif embed is False and not restore_original_videos:
        reference_mode = VideoReferenceMode.PRESERVE_SOURCE
    else:
        reference_mode = VideoReferenceMode.EMBED

    write_videos(
        labels_path,
        labels.videos,
        reference_mode=reference_mode,
        original_videos=original_videos,
        verbose=verbose,
        prefer_metadata=prefer_metadata,
    )
    # Emit virtual crop records (after videos_json so indices line up). Omitted
    # entirely when no video is cropped (uncropped files stay byte-identical).
    write_video_crops(labels_path, labels)
    write_tracks(labels_path, labels.tracks)
    # Auto-collect any detection identity not yet registered in the catalog
    # (object-identity deduped) so post-hoc `inst.identity` / `mask.identity`
    # assignments are not silently dropped on save. Mutates labels.identities
    # (eager path only).
    labels._collect_identities()
    write_identities(labels_path, labels.identities)
    write_identity_links(labels_path, labels)
    # Auto-collect any detection category not yet registered in the catalog
    # (object-identity deduped), mirroring _collect_identities, then persist the
    # /categories catalog + per-detection links (SLP 2.7+). Additive: omitted
    # entirely when there are no categories (files stay byte-identical).
    labels._collect_categories()
    write_categories(labels_path, labels.categories)
    write_category_links(labels_path, labels)
    # Frame-spanning events + their catalog (SLP 2.6+). Additive groups; omitted
    # entirely when there are no events / event types (files stay byte-identical).
    write_event_types(labels_path, labels.event_types)
    write_events(
        labels_path,
        labels.events,
        labels.videos,
        labels.event_types,
        labels.tracks,
        labels.identities,
    )
    # Identity links always persist; the (large) appearance vectors are gated by
    # save_embedding_vectors so a producer can keep them in memory but off disk.
    if save_embedding_vectors:
        write_embeddings(labels_path, labels)
        # Category appearance vectors: parallel category_* datasets in /embeddings.
        write_category_embeddings(labels_path, labels)
    write_suggestions(labels_path, labels.suggestions, labels.videos)
    write_sessions(
        labels_path,
        labels.sessions,
        labels.videos,
        labels.labeled_frames,
        identities=labels.identities,
    )
    write_metadata(labels_path, labels)
    write_lfs(labels_path, labels)
    write_negative_frames(labels_path, labels)

    # Collect all instances and build annotation lists with routing contexts
    all_instances: list[Instance | PredictedInstance] = []
    all_centroids: list = []
    centroid_contexts: list[tuple[int, int]] = []
    all_bboxes: list = []
    bbox_contexts: list[tuple[int, int]] = []
    all_masks: list = []
    mask_contexts: list[tuple[int, int]] = []
    all_label_images: list = []
    li_contexts: list[tuple[int, int]] = []
    all_rois: list = []
    roi_contexts: list[tuple[int, int]] = []

    for lf in labels.labeled_frames:
        all_instances.extend(lf.instances)
        vid_idx = labels.videos.index(lf.video) if lf.video in labels.videos else -1
        ctx = (vid_idx, lf.frame_idx)
        for c in lf.centroids:
            all_centroids.append(c)
            centroid_contexts.append(ctx)
        for b in lf.bboxes:
            all_bboxes.append(b)
            bbox_contexts.append(ctx)
        for m in lf.masks:
            all_masks.append(m)
            mask_contexts.append(ctx)
        for li in lf.label_images:
            all_label_images.append(li)
            li_contexts.append(ctx)
        for r in lf.rois:
            all_rois.append(r)
            roi_contexts.append(ctx)

    # Add static ROIs (not tied to any frame)
    for r in labels.static_rois:
        all_rois.append(r)
        roi_contexts.append(
            (labels.videos.index(r.video) if r.video in labels.videos else -1, -1)
        )

    write_rois(
        labels_path,
        all_rois,
        labels.videos,
        labels.tracks,
        all_instances,
        contexts=roi_contexts,
    )
    write_masks(
        labels_path,
        all_masks,
        labels.videos,
        labels.tracks,
        all_instances,
        contexts=mask_contexts,
    )
    write_bboxes(
        labels_path,
        all_bboxes,
        labels.videos,
        labels.tracks,
        all_instances,
        contexts=bbox_contexts,
    )
    write_centroids(
        labels_path,
        all_centroids,
        labels.videos,
        labels.tracks,
        all_instances,
        contexts=centroid_contexts,
    )
    write_label_images(
        labels_path,
        all_label_images,
        labels.videos,
        labels.tracks,
        all_instances,
        contexts=li_contexts,
    )

    # Re-emit any unknown members captured from the source file.
    _restore_unknown_hdf5(labels_path, unknown_stash)

Video I/O

sleap_io.io.slp.read_videos(labels_path, open_backend=True, *, _hdf5_file=None, _url_headers=None, _url_stream_mode='blockcache')

Read Video dataset in a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file.	required
open_backend	bool	If True (the default), attempt to open the video backend for I/O. If False, the backend will not be opened (useful for reading metadata when the video files are not available).	True
_hdf5_file	File | None	An already-open h5py.File handle to read from. If provided, the data is read from this handle (which is left open for the caller to close) and threaded into make_video; otherwise labels_path is opened and closed internally. This is a private argument used to thread a single open handle through multiple reads.	None
_url_headers	dict[str, str] | None	HTTP headers forwarded to each video backend when labels_path is a URL (so the construction-time probe is authenticated). Private; ignored for local files.	None
_url_stream_mode	str	Remote streaming strategy for URL-backed videos. Private; ignored for local files.	'blockcache'

Returns:

Type	Description
list[Video]	A list of Video objects.

Source code in sleap_io/io/slp.py

def read_videos(
    labels_path: str,
    open_backend: bool = True,
    *,
    _hdf5_file: h5py.File | None = None,
    _url_headers: dict[str, str] | None = None,
    _url_stream_mode: str = "blockcache",
) -> list[Video]:
    """Read `Video` dataset in a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file.
        open_backend: If `True` (the default), attempt to open the video backend for
            I/O. If `False`, the backend will not be opened (useful for reading metadata
            when the video files are not available).
        _hdf5_file: An already-open `h5py.File` handle to read from. If provided,
            the data is read from this handle (which is left open for the caller
            to close) and threaded into `make_video`; otherwise `labels_path` is
            opened and closed internally. This is a private argument used to thread
            a single open handle through multiple reads.
        _url_headers: HTTP headers forwarded to each video backend when
            `labels_path` is a URL (so the construction-time probe is
            authenticated). Private; ignored for local files.
        _url_stream_mode: Remote streaming strategy for URL-backed videos.
            Private; ignored for local files.

    Returns:
        A list of `Video` objects.
    """
    videos = []

    def _read_videos(f: h5py.File) -> None:
        # Read the per-video crop records once (absent on old/uncropped files).
        video_crops = read_video_crops(labels_path, _hdf5_file=f)
        videos_metadata = f["videos_json"][:]
        for video_index, video_data in enumerate(videos_metadata):
            video_json = json.loads(video_data)
            video = make_video(
                labels_path,
                video_json,
                open_backend=open_backend,
                _hdf5_file=f,
                _url_headers=_url_headers,
                _url_stream_mode=_url_stream_mode,
                _crop_entry=video_crops.get(video_index),
            )
            videos.append(video)

    # Open file once and pass handle to make_video to avoid repeated opens
    # for embedded videos (which would otherwise open the file per video).
    if _hdf5_file is not None:
        _read_videos(_hdf5_file)
    else:
        with h5py.File(labels_path, "r") as f:
            _read_videos(f)
    return videos

sleap_io.io.slp.write_videos(labels_path, videos, restore_source=False, reference_mode=None, original_videos=None, verbose=True, prefer_metadata=True)

Write video metadata to a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file.	required
videos	list[Video]	A list of Video objects to store the metadata for.	required
restore_source	bool	Deprecated. Use reference_mode instead. If True, restore source videos if available and will not re-embed the embedded images. If False (the default), will re-embed images that were previously embedded.	False
reference_mode	VideoReferenceMode | None	How to handle video references: - EMBED: Re-embed frames that were previously embedded - RESTORE_ORIGINAL: Use original video if available - PRESERVE_SOURCE: Keep reference to source file (e.g., .pkg.slp)	None
original_videos	list[Video] | None	Optional list of original video objects before embedding. Used when reference_mode is EMBED to preserve metadata.	None
verbose	bool	If True (the default), display a progress bar when embedding frames.	True
prefer_metadata	bool	If True (the default), serialize each uncropped video's shape/grayscale/fps from its backend_metadata when recorded there instead of querying the live backend (avoids decoding frames just to recompute already-known metadata). Set to False to always read through the live backend. See video_to_dict.	True

Source code in sleap_io/io/slp.py

def write_videos(
    labels_path: str,
    videos: list[Video],
    restore_source: bool = False,
    reference_mode: VideoReferenceMode | None = None,
    original_videos: list[Video] | None = None,
    verbose: bool = True,
    prefer_metadata: bool = True,
):
    """Write video metadata to a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file.
        videos: A list of `Video` objects to store the metadata for.
        restore_source: Deprecated. Use reference_mode instead. If `True`, restore
            source videos if available and will not re-embed the embedded images.
            If `False` (the default), will re-embed images that were previously
            embedded.
        reference_mode: How to handle video references:
            - EMBED: Re-embed frames that were previously embedded
            - RESTORE_ORIGINAL: Use original video if available
            - PRESERVE_SOURCE: Keep reference to source file (e.g., .pkg.slp)
        original_videos: Optional list of original video objects before embedding.
            Used when reference_mode is EMBED to preserve metadata.
        verbose: If `True` (the default), display a progress bar when embedding frames.
        prefer_metadata: If `True` (the default), serialize each uncropped video's
            shape/grayscale/fps from its `backend_metadata` when recorded there instead
            of querying the live backend (avoids decoding frames just to recompute
            already-known metadata). Set to `False` to always read through the live
            backend. See `video_to_dict`.
    """
    # Handle backwards compatibility
    if reference_mode is None:
        if restore_source:
            reference_mode = VideoReferenceMode.RESTORE_ORIGINAL
        else:
            reference_mode = VideoReferenceMode.EMBED

    videos_to_embed = []
    videos_to_write = []
    videos_to_copy = []  # For embedded videos without backend (raw HDF5 copy)

    from sleap_io.io.video_reading import CropVideoBackend

    # First determine which videos need embedding
    for video_ind, video in enumerate(videos):
        # Crop-over-embedded (D-125): a virtual crop over an embedded HDF5Video
        # must route through the embed machinery, not the plain-external else
        # branch. Test embedded-ness / collect source_inds from the INNER backend;
        # the crop itself still rides /video_crops.
        inner_backend = (
            video.backend.inner
            if isinstance(video.backend, CropVideoBackend)
            else video.backend
        )
        # Check if video has an open backend with embedded images
        has_backend_with_embedded = (
            type(inner_backend) is HDF5Video and inner_backend.has_embedded_images
        )
        # Also detect embedded videos via metadata (when backend is None)
        has_embedded_via_metadata = (
            video.backend is None and _is_embedded_video_metadata(video)
        )

        if has_backend_with_embedded:
            if reference_mode == VideoReferenceMode.RESTORE_ORIGINAL:
                if video.source_video is None:
                    # No source video available, reference the current embedded video
                    # file
                    videos_to_write.append((video_ind, video))
                else:
                    # Use the source video
                    videos_to_write.append((video_ind, video.source_video))
            elif reference_mode == VideoReferenceMode.PRESERVE_SOURCE:
                # Keep the reference to the source .pkg.slp file
                videos_to_write.append((video_ind, video))
            else:  # EMBED mode
                # If the video has embedded images, check if we need to re-embed them
                already_embedded = False
                if Path(labels_path).exists():
                    with h5py.File(labels_path, "r") as f:
                        already_embedded = f"video{video_ind}/video" in f

                if already_embedded:
                    videos_to_write.append((video_ind, video))
                else:
                    # Collect information for embedding (source_inds live on the
                    # inner backend for a crop-over-embedded video, D-125). Embed
                    # ``video`` itself (the crop facade): process_and_embed_frames
                    # special-cases a CropVideoBackend and reads the UNCROPPED frame
                    # straight from the embedded inner, so the embedded data + its
                    # videos_json entry describe the full frame WITHOUT touching the
                    # (possibly missing external) source_video. The crop rides
                    # /video_crops and is re-applied on read.
                    embed_target = video
                    frames_to_embed = [
                        (embed_target, frame_idx)
                        for frame_idx in inner_backend.source_inds
                    ]
                    videos_to_embed.append((video_ind, embed_target, frames_to_embed))
        elif has_embedded_via_metadata:
            # Video has embedded frames but backend is not open (open_videos=False)
            if reference_mode == VideoReferenceMode.RESTORE_ORIGINAL:
                if video.source_video is None:
                    videos_to_write.append((video_ind, video))
                else:
                    videos_to_write.append((video_ind, video.source_video))
            elif reference_mode == VideoReferenceMode.PRESERVE_SOURCE:
                videos_to_write.append((video_ind, video))
            else:  # EMBED mode
                # Check if already embedded in destination
                already_embedded = False
                if Path(labels_path).exists():
                    with h5py.File(labels_path, "r") as f:
                        already_embedded = f"video{video_ind}/video" in f

                if already_embedded:
                    videos_to_write.append((video_ind, video))
                else:
                    # Need to copy raw HDF5 data from source file
                    videos_to_copy.append((video_ind, video))
        else:
            videos_to_write.append((video_ind, video))

    # Process videos that need embedding
    if videos_to_embed:
        # Prepare all frames to embed
        all_frames_to_embed = []
        for video_ind, video, frames in videos_to_embed:
            for frame in frames:
                all_frames_to_embed.append(frame)

        # Create a temporary Labels object for embedding
        temp_labels = Labels(
            videos=[v for _, v, _ in videos_to_embed], labeled_frames=[]
        )

        # Prepare and embed all frames in a single process
        frames_metadata = prepare_frames_to_embed(
            labels_path, temp_labels, all_frames_to_embed
        )
        replaced_videos = process_and_embed_frames(
            labels_path,
            frames_metadata,
            image_format=[
                v.backend.image_format if hasattr(v.backend, "image_format") else "png"
                for _, v, _ in videos_to_embed
            ][0],  # Use the first video's format
            verbose=verbose,
        )

        # Add the embedded videos to the list
        for video_ind, video, _ in videos_to_embed:
            if video in replaced_videos:
                videos_to_write.append((video_ind, replaced_videos[video]))

    # Copy raw HDF5 data for embedded videos without backends
    if videos_to_copy:
        for video_ind, video in videos_to_copy:
            # Get the source file path (video.filename points to the source pkg.slp)
            source_path = video.filename
            if not Path(source_path).exists():
                # Can't copy if source doesn't exist, just write metadata
                videos_to_write.append((video_ind, video))
                continue

            # Get the source dataset name from backend_metadata
            meta = video.backend_metadata
            source_dataset = meta.get("dataset", "") if meta else ""
            if not source_dataset:
                videos_to_write.append((video_ind, video))
                continue

            # Extract the video group name (e.g., "video0" from "video0/video")
            source_group = source_dataset.split("/")[0] if "/" in source_dataset else ""
            if not source_group:
                videos_to_write.append((video_ind, video))
                continue

            # Destination group name uses the current video index
            dest_group = f"video{video_ind}"

            # Copy the entire video group from source to destination
            with h5py.File(source_path, "r") as src_f:
                if source_group not in src_f:
                    videos_to_write.append((video_ind, video))
                    continue

                with h5py.File(labels_path, "a") as dst_f:
                    # Copy the video group with all its datasets and attributes
                    src_f.copy(source_group, dst_f, name=dest_group)

            # Add to videos_to_write - the metadata will reference the copied data
            videos_to_write.append((video_ind, video))

    # Write video metadata
    video_jsons = []
    for video_ind, video in sorted(videos_to_write, key=lambda x: x[0]):
        video_json = video_to_dict(video, labels_path, prefer_metadata=prefer_metadata)
        video_jsons.append(np.bytes_(json.dumps(video_json, separators=(",", ":"))))

    with h5py.File(labels_path, "a") as f:
        if "videos_json" not in f:
            f.create_dataset("videos_json", data=video_jsons, maxshape=(None,))

    # Save source_video lineage metadata in a separate pass to ensure video groups exist
    # Note: original_video is now a computed property derived from source_video chain,
    # so we only need to store source_video (immediate parent).
    with h5py.File(labels_path, "a") as f:
        for video_ind, video in enumerate(videos):
            dataset = f"video{video_ind}"

            # If original_videos is provided (e.g., during embedding), use those
            pre_embed_video = original_videos[video_ind] if original_videos else video

            # Determine source_video to save based on reference mode
            source_to_save = None
            if reference_mode != VideoReferenceMode.PRESERVE_SOURCE:
                if reference_mode == VideoReferenceMode.EMBED and original_videos:
                    # For embed mode, save the pre-embedding video as source
                    source_to_save = pre_embed_video
                elif (
                    pre_embed_video is not None
                    and pre_embed_video.source_video is not None
                ):
                    source_to_save = pre_embed_video.source_video

            # Write source_video metadata to the video group
            if dataset in f and source_to_save is not None:
                video_group = f[dataset]

                # For EMBED mode with original_videos, we need to overwrite
                # source_video because embed_videos saves the wrong metadata
                if (
                    reference_mode == VideoReferenceMode.EMBED
                    and original_videos
                    and "source_video" in video_group
                ):
                    del video_group["source_video"]

                if "source_video" not in video_group:
                    source_grp = video_group.require_group("source_video")
                    source_json = video_to_dict(
                        source_to_save, labels_path, prefer_metadata=prefer_metadata
                    )
                    _write_source_video_json(source_grp, source_json)

sleap_io.io.slp.embed_videos(labels_path, labels, embed, verbose=True, plugin=None, embed_all_videos=True, progress_callback=None)

Embed videos in a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file to save.	required
labels	Labels	A Labels object to save.	required
embed	bool | str | list[tuple[Video, int]]	Frames to embed in the saved labels file. One of None, True, "all", "user", "suggestions", "user+suggestions", "source" or list of tuples of (video, frame_idx). If None is specified (the default) and the labels contains embedded frames, those embedded frames will be re-saved to the new file. If True or "all", all labeled frames and suggested frames will be embedded.	required
verbose	bool	If True (the default), display a progress bar for the embedding process.	True
plugin	str | None	Image plugin to use for encoding. One of "opencv" or "imageio". If None, uses the global default from get_default_image_plugin(). If "source" is specified, no images will be embedded and the source video will be restored if available. This argument is only valid for the SLP backend.	None
embed_all_videos	bool	If True (the default), all videos in the labels will be converted to embedded references, even if they have no frames to embed. This ensures package files are portable. If False, only videos with frames to embed are converted.	True
progress_callback	Callable[[int, int, str], bool] | None	Optional callback function called during embedding with (current, total, phase) arguments, where phase is "embed" (frames loaded/encoded/byte-copied) or "write" (bytes flushed to the HDF5 file). If it returns False, the operation is cancelled and ExportCancelled is raised. The phase argument is a breaking change from the previous (current, total) signature.	None

Source code in sleap_io/io/slp.py

def embed_videos(
    labels_path: str,
    labels: Labels,
    embed: bool | str | list[tuple[Video, int]],
    verbose: bool = True,
    plugin: str | None = None,
    embed_all_videos: bool = True,
    progress_callback: Callable[[int, int, str], bool] | None = None,
):
    """Embed videos in a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file to save.
        labels: A `Labels` object to save.
        embed: Frames to embed in the saved labels file. One of `None`, `True`,
            `"all"`, `"user"`, `"suggestions"`, `"user+suggestions"`, `"source"` or list
            of tuples of `(video, frame_idx)`.

            If `None` is specified (the default) and the labels contains embedded
            frames, those embedded frames will be re-saved to the new file.

            If `True` or `"all"`, all labeled frames and suggested frames will be
            embedded.
        verbose: If `True` (the default), display a progress bar for the embedding
            process.
        plugin: Image plugin to use for encoding. One of "opencv" or "imageio".
            If None, uses the global default from `get_default_image_plugin()`.

            If `"source"` is specified, no images will be embedded and the source video
            will be restored if available.

            This argument is only valid for the SLP backend.
        embed_all_videos: If `True` (the default), all videos in the labels will be
            converted to embedded references, even if they have no frames to embed.
            This ensures package files are portable. If `False`, only videos with
            frames to embed are converted.
        progress_callback: Optional callback function called during embedding with
            `(current, total, phase)` arguments, where ``phase`` is ``"embed"``
            (frames loaded/encoded/byte-copied) or ``"write"`` (bytes flushed to the
            HDF5 file). If it returns `False`, the operation is cancelled and
            `ExportCancelled` is raised. The ``phase`` argument is a breaking change
            from the previous ``(current, total)`` signature.
    """
    if embed is True:
        embed = "all"
    if embed == "user":
        embed = [(lf.video, lf.frame_idx) for lf in labels.user_labeled_frames]
    elif embed == "suggestions":
        embed = [(sf.video, sf.frame_idx) for sf in labels.suggestions]
    elif embed == "user+suggestions":
        embed = [(lf.video, lf.frame_idx) for lf in labels.user_labeled_frames]
        embed += [(sf.video, sf.frame_idx) for sf in labels.suggestions]
    elif embed == "all":
        embed = [(lf.video, lf.frame_idx) for lf in labels]
        embed += [(sf.video, sf.frame_idx) for sf in labels.suggestions]
    elif embed == "source":
        embed = []
    elif isinstance(embed, list):
        embed = embed
    else:
        raise ValueError(f"Invalid value for embed: {embed}")

    embed_frames(
        labels_path,
        labels,
        embed,
        verbose=verbose,
        plugin=plugin,
        embed_all_videos=embed_all_videos,
        progress_callback=progress_callback,
    )

Skeleton I/O

sleap_io.io.slp.read_skeletons(labels_path, *, _hdf5_file=None)

Read Skeleton dataset from a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string that contains the path to the labels file.	required
_hdf5_file	File | None	An already-open h5py.File handle to read from. If provided, the data is read from this handle (left open for the caller to close); otherwise labels_path is opened and closed internally. This is a private argument used to thread a single open handle through reads.	None

Returns:

Type	Description
list[Skeleton]	A list of Skeleton objects.

Source code in sleap_io/io/slp.py

def read_skeletons(
    labels_path: str, *, _hdf5_file: h5py.File | None = None
) -> list[Skeleton]:
    """Read `Skeleton` dataset from a SLEAP labels file.

    Args:
        labels_path: A string that contains the path to the labels file.
        _hdf5_file: An already-open `h5py.File` handle to read from. If provided,
            the data is read from this handle (left open for the caller to close);
            otherwise `labels_path` is opened and closed internally. This is a
            private argument used to thread a single open handle through reads.

    Returns:
        A list of `Skeleton` objects.
    """
    metadata = read_metadata(labels_path, _hdf5_file=_hdf5_file)

    # Get node names. This is a superset of all nodes across all skeletons. Note that
    # node ordering is specific to each skeleton, so we'll need to fix this afterwards.
    node_names = [x["name"] for x in metadata["nodes"]]

    # Use the SLP skeleton decoder
    decoder = SkeletonSLPDecoder()
    return decoder.decode(metadata, node_names)

sleap_io.io.slp.serialize_skeletons(skeletons)

Serialize a list of Skeleton objects to JSON-compatible dicts.

Parameters:

Name	Type	Description	Default
skeletons	list[Skeleton]	A list of Skeleton objects.	required

Returns:

Type	Description
tuple[list[dict], list[dict]]	A tuple of skeletons_dicts, nodes_dicts. nodes_dicts is a list of dicts containing the nodes in all the skeletons. skeletons_dicts is a list of dicts containing the skeletons.

Notes

This function attempts to replicate the serialization of skeletons in legacy SLEAP which relies on a combination of networkx's graph serialization and our own metadata used to store nodes and edges independent of the graph structure.

However, because sleap-io does not currently load in the legacy metadata, this function will not produce byte-level compatible serialization with legacy formats, even though the ordering and all attributes of nodes and edges should match up.

Source code in sleap_io/io/slp.py

def serialize_skeletons(skeletons: list[Skeleton]) -> tuple[list[dict], list[dict]]:
    """Serialize a list of `Skeleton` objects to JSON-compatible dicts.

    Args:
        skeletons: A list of `Skeleton` objects.

    Returns:
        A tuple of `skeletons_dicts, nodes_dicts`.

        `nodes_dicts` is a list of dicts containing the nodes in all the skeletons.

        `skeletons_dicts` is a list of dicts containing the skeletons.

    Notes:
        This function attempts to replicate the serialization of skeletons in legacy
        SLEAP which relies on a combination of networkx's graph serialization and our
        own metadata used to store nodes and edges independent of the graph structure.

        However, because sleap-io does not currently load in the legacy metadata, this
        function will not produce byte-level compatible serialization with legacy
        formats, even though the ordering and all attributes of nodes and edges should
        match up.
    """
    # Use the SLP skeleton encoder
    encoder = SkeletonSLPEncoder()
    return encoder.encode_skeletons(skeletons)

Instance I/O

sleap_io.io.slp.read_instances(labels_path, skeletons, tracks, points, pred_points, format_id, *, _hdf5_file=None)

Read Instance dataset in a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file.	required
skeletons	list[Skeleton]	A list of Skeleton objects (see read_skeletons).	required
tracks	list[Track]	A list of Track objects (see read_tracks).	required
points	ndarray	A structured array of point data (see read_points).	required
pred_points	ndarray	A structured array of predicted point data (see read_pred_points).	required
format_id	float	The format version identifier used to specify the format of the input file.	required
_hdf5_file	File | None	An already-open h5py.File handle to read from. If provided, the data is read from this handle (left open for the caller to close); otherwise labels_path is opened and closed internally. This is a private argument used to thread a single open handle through reads.	None

Returns:

Type	Description
list[Instance | PredictedInstance]	A list of Instance and/or PredictedInstance objects.

Source code in sleap_io/io/slp.py

def read_instances(
    labels_path: str,
    skeletons: list[Skeleton],
    tracks: list[Track],
    points: np.ndarray,
    pred_points: np.ndarray,
    format_id: float,
    *,
    _hdf5_file: h5py.File | None = None,
) -> list[Instance | PredictedInstance]:
    """Read `Instance` dataset in a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file.
        skeletons: A list of `Skeleton` objects (see `read_skeletons`).
        tracks: A list of `Track` objects (see `read_tracks`).
        points: A structured array of point data (see `read_points`).
        pred_points: A structured array of predicted point data (see
            `read_pred_points`).
        format_id: The format version identifier used to specify the format of the input
            file.
        _hdf5_file: An already-open `h5py.File` handle to read from. If provided,
            the data is read from this handle (left open for the caller to close);
            otherwise `labels_path` is opened and closed internally. This is a
            private argument used to thread a single open handle through reads.

    Returns:
        A list of `Instance` and/or `PredictedInstance` objects.
    """
    instances_data = read_hdf5_dataset(labels_path, "instances", _hdf5_file=_hdf5_file)

    instances = {}
    from_predicted_pairs = []
    for instance_data in instances_data:
        if format_id < 1.2:
            (
                instance_id,
                instance_type,
                frame_id,
                skeleton_id,
                track_id,
                from_predicted,
                instance_score,
                point_id_start,
                point_id_end,
            ) = instance_data
            tracking_score = 0.0
        elif format_id >= 1.2:
            (
                instance_id,
                instance_type,
                frame_id,
                skeleton_id,
                track_id,
                from_predicted,
                instance_score,
                point_id_start,
                point_id_end,
                tracking_score,
            ) = instance_data

        # Cast index values to int for h5wasm compatibility. h5wasm may write
        # all columns as float64, which can't be used as list indices or slice
        # bounds. Safe for compound dtypes too: int(numpy.int64(x)) -> int.
        instance_id = int(instance_id)
        skeleton_id = int(skeleton_id)
        track_id = int(track_id)
        from_predicted = int(from_predicted)
        point_id_start = int(point_id_start)
        point_id_end = int(point_id_end)

        skeleton = skeletons[skeleton_id]
        track = tracks[track_id] if track_id >= 0 else None

        if instance_type == InstanceType.USER:
            pts_data = points[point_id_start:point_id_end]
            # Fast path: Build PointsArray directly from HDF5 data
            points_array = _points_from_hdf5_data(
                pts_data, skeleton, is_predicted=False
            )
            if format_id < 1.1:
                # Legacy coordinate system: top-left of pixel is (0, 0)
                # Adjust to new system: center of pixel is (0, 0)
                points_array["xy"] -= 0.5
            inst = Instance(
                points_array,
                skeleton=skeleton,
                track=track,
                tracking_score=tracking_score,
            )
            instances[instance_id] = inst

        elif instance_type == InstanceType.PREDICTED:
            pts_data = pred_points[point_id_start:point_id_end]
            # Fast path: Build PredictedPointsArray directly from HDF5 data
            points_array = _points_from_hdf5_data(pts_data, skeleton, is_predicted=True)
            if format_id < 1.1:
                # Legacy coordinate system: top-left of pixel is (0, 0)
                # Adjust to new system: center of pixel is (0, 0)
                points_array["xy"] -= 0.5
            inst = PredictedInstance(
                points_array,
                skeleton=skeleton,
                track=track,
                score=instance_score,
                tracking_score=tracking_score,
            )
            instances[instance_id] = inst

        if from_predicted >= 0:
            from_predicted_pairs.append((instance_id, from_predicted))

    # Link instances based on from_predicted field.
    for instance_id, from_predicted in from_predicted_pairs:
        instances[instance_id].from_predicted = instances[from_predicted]

    # Convert instances back to list.
    instances = list(instances.values())

    return instances

sleap_io.io.slp.read_points(labels_path, *, _hdf5_file=None)

Read points dataset from a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file.	required
_hdf5_file	File | None	An already-open h5py.File handle to read from. If provided, the data is read from this handle (left open for the caller to close); otherwise labels_path is opened and closed internally. This is a private argument used to thread a single open handle through reads.	None

Returns:

Type	Description
ndarray	A structured array of point data.

Source code in sleap_io/io/slp.py

def read_points(labels_path: str, *, _hdf5_file: h5py.File | None = None) -> np.ndarray:
    """Read points dataset from a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file.
        _hdf5_file: An already-open `h5py.File` handle to read from. If provided,
            the data is read from this handle (left open for the caller to close);
            otherwise `labels_path` is opened and closed internally. This is a
            private argument used to thread a single open handle through reads.

    Returns:
        A structured array of point data.
    """
    pts = read_hdf5_dataset(labels_path, "points", _hdf5_file=_hdf5_file)
    return pts

sleap_io.io.slp.read_pred_points(labels_path, *, _hdf5_file=None)

Read predicted points dataset from a SLEAP labels file.

Parameters:

Name	Type	Description	Default
labels_path	str	A string path to the SLEAP labels file.	required
_hdf5_file	File | None	An already-open h5py.File handle to read from. If provided, the data is read from this handle (left open for the caller to close); otherwise labels_path is opened and closed internally. This is a private argument used to thread a single open handle through reads.	None

Returns:

Type	Description
ndarray	A structured array of predicted point data.

Source code in sleap_io/io/slp.py

def read_pred_points(
    labels_path: str, *, _hdf5_file: h5py.File | None = None
) -> np.ndarray:
    """Read predicted points dataset from a SLEAP labels file.

    Args:
        labels_path: A string path to the SLEAP labels file.
        _hdf5_file: An already-open `h5py.File` handle to read from. If provided,
            the data is read from this handle (left open for the caller to close);
            otherwise `labels_path` is opened and closed internally. This is a
            private argument used to thread a single open handle through reads.

    Returns:
        A structured array of predicted point data.
    """
    pred_pts = read_hdf5_dataset(labels_path, "pred_points", _hdf5_file=_hdf5_file)
    return pred_pts

Lazy Loading

sleap_io.io.slp_lazy.LazyDataStore

Holds raw HDF5 data and provides lazy access methods.

Attributes:

Name	Type	Description
frames_data		Structured array from /frames HDF5 dataset. Fields: frame_id, video_id, frame_idx, instance_id_start, instance_id_end.
instances_data		Structured array from /instances HDF5 dataset. Fields vary by format_id but include: instance_id, instance_type, frame_id, skeleton_id, track_id, from_predicted, instance_score, point_id_start, point_id_end, and optionally tracking_score.
pred_points_data		Structured array from /pred_points HDF5 dataset. Fields: x, y, visible, complete, score.
points_data		Structured array from /points HDF5 dataset. Fields: x, y, visible, complete.
videos		List of eagerly loaded Video objects.
skeletons		List of eagerly loaded Skeleton objects.
tracks		List of eagerly loaded Track objects.
format_id		SLP format version.
_source_path		Path to source SLP file (for debugging).

Methods:

Name	Description
materialize_frame	Create a fully materialized LabeledFrame.
materialize_all	Materialize all frames.
to_numpy	Build numpy array directly from raw data (fast path).
get_user_frame_indices	Find indices of frames containing user (non-predicted) instances.

materialize_frame(idx)

Create a fully materialized LabeledFrame.

Parameters:

Name	Type	Description	Default
idx	int	Index into frames_data array.	required

Returns:

Type	Description
LabeledFrame	A real LabeledFrame with real Instance objects.

materialize_all()

Materialize all frames.

Returns:

Type	Description
list[LabeledFrame]	List of all LabeledFrame objects.

to_numpy(video=None, untracked=False, return_confidence=False, user_instances=True)

Build numpy array directly from raw data (fast path).

This method builds the output array directly from raw HDF5 data without creating any Instance or LabeledFrame objects, providing significant performance improvement for workflows that only need numpy output.

Parameters:

Name	Type	Description	Default
video	Video | None	Video to filter by. If None, uses the first video.	None
untracked	bool	If True, index by instance order instead of tracks. If False (default), organize instances by their track assignment.	False
return_confidence	bool	If True, include confidence as third coordinate. For user instances, confidence is set to 1.0.	False
user_instances	bool	If True (default), prefer user instances over predicted instances. If False, only include predicted instances.	True

Returns:

Type	Description
ndarray	Array of shape (n_frames, n_tracks, n_nodes, 2) or (n_frames, n_tracks, n_nodes, 3) if return_confidence is True. Missing data is filled with np.nan.

get_user_frame_indices()

Find indices of frames containing user (non-predicted) instances.

This also includes frames marked as negative (is_negative=True), since those are considered user-labeled even though they have no instances.

Returns:

Type	Description
list[int]	List of frame indices (into frames_data) that have at least one user instance or are marked as negative.

sleap_io.io.slp_lazy.LazyFrameList

List-like proxy that materializes LabeledFrame objects on access.

This provides backward compatibility for code that accesses labels.labeled_frames directly. Frames are created on-demand when accessed via indexing or iteration.

Mutations are blocked with helpful error messages suggesting to call labels.materialize() first.

Methods:

Name	Description
__delitem__	Block item deletion with helpful error.
__getitem__	Get frame(s) by index or slice.
__init__	Initialize with a LazyDataStore.
__iter__	Iterate over frames, materializing each.
__len__	Return number of frames.
__repr__	Return informative representation.
__setitem__	Block item assignment with helpful error.
append	Block append with helpful error.
extend	Block extend with helpful error.
insert	Block insert with helpful error.

Attributes:

Name	Type	Description
__dict__		Read-only proxy of a mapping.
__doc__		str(object='') -> str
__firstlineno__		int([x]) -> integer
__module__		str(object='') -> str
__static_attributes__		Built-in immutable sequence.
__weakref__		list of weak references to the object

__dict__ = mappingproxy({'__module__': 'sleap_io.io.slp_lazy', '__firstlineno__': 856, '__doc__': 'List-like proxy that materializes LabeledFrame objects on access.\n\nThis provides backward compatibility for code that accesses\nlabels.labeled_frames directly. Frames are created on-demand when\naccessed via indexing or iteration.\n\nMutations are blocked with helpful error messages suggesting to call\nlabels.materialize() first.\n', '__init__': <function LazyFrameList.__init__ at 0x7f07ecdf9800>, '__len__': <function LazyFrameList.__len__ at 0x7f07ecdf98a0>, '__getitem__': <function LazyFrameList.__getitem__ at 0x7f07ecdf9d00>, '__iter__': <function LazyFrameList.__iter__ at 0x7f07ecdfb1a0>, '__repr__': <function LazyFrameList.__repr__ at 0x7f07ecdfb240>, '_mutation_error': <function LazyFrameList._mutation_error at 0x7f07ecdfb2e0>, 'append': <function LazyFrameList.append at 0x7f07ecdfb380>, 'extend': <function LazyFrameList.extend at 0x7f07ecdfb420>, 'insert': <function LazyFrameList.insert at 0x7f07ecdfb4c0>, '__setitem__': <function LazyFrameList.__setitem__ at 0x7f07ecdfb560>, '__delitem__': <function LazyFrameList.__delitem__ at 0x7f07ecdfb600>, '__static_attributes__': ('_store', '_supplementary'), '__dict__': <attribute '__dict__' of 'LazyFrameList' objects>, '__weakref__': <attribute '__weakref__' of 'LazyFrameList' objects>}) class-attribute

Read-only proxy of a mapping.

__doc__ = 'List-like proxy that materializes LabeledFrame objects on access.\n\nThis provides backward compatibility for code that accesses\nlabels.labeled_frames directly. Frames are created on-demand when\naccessed via indexing or iteration.\n\nMutations are blocked with helpful error messages suggesting to call\nlabels.materialize() first.\n' class-attribute

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__firstlineno__ = 856 class-attribute

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.

int('0b100', base=0) 4

__module__ = 'sleap_io.io.slp_lazy' class-attribute

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__static_attributes__ = ('_store', '_supplementary') class-attribute

Built-in immutable sequence.

If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.

If the argument is a tuple, the return value is the same object.

__weakref__ property

list of weak references to the object

__delitem__(idx)

Block item deletion with helpful error.

Raises:

Type	Description
RuntimeError	Always, with guidance to materialize first.

__getitem__(idx)

Get frame(s) by index or slice.

Parameters:

Name	Type	Description	Default
idx	int | slice	Integer index or slice object.	required

Returns:

Type	Description
LabeledFrame | list[LabeledFrame]	A single LabeledFrame for integer indexing, or a list of LabeledFrames for slicing.

Raises:

Type	Description
IndexError	If index is out of range.

__init__(store)

Initialize with a LazyDataStore.

Parameters:

Name	Type	Description	Default
store	LazyDataStore	The LazyDataStore containing raw frame data.	required

__iter__()

Iterate over frames, materializing each.

__len__()

Return number of frames.

__repr__()

Return informative representation.

__setitem__(idx, value)

Block item assignment with helpful error.

Raises:

Type	Description
RuntimeError	Always, with guidance to materialize first.

append(item)

Block append with helpful error.

Raises:

Type	Description
RuntimeError	Always, with guidance to materialize first.

extend(items)

Block extend with helpful error.

Raises:

Type	Description
RuntimeError	Always, with guidance to materialize first.

insert(idx, item)

Block insert with helpful error.

Raises:

Type	Description
RuntimeError	Always, with guidance to materialize first.