Codecs¶

Convert pose tracking data to dictionaries, NumPy arrays, and DataFrames for analysis and interoperability.

What are codecs?¶

Codecs perform in-memory format conversion between Labels objects and common data representations:

Codec	Output	Use Case
Dictionary	`dict`	JSON export, serialization, inspection
NumPy	`np.ndarray`	Numerical analysis, ML pipelines
DataFrame	`pd.DataFrame` / `pl.DataFrame`	Tabular analysis, visualization

Codecs are distinct from I/O operations: they convert data structures in memory rather than reading/writing files. This separation enables flexible workflows:

# Codec + I/O composition
df = labels.to_dataframe()     # Codec: Labels → DataFrame
df.to_csv("poses.csv")         # I/O: DataFrame → disk

Quick start¶

import sleap_io as sio

# Load pose tracking data
labels = sio.load_file("predictions.slp")

# Convert to dictionary (JSON-serializable)
data = labels.to_dict()

# Convert to NumPy array
# Shape: (n_frames, n_tracks, n_nodes, 2); n_frames == len(video) in v0.7.0
tracks = labels.numpy()

# Convert to pandas DataFrame
df = labels.to_dataframe()

Each codec also supports decoding (reconstruction from the converted format):

from sleap_io.codecs import from_dict, from_dataframe, from_numpy

# Reconstruct Labels from dictionary
labels = from_dict(data)

# Reconstruct from DataFrame (requires video/skeleton context)
labels = from_dataframe(df, video=video, skeleton=skeleton)

# Reconstruct from NumPy array
labels = from_numpy(tracks, video=video, skeleton=skeleton)

Dictionary codec¶

The dictionary codec converts Labels to a JSON-serializable Python dictionary with complete structural metadata.

Encoding with `to_dict()`¶

import sleap_io as sio

labels = sio.load_file("predictions.slp")
data = labels.to_dict()

Output structure:

{
    "version": "1.0.0",
    "skeletons": [
        {
            "name": "fly",
            "nodes": ["head", "thorax", "abdomen"],
            "edges": [[0, 1], [1, 2]]
        }
    ],
    "videos": [
        {"filename": "video.mp4", "shape": [1000, 1024, 1024, 1]}
    ],
    "tracks": [
        {"name": "fly_1"}
    ],
    "labeled_frames": [
        {
            "frame_idx": 0,
            "video_idx": 0,
            "instances": [
                {
                    "type": "predicted_instance",
                    "skeleton_idx": 0,
                    "points": [
                        {"x": 100.0, "y": 200.0, "visible": true, "complete": false},
                        {"x": 150.0, "y": 250.0, "visible": true, "complete": false},
                        {"x": 200.0, "y": 300.0, "visible": true, "complete": false}
                    ],
                    "track_idx": 0,
                    "score": 0.91
                }
            ]
        },
        {
            "frame_idx": 12,
            "video_idx": 0,
            "instances": [],
            "is_negative": true
        }
    ],
    "suggestions": [],
    "provenance": {}
}

The dictionary uses index-based references (skeleton_idx, video_idx, track_idx) for compactness. All values are JSON-serializable primitives.

Frames explicitly marked as containing no instances (negative training examples) are preserved in v0.7.0 via an is_negative: true field on each frame dict — round-tripping through to_dict() / from_dict() no longer drops them (PR #369).

Parameters:

video: Filter to a specific video (by object or index)
skip_empty_frames: Exclude frames with no instances

# Export only frames with instances from the first video
data = labels.to_dict(video=0, skip_empty_frames=True)

Decoding with `from_dict()`¶

from sleap_io.codecs import from_dict

# Reconstruct Labels from dictionary
labels = from_dict(data)

The decoder reconstructs the full object graph including skeletons, videos, tracks, and instances.

Limitations

Video backends are not restored (videos are created with filename only)
The from_predicted relationship is indicated but not fully linked

JSON export workflow¶

import json

# Export to JSON file
data = labels.to_dict()
with open("labels.json", "w") as f:
    json.dump(data, f, indent=2)

# Import from JSON file
with open("labels.json", "r") as f:
    data = json.load(f)
labels = from_dict(data)

NumPy codec¶

The NumPy codec converts Labels to a 4D array of track coordinates, suitable for numerical analysis and ML pipelines.

Encoding with `to_numpy()` / `Labels.numpy()`¶

import sleap_io as sio

labels = sio.load_file("predictions.slp")
tracks = labels.numpy()

print(tracks.shape)
# (n_frames, n_tracks, n_nodes, 2)

In v0.7.0, n_frames == len(video) — the full video duration. Frames past the last labeled frame are NaN-padded along all coordinate axes (PR #368). Code that previously assumed tracks.shape[0] == last_labeled_frame + 1 may need updating.

Output array:

Shape: (2, 2, 2, 2)
       (n_frames=2, n_tracks=2, n_nodes=2, coords=2)

Frame 0, Track 0:
[[100. 200.]    # head: x=100, y=200
 [150. 250.]]   # tail: x=150, y=250

Frame 0, Track 1:
[[300. 400.]    # head: x=300, y=400
 [350. 450.]]   # tail: x=350, y=450

Missing points are represented as np.nan.

Include confidence scores:

tracks = labels.numpy(return_confidence=True)
print(tracks.shape)
# (n_frames, n_tracks, n_nodes, 3)  # x, y, score

# Frame 0, Track 0 with scores:
# [[100.   200.     0.95]    # head: x, y, score
#  [150.   250.     0.92]]   # tail: x, y, score

Parameters:

Parameter	Default	Description
`video`	`None` (first)	Video to convert
`untracked`	`False`	Include untracked instances
`return_confidence`	`False`	Include point scores (3^rd coord)
`user_instances`	`True`	Include user-labeled instances
`predicted_instances`	`True`	Include predicted instances

# Get only predicted instances with confidence scores
tracks = labels.numpy(
    user_instances=False,
    predicted_instances=True,
    return_confidence=True
)

Decoding with `from_numpy()` / `Labels.from_numpy()`¶

from sleap_io.codecs import from_numpy
import numpy as np

# Create array: 100 frames, 2 tracks, 3 nodes, xy coordinates
tracks = np.random.rand(100, 2, 3, 2) * 500

# Reconstruct Labels (requires skeleton and video)
labels = from_numpy(
    tracks,
    video=video,
    skeleton=skeleton,
    track_names=["animal_1", "animal_2"]
)

Parameters:

Parameter	Description
`tracks_array`	4D array of shape `(frames, tracks, nodes, 2 or 3)`
`video`	Video object for the labels
`skeleton`	Skeleton defining node structure
`tracks`	List of Track objects (or auto-created)
`track_names`	Names for auto-created tracks
`first_frame`	Starting frame index (default: 0)
`return_confidence`	Whether array contains scores

DataFrame codec¶

The DataFrame codec converts Labels to tabular format with multiple layout options, supporting both pandas and polars backends.

Encoding with `to_dataframe()` / `Labels.to_dataframe()`¶

import sleap_io as sio

labels = sio.load_file("predictions.slp")
df = labels.to_dataframe()

Four output formats are available:

Format	Structure	Use Case
`points`	One row per point	Normalized, round-trip compatible
`instances`	One row per instance	Feature extraction, ML
`frames`	One row per frame	Trajectory analysis
`multi_index`	Hierarchical columns	NWB-style, pivot tables

Points format (default)¶

One row per (frame, instance, node) combination—the most normalized representation.

df = labels.to_dataframe(format="points")

   frame_idx     node      x      y  track  track_score  instance_score  score
0          0     head  100.0  200.0   None          NaN             NaN    NaN
1          0   thorax  150.0  250.0   None          NaN             NaN    NaN
2          0  abdomen  200.0  300.0   None          NaN             NaN    NaN
3          0     head  105.0  205.0  fly_1         0.98            0.91   0.95
4          0   thorax  155.0  255.0  fly_1         0.98            0.91   0.92
5          0  abdomen  205.0  305.0  fly_1         0.98            0.91   0.88

Columns:

frame_idx: Frame number
node: Body part name
x, y: Point coordinates
track: Track name (or None for untracked)
track_score: Tracking confidence
instance_score: Instance-level prediction score
score: Per-point confidence score

This format supports full round-trip decoding with from_dataframe().

Instances format¶

One row per instance—coordinates spread across columns.

df = labels.to_dataframe(format="instances")

   frame_idx  track  track_score  score  head.x  head.y  head.score  tail.x  tail.y  tail.score
0          0  fly_1         0.98   0.91   100.0   200.0        0.95   150.0   250.0        0.92
1          0  fly_2         0.95   0.86   300.0   400.0        0.88   350.0   450.0        0.85

Each node has columns {node}.x, {node}.y, and {node}.score. This format is useful for computing per-instance features.

Frames format¶

One row per frame—all instances multiplexed across columns.

df = labels.to_dataframe(format="frames")

With instance_id="index" (default):

   frame_idx  inst0.track  inst0.score  inst0.head.x  inst0.head.y  ...  inst1.track  inst1.head.x  ...
0          0        fly_1         0.91         100.0         200.0  ...        fly_2         300.0  ...

With instance_id="track":

df = labels.to_dataframe(format="frames", instance_id="track")

   frame_idx  fly_1.score  fly_1.head.x  fly_1.head.y  ...  fly_2.score  fly_2.head.x  ...
0          0         0.91         100.0         200.0  ...         0.86         300.0  ...

The instance_id parameter controls column naming:

"index": Use positional prefixes (inst0, inst1, ...)
"track": Use track names as prefixes

Track mode requirements

Using instance_id="track" requires all instances to have tracks assigned. Use untracked="ignore" to skip untracked instances, or untracked="error" (default) to raise an error.

Multi-index format¶

Hierarchical column structure with frame as row index.

df = labels.to_dataframe(format="multi_index")

           inst0                                    inst1
           track track_score score   head           track track_score score   head
                                        x      y                                  x      y
frame_idx
0          fly_1        0.98  0.91  100.0  200.0    fly_2        0.95  0.86  300.0  400.0

This format is compatible with NWB pose estimation conventions and works well with pandas pivot operations.

Decoding with `from_dataframe()`¶

Reconstruct Labels from a DataFrame:

from sleap_io.codecs import from_dataframe

# Decode points format
labels = from_dataframe(df, video=video, skeleton=skeleton, format="points")

All four formats support decoding:

# Decode instances format
labels = from_dataframe(df_instances, video=video, skeleton=skeleton, format="instances")

# Decode frames format
labels = from_dataframe(df_frames, video=video, skeleton=skeleton, format="frames")

# Decode multi_index format
labels = from_dataframe(df_multi, video=video, skeleton=skeleton, format="multi_index")

Skeleton inference

If not provided, the skeleton is inferred from column names (node names extracted from {node}.x pattern).

Common parameters¶

All to_dataframe() calls support these options:

Parameter	Default	Description
`video`	`None`	Filter to specific video
`include_metadata`	`True`	Include track/score columns
`include_score`	`True`	Include confidence scores
`include_user_instances`	`True`	Include user-labeled instances
`include_predicted_instances`	`True`	Include predictions
`video_id`	`"path"`	How to represent videos
`include_video`	`None`	Force video column on/off
`backend`	`"pandas"`	Output type (`"pandas"` or `"polars"`)

Video representation options (video_id):

Value	Column	Example
`"path"`	`video_path`	`"/data/video.mp4"`
`"index"`	`video_idx`	`0`
`"name"`	`video_path`	`"video.mp4"`
`"object"`	`video`	`<Video>`

Working with large datasets¶

For datasets with millions of frames, use streaming conversion to avoid memory issues.

Chunked iteration with `to_dataframe_iter()`¶

from sleap_io.codecs import to_dataframe_iter

# Iterate in chunks of 10,000 rows
for chunk in to_dataframe_iter(labels, format="points", chunk_size=10000):
    # Process each chunk
    process(chunk)

Output:

Chunk 1: 10000 rows
Chunk 2: 10000 rows
Chunk 3: 5000 rows  # Final partial chunk

The chunk_size parameter specifies rows per chunk:

points format: One point per row
instances format: One instance per row
frames / multi_index format: One frame per row

Streaming to disk¶

# Stream to Parquet files
for i, chunk in enumerate(to_dataframe_iter(labels, chunk_size=100000)):
    chunk.to_parquet(f"poses_part{i:04d}.parquet")

Memory-efficient CSV export¶

first_chunk = True
for chunk in to_dataframe_iter(labels, chunk_size=50000):
    chunk.to_csv(
        "poses.csv",
        mode="w" if first_chunk else "a",
        header=first_chunk,
        index=False
    )
    first_chunk = False

Labels wrapper method¶

The iterator is also available on Labels:

for chunk in labels.to_dataframe_iter(chunk_size=10000):
    process(chunk)

Choosing a backend¶

The DataFrame codec supports both pandas and polars backends.

Pandas (default)¶

df = labels.to_dataframe(backend="pandas")
# Returns: pandas.DataFrame

Pandas is the default and works out of the box.

Polars¶

df = labels.to_dataframe(backend="polars")
# Returns: polars.DataFrame

shape: (6, 8)
┌───────────┬─────────┬───────┬───────┬───────┬─────────────┬────────────────┬───────┐
│ frame_idx ┆ node    ┆ x     ┆ y     ┆ track ┆ track_score ┆ instance_score ┆ score │
│ ---       ┆ ---     ┆ ---   ┆ ---   ┆ ---   ┆ ---         ┆ ---            ┆ ---   │
│ i64       ┆ str     ┆ f64   ┆ f64   ┆ str   ┆ f64         ┆ f64            ┆ f64   │
╞═══════════╪═════════╪═══════╪═══════╪═══════╪═════════════╪════════════════╪═══════╡
│ 0         ┆ head    ┆ 100.0 ┆ 200.0 ┆ fly_1 ┆ 0.98        ┆ 0.91           ┆ 0.95  │
│ 0         ┆ thorax  ┆ 150.0 ┆ 250.0 ┆ fly_1 ┆ 0.98        ┆ 0.91           ┆ 0.92  │
│ ...       ┆ ...     ┆ ...   ┆ ...   ┆ ...   ┆ ...         ┆ ...            ┆ ...   │
└───────────┴─────────┴───────┴───────┴───────┴─────────────┴────────────────┴───────┘

Installation

Polars is optional. Install with:

pip install sleap-io[polars]

When to use polars¶

Scenario	Recommendation
Interactive analysis	pandas (broader ecosystem)
Large datasets	polars (faster, lower memory)
Streaming to Parquet	polars (native Arrow support)
Integration with ML libraries	pandas (wider compatibility)

Performance comparison (10,000 frames, 3 tracks, 10 nodes):

Format	Native Polars	Pandas→Polars	Speedup
points	504ms	544ms	1.08x
instances	387ms	438ms	1.13x
frames	436ms	477ms	1.09x
multi_index	426ms	550ms	1.29x

The native polars backend constructs DataFrames directly without pandas conversion overhead.

Common patterns¶

Export to CSV¶

# Simple export
df = labels.to_dataframe(format="points")
df.to_csv("poses.csv", index=False)

# With specific columns
df = labels.to_dataframe(format="instances", include_score=False)
df.to_csv("poses_no_scores.csv", index=False)

Export to Parquet¶

# Pandas
df = labels.to_dataframe()
df.to_parquet("poses.parquet")

# Polars (more efficient for large files)
df = labels.to_dataframe(backend="polars")
df.write_parquet("poses.parquet")

Filter before conversion¶

# Only predicted instances from first video
df = labels.to_dataframe(
    video=0,
    include_user_instances=False,
    include_predicted_instances=True
)

Analyze trajectories¶

df = labels.to_dataframe(format="points")

# Mean position per track
df.groupby("track")[["x", "y"]].mean()

# Velocity calculation
df_sorted = df.sort_values(["track", "node", "frame_idx"])
df_sorted["vx"] = df_sorted.groupby(["track", "node"])["x"].diff()
df_sorted["vy"] = df_sorted.groupby(["track", "node"])["y"].diff()

Round-trip editing¶

# Convert to DataFrame
df = labels.to_dataframe(format="points")

# Edit (e.g., apply calibration)
df["x"] = df["x"] * scale_x + offset_x
df["y"] = df["y"] * scale_y + offset_y

# Convert back
labels_calibrated = from_dataframe(
    df,
    video=labels.videos[0],
    skeleton=labels.skeletons[0],
    format="points"
)

Troubleshooting¶

Polars not installed¶

ImportError: Polars backend requires polars to be installed

Solution: Install polars with pip install sleap-io[polars] or pip install polars.

Untracked instances in track mode¶

ValueError: Instance has no track assigned (use untracked='ignore' to skip)

Solution: Either assign tracks to all instances, or use untracked="ignore":

df = labels.to_dataframe(format="frames", instance_id="track", untracked="ignore")

Missing video/skeleton in decoding¶

ValueError: video is required when DataFrame does not contain video column

Solution: Provide the required context:

labels = from_dataframe(df, video=video, skeleton=skeleton)

Array dimension mismatch in `from_numpy()`¶

ValueError: tracks_array must be 4-dimensional (frames, tracks, nodes, coords)

Solution: Ensure your array has shape (n_frames, n_tracks, n_nodes, 2) or (n_frames, n_tracks, n_nodes, 3) if including confidence scores.