# Data formats

sleap-io provides a unified interface for reading and writing pose tracking data across multiple formats. The library automatically detects file formats and provides harmonized I/O operations.

## Universal I/O Functions

::: sleap_io.io.main.load_file

::: sleap_io.io.main.save_file

## Video I/O

::: sleap_io.io.main.load_video

::: sleap_io.io.main.save_video

Media videos can also be read from `http`/`https` URLs with
[`load_video`][sleap_io.load_video] (requires the `pyav` extra; cloud schemes
and Google Drive are not supported for video). See
[Remote video](../remote.md#remote-video).

### Norpix .seq Format

The `.seq` format is used by StreamPix / Norpix for high-speed video recording, commonly used in behavioral neuroscience. sleap-io provides native read support for `.seq` files via the [`SeqVideo`][sleap_io.SeqVideo] backend.

```python
video = sio.load_video("recording.seq")
frame = video[0]  # Read first frame
```

Supported codecs: uncompressed grayscale (`monoraw`), uncompressed BGR (`raw`), JPEG compressed (`monojpg`, `jpg`), and PNG compressed (`monopng`, `png`). Per-frame timestamps are accessible via the backend:

```python
backend = video.backend  # SeqVideo instance
ts = backend.get_timestamp(0)    # Timestamp of first frame (seconds since epoch)
ts_all = backend.get_timestamps()  # All timestamps as numpy array
```

To convert `.seq` to MP4:

```bash
sio reencode recording.seq -o recording.mp4
```

## Format-Specific Functions

### SLEAP Native Format (.slp)

The native SLEAP format stores complete pose tracking projects including videos, skeletons, and annotations. SLP is the primary format with full round-trip support for bounding boxes (format 1.7+), regions of interest (ROIs), and segmentation masks (format 1.5+).

`.slp` and `.pkg.slp` files can also be loaded directly from `http`/`https`, cloud (`s3://`, `gs://`, `az://`), and Google Drive URLs with lazy range-based streaming via [`load_slp`][sleap_io.load_slp] — see [Loading from URLs](../remote.md).

!!! tip "Detailed Format Specification"
    For comprehensive documentation of the SLP file format including HDF5 layout, data structures, and version history, see the **[SLP File Format Reference](slp.md)**.

::: sleap_io.io.main.load_slp

::: sleap_io.io.main.save_slp

#### Lazy Loading for Large Files

When working with large SLP files (hundreds of thousands of frames), loading can be slow due to the creation of many Python objects. sleap-io provides a **lazy loading** mode that defers object creation until needed, significantly speeding up common workflows.

##### When to Use Lazy Loading

Lazy loading is recommended when:

- You only need to convert data to NumPy arrays (`labels.numpy()`)
- You're saving to a different file without modifications
- You're accessing a small subset of frames
- You want fast load times for large files

##### Basic Usage

```python
import sleap_io as sio

# Load lazily (up to 90x faster than eager loading!)
labels = sio.load_slp("predictions.slp", lazy=True)

# Check if labels is lazy
print(labels.is_lazy)  # True

# Fast path: convert directly to NumPy (no object creation)
poses = labels.numpy()

# Fast path: save without materialization
sio.save_slp(labels, "copy.slp")
```

##### Accessing Frames

Lazy-loaded `Labels` support standard read operations:

```python
# These work normally (frames materialized on-demand)
print(len(labels))  # Number of frames
first_frame = labels[0]  # Access single frame
last_frame = labels[-1]  # Negative indexing
subset = labels[10:20]  # Slicing

# Iteration (materializes each frame)
for lf in labels:
    print(f"Frame {lf.frame_idx}: {len(lf)} instances")
```

##### Modifying Lazy Labels

Lazy `Labels` are read-only. To make modifications, first materialize:

```python
# This raises RuntimeError
labels.append(new_frame)  # Error: Cannot append on lazy-loaded Labels

# Materialize first to enable modifications
labels = labels.materialize()  # Creates eager copy
labels.append(new_frame)  # Now works
```

##### Performance Comparison

| Operation | Eager | Lazy | Speedup |
|-----------|-------|------|---------|
| Load only | 0.47s | 0.005s | **~90x** |
| Load + numpy() | 0.86s | 0.38s | **~2x** |
| Full iteration | 0.0002s | 0.41s | Eager faster |

*Benchmarks on 18,000 frames with ~40,000 instances.*

Lazy loading excels at avoiding unnecessary work. If you need to iterate over all frames, eager loading is faster.

##### API Reference

**Labels properties and methods for lazy loading:**

- `Labels.is_lazy` - `True` if lazy-loaded
- `Labels.materialize()` - Convert to eager `Labels` (returns self if already eager)
- `Labels.numpy()` - Uses fast path when lazy (no object creation)
- `Labels.to_dataframe()` - Uses fast path when lazy (no object creation)

**Fast statistics (O(1) for lazy-loaded Labels):**

- `Labels.n_user_instances` - Total number of user-labeled instances
- `Labels.n_pred_instances` - Total number of predicted instances
- `Labels.n_frames_per_video()` - Dictionary mapping videos to frame counts
- `Labels.n_instances_per_track()` - Dictionary mapping tracks to instance counts

### NWB Format (.nwb)

[Neurodata Without Borders (NWB)](https://www.nwb.org/) is a standardized
neurophysiology format with full read/write support for pose tracking. See
the **[NWB Format](nwb.md)** page for harmonized I/O, the `nwb_format` types,
the advanced annotations API, multi-subject export, and metadata handling.

### JABS Format (.h5)

[JABS](https://github.com/KumarLabJax/JABS-behavior-classifier) (JAX Animal Behavior System) format for behavior classification.

::: sleap_io.io.main.load_jabs

::: sleap_io.io.main.save_jabs

### SLEAP Analysis HDF5 Format (.h5)

The SLEAP Analysis HDF5 format is a portable format for exporting pose tracking predictions as dense numpy arrays. This is the format produced by SLEAP's "Export Analysis HDF5" feature, designed for easy loading in MATLAB and Python analysis pipelines.

::: sleap_io.io.main.load_analysis_h5

::: sleap_io.io.main.save_analysis_h5

#### Axis Ordering Presets

The format supports configurable axis ordering via presets:

| Preset | Description | tracks shape |
|--------|-------------|--------------|
| `matlab` (default) | SLEAP-compatible, optimized for MATLAB | `(tracks, 2, nodes, frames)` |
| `standard` | Python-native, intuitive indexing | `(frames, tracks, nodes, 2)` |

```python
import sleap_io as sio

labels = sio.load_slp("predictions.slp")

# Default (MATLAB-compatible) - matches SLEAP's export
sio.save_analysis_h5(labels, "output.h5")

# Python-native ordering for easier numpy indexing
sio.save_analysis_h5(labels, "output.h5", preset="standard")

# Filter tracks with <50% occupancy
sio.save_analysis_h5(labels, "output.h5", min_occupancy=0.5)

# Load back
loaded = sio.load_analysis_h5("output.h5")
```

#### Self-Documenting Format

Each dataset stores its dimension names in the `dims` HDF5 attribute, making files self-documenting:

```python
import h5py

with h5py.File("output.h5", "r") as f:
    print(f["tracks"].attrs["dims"])  # e.g., '["track", "xy", "node", "frame"]'
    print(f.attrs["preset"])  # "matlab", "standard", or "custom"
```

### Label Studio Format (.json)

[Label Studio](https://labelstud.io/) is a multi-modal annotation platform. Export annotations from Label Studio and load them into SLEAP.

::: sleap_io.io.main.load_labelstudio

::: sleap_io.io.main.save_labelstudio

### DeepLabCut Format (.csv)

Load annotations from [DeepLabCut](http://www.mackenziemathislab.org/deeplabcut). See the
**[DeepLabCut Format](dlc.md)** page for single-CSV loading with `config.yaml`
discovery (skeleton edges + source-video links), whole-project imports with
`load_dlc_project`, and train/test split recovery with `load_dlc_splits`.

### CSV Format (.csv)

sleap-io provides comprehensive CSV support for reading and writing pose tracking data, enabling interoperability with spreadsheet tools, custom pipelines, and other pose estimation frameworks.

#### Supported CSV Formats

| Format | Description | Use Case |
|--------|-------------|----------|
| `sleap` | SLEAP Analysis CSV (default) | Native SLEAP exports, one row per instance |
| `dlc` | DeepLabCut format | DLC compatibility, multi-header structure |
| `points` | One row per point | Most normalized, database-friendly |
| `instances` | One row per instance | Compact, analysis-friendly |
| `frames` | One row per frame | Wide format, all instances in columns |

#### Basic Usage

```python
import sleap_io as sio

# Load CSV (auto-detects format)
labels = sio.load_csv("predictions.csv")

# Save in SLEAP Analysis format (default)
sio.save_csv(labels, "output.csv")

# Save in DLC format
sio.save_csv(labels, "dlc_output.csv", format="dlc", scorer="MyModel")

# Save with metadata for full round-trip support
sio.save_csv(labels, "output.csv", save_metadata=True)
# Creates: output.csv + output.json (metadata)
```

#### Round-Trip with Metadata

CSV files cannot store all Labels information (skeleton edges, symmetries, suggestions). To enable full round-trip reconstruction, use `save_metadata=True`:

```python
# Save with metadata sidecar file
sio.save_csv(labels, "data.csv", save_metadata=True)
# Creates: data.csv and data.json

# Load back with full metadata
labels = sio.load_csv("data.csv")
# Automatically loads data.json if present
```

The metadata JSON file contains:

- Video paths and backend metadata
- Skeleton definitions (nodes, edges, symmetries)
- Track names
- Suggested frames
- Provenance information

#### Format-Specific Examples

##### SLEAP Analysis Format

The default format matches SLEAP's "Export Analysis CSV" output:

```python
# Write SLEAP format
sio.save_csv(labels, "analysis.csv", format="sleap")
```

Output columns: `track, frame_idx, instance.score, {node}.x, {node}.y, {node}.score, ...`

##### DeepLabCut Format

For compatibility with DeepLabCut workflows:

```python
# Write DLC format with custom scorer name
sio.save_csv(labels, "dlc_output.csv", format="dlc", scorer="MyNetwork")

# Multi-animal DLC format (auto-detected from tracks)
sio.save_csv(multi_animal_labels, "multi_dlc.csv", format="dlc")
```

DLC format uses multi-row headers (scorer, bodyparts, coords) and is compatible with DLC's analysis tools.

##### DataFrame Codec Formats

For custom analysis pipelines, use the normalized formats from the DataFrame codec:

```python
# Points format: most normalized (one row per point)
sio.save_csv(labels, "points.csv", format="points")

# Instances format: one row per instance
sio.save_csv(labels, "instances.csv", format="instances")

# Frames format: one row per frame (wide format)
sio.save_csv(labels, "frames.csv", format="frames")
```

::: sleap_io.io.main.load_csv

::: sleap_io.io.main.save_csv

### AlphaTracker Format

Load predictions from [AlphaTracker](https://github.com/yinaanyachukwu/AlphaTracker), a tracking system for socially-housed animals.

::: sleap_io.io.main.load_alphatracker

### LEAP Format (.mat)

Load predictions from [LEAP](https://github.com/talmo/leap), a SLEAP predecessor. Requires `scipy` for .mat file support.

::: sleap_io.io.main.load_leap

### COCO Format (.json)

[COCO](https://cocodataset.org/) (Common Objects in Context) is widely used in
computer vision and pose estimation. See the **[COCO Format](coco.md)** page
for full read/write support (compatible with mmpose, CVAT, and other COCO-based
tools) and empty-frame handling.

### TIFF Label Images (.tif, .tiff)

TIFF files store dense integer label images for instance segmentation, where each pixel value encodes which object occupies that location. This is the standard output of tools like [Cellpose](https://www.cellpose.org/) and [StarDist](https://github.com/stardist/stardist).

sleap-io supports three TIFF layouts: single files, multi-page stacks, and directories of per-frame TIFFs. A JSON sidecar (`.meta.json`) is written alongside the TIFF to preserve track names and categories.

!!! tip "Detailed Format Documentation"
    For comprehensive documentation of TIFF label image I/O including file structures and sidecar metadata, see the **[TIFF Format Reference](tiff.md)**.

::: sleap_io.io.main.load_label_images

::: sleap_io.io.main.save_label_images

### COCO Panoptic Segmentation

[COCO panoptic](https://cocodataset.org/#panoptic-2018) format represents per-pixel segmentation using a JSON annotation file and per-frame PNG label images. Each pixel is encoded as `R + G * 256 + B * 256^2`. "Thing" segments (countable objects) get track identities; "stuff" segments (uncountable regions) do not.

```python
from sleap_io.io.coco import read_coco_panoptic, write_coco_panoptic

# Read panoptic annotations
labels = read_coco_panoptic("panoptic.json", images_dir="panoptic_pngs/")

# Write panoptic annotations
write_coco_panoptic("output.json", labels, images_dir="output_pngs/")
```

::: sleap_io.io.coco.read_coco_panoptic

::: sleap_io.io.coco.write_coco_panoptic

### Ultralytics YOLO Format

Read and write the [Ultralytics YOLO](https://docs.ultralytics.com/) pose
format. See the **[Ultralytics YOLO Format](ultralytics.md)** page for details.

### GeoJSON Format (.geojson)

[GeoJSON](https://geojson.org/) (RFC 7946) stores ROIs as a human-readable,
standalone format that interoperates with the
[movement](https://github.com/neuroinformatics-unit/movement) library and the
geospatial Python ecosystem (Shapely, GeoPandas, QGIS). See the
**[GeoJSON Format](geojson.md)** page for the schema and examples.

## Working with Multiple Datasets

### Load Multiple Files

Load and combine multiple pose tracking files:

::: sleap_io.io.main.load_labels_set

## Skeleton Files

Load and save skeleton definitions separately:

::: sleap_io.io.main.load_skeleton

::: sleap_io.io.main.save_skeleton

## Format Detection

sleap-io automatically detects file formats based on:

1. **File extension**: `.slp`, `.nwb`, `.h5`, `.json`, `.geojson`, `.mat`, `.csv`, `.tif`, `.tiff`
2. **File content**: For ambiguous extensions like `.h5` (JABS vs Analysis HDF5) or `.json` (Label Studio vs COCO)
3. **Explicit format**: Pass `format` parameter to override auto-detection

For URLs, ambiguous extensions (`.h5`, `.json`, `.csv`) are disambiguated with a magic-byte sniff via a Range request, controllable with [`load_file`][sleap_io.load_file]'s `sniff=` argument. See [Loading from URLs](../remote.md).

## Format Conversion Examples

### Convert Between Formats

```python
import sleap_io as sio

# Load from any supported format
labels = sio.load_file("data.slp")

# Save to different formats
labels.save("data.nwb")  # NWB format
labels.save("data.labelstudio.json")  # Label Studio
labels.save("data_yolo/")  # Ultralytics YOLO
```

### Batch Conversion

```python
import sleap_io as sio
from pathlib import Path

# Convert all SLEAP files to NWB
for slp_file in Path("data/").glob("*.slp"):
    labels = sio.load_file(slp_file)
    nwb_file = slp_file.with_suffix(".nwb")
    labels.save(nwb_file)
```

### Round-Trip Preservation

Most formats preserve data during round-trip conversion:

```python
import sleap_io as sio

# Load original
labels_original = sio.load_file("data.slp")

# Save and reload
labels_original.save("temp.nwb")
labels_reloaded = sio.load_file("temp.nwb")

# Data is preserved
assert len(labels_original) == len(labels_reloaded)
assert labels_original.skeleton == labels_reloaded.skeleton
```

## Format Limitations

Different formats have varying capabilities:

| Format | Read | Write | Videos | Skeletons | Tracks | Confidence | User/Predicted | BBoxes | ROIs/Masks | Label Images |
|--------|------|-------|--------|-----------|--------|------------|----------------|--------|------------|--------------|
| SLEAP (.slp) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| NWB (.nwb) | ✅ | ✅ | ✅* | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| JABS (.h5) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Analysis HDF5 | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Label Studio | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| CSV (.csv) | ✅ | ✅ | ❌ | ✅** | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| DeepLabCut | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| [TrackMate](trackmate.md) (.csv) | ✅ | ❌ | ✅******* | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| AlphaTracker | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| LEAP (.mat) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| COCO (.json) | ✅ | ✅ | ❌ | ✅ | ✅*** | ❌ | ✅ | ✅ | ✅ | ❌ |
| COCO Panoptic | ✅ | ✅ | ❌ | ❌ | ✅**** | ❌ | ❌ | ❌ | ❌ | ✅ |
| TIFF (.tif) | ✅ | ✅ | ❌ | ❌ | ✅***** | ❌ | ❌ | ❌ | ❌ | ✅ |
| Ultralytics | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅****** | ❌ |
| GeoJSON (.geojson) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |

*NWB can embed videos with `annotations_export` format
**CSV skeleton edges/symmetries preserved via optional metadata JSON sidecar
***COCO tracks are stored via `attributes.object_id` (CVAT-compatible)
****COCO panoptic tracks for "thing" segments only
*****TIFF tracks via `.meta.json` sidecar
******Ultralytics segmentation polygons stored as ROIs
*******TrackMate auto-detects sibling `.tif`/`.tiff` video files

!!! note "Remote URL loading"
    Loading from a URL is currently supported only for SLEAP `.slp`/`.pkg.slp` (labels) and `http`/`https` media video; all other labels formats raise `NotImplementedError` over a URL — download the file locally first. See [Loading from URLs](../remote.md).

## See Also

- [Data Model](../model/index.md): Understanding the core data structures
- [Examples](../examples.md): More usage examples and recipes
- [Merging](../merging.md): Combining data from multiple sources