COCO Format (.json)¶

COCO (Common Objects in Context) format is widely used in computer vision and pose estimation. sleap-io provides full read and write support, making it compatible with tools like mmpose, CVAT, and other COCO-compatible frameworks.

sleap-io reads all three COCO flavors:

Pose datasets (categories with keypoints) → Instance objects.
Detection datasets (annotations with bbox) → BoundingBox objects.
Instance-segmentation datasets (annotations with segmentation) → SegmentationMask (or ROI) objects.

These are not mutually exclusive: an annotation that carries both keypoints and a segmentation/bbox preserves all of them, with the segmentation/bbox linked back to the keypoint Instance.

Unannotated images become empty frames

load_coco creates a LabeledFrame for every entry in the images array, including images with zero annotations — these become empty LabeledFrames, so the frame count matches the input and unannotated images round-trip losslessly. One caveat: an image whose file path cannot be resolved is skipped, so a 0-annotation image with a missing file is dropped rather than preserved.

Segmentation handling¶

COCO encodes segmentation either as a polygon (a list of [x1, y1, x2, y2, ...] rings) or as RLE (a {"counts": ..., "size": ...} dict). RLE is always read as a SegmentationMask. Polygon handling is controlled by the segmentation_format argument of load_coco / coco.read_labels:

"mask" (the default): each annotation's polygon(s) are rasterized into a single SegmentationMask at the image resolution. Multiple rings of one annotation collapse into one object mask. This is the representation that exercises the segmentation data model and round-trips through .slp.
"roi": keep the native vector geometry as ROI objects (one per ring), without rasterizing.

Breaking change in 0.8.0

The default polygon segmentation handling changed in v0.8.0. In v0.7.1, polygon segmentation was read as vector ROI objects (one per ring). The new default (segmentation_format="mask") rasterizes each annotation's polygon(s) into a single SegmentationMask. To restore the pre-0.8.0 vector-ROI behavior, pass segmentation_format="roi".

import sleap_io as sio

# Polygon segmentation -> SegmentationMask (default).
labels = sio.load_file("annotations.coco.json")
masks = labels.labeled_frames[0].masks

# Keep polygons as vector ROIs instead.
labels = sio.load_coco("annotations.coco.json", segmentation_format="roi")
rois = labels.labeled_frames[0].rois

Mask rasterization needs image dimensions

Rasterizing a polygon requires the image height/width from the images entry. In "mask" mode, a polygon whose image lacks those fields falls back to an ROI since there is no extent to rasterize into. When written back to COCO, SegmentationMask objects are exported as RLE.

Predicted vs. user segmentation

A detection annotation carrying a score (i.e. a model prediction) is read as a PredictedSegmentationMask / PredictedROI with that score; annotations without a score become the User* variants. This mirrors how bbox annotations select PredictedBoundingBox vs. UserBoundingBox.

Categories as identities¶

In a standard COCO dataset the category is an object class (e.g. "person", "car"). Some datasets instead use the category to encode a persistent identity — for example a multi-animal segmentation dataset where each animal is its own category. Set category_as_track=True to map each category to a shared Track (named after the category) and assign it to every annotation of that category (masks, ROIs, bounding boxes, and keypoint instances without an explicit track id):

labels = sio.load_coco("annotations.coco.json", category_as_track=True)
[t.name for t in labels.tracks]      # one Track per category
labels.labeled_frames[0].masks[0].track.name  # == that mask's category

The identity tracks are persisted to .slp and survive a save/load round-trip.

`sleap_io.io.main.load_coco(json_path, dataset_root=None, grayscale=False, segmentation_format='mask', category_as_track=False, **kwargs)` ¶

Load a COCO-style dataset and return a Labels object.

Supports pose (keypoint), detection (bbox), and instance-segmentation (polygon or RLE) COCO datasets.

Parameters:

Name	Type	Description	Default
`json_path`	`str`	Path to the COCO annotation JSON file.	required
`dataset_root`	`str \| None`	Root directory of the dataset. If None, uses parent directory of json_path.	`None`
`grayscale`	`bool`	If True, load images as grayscale (1 channel). If False, load as RGB (3 channels). Default is False.	`False`
`segmentation_format`	`str`	How to represent polygon segmentation. `"mask"` (the default) rasterizes polygons into `SegmentationMask` objects; `"roi"` keeps them as vector `ROI` objects. RLE segmentation is always read as a `SegmentationMask`.	`'mask'`
`category_as_track`	`bool`	If True, treat each COCO category as a persistent identity, creating one `Track` per category and assigning it to that category's annotations. Useful for instance-segmentation datasets where the category encodes identity. Default is False.	`False`
`**kwargs`		Additional arguments (currently unused).	required

Returns:

Type	Description
`Labels`	The dataset as a `Labels` object.

Source code in sleap_io/io/main.py

def load_coco(
    json_path: str,
    dataset_root: str | None = None,
    grayscale: bool = False,
    segmentation_format: str = "mask",
    category_as_track: bool = False,
    **kwargs,
) -> Labels:
    """Load a COCO-style dataset and return a Labels object.

    Supports pose (keypoint), detection (bbox), and instance-segmentation
    (polygon or RLE) COCO datasets.

    Args:
        json_path: Path to the COCO annotation JSON file.
        dataset_root: Root directory of the dataset. If None, uses parent directory
                     of json_path.
        grayscale: If True, load images as grayscale (1 channel). If False, load as
                   RGB (3 channels). Default is False.
        segmentation_format: How to represent polygon segmentation. ``"mask"`` (the
            default) rasterizes polygons into `SegmentationMask` objects; ``"roi"``
            keeps them as vector `ROI` objects. RLE segmentation is always read as a
            `SegmentationMask`.
        category_as_track: If True, treat each COCO category as a persistent
            identity, creating one `Track` per category and assigning it to that
            category's annotations. Useful for instance-segmentation datasets
            where the category encodes identity. Default is False.
        **kwargs: Additional arguments (currently unused).

    Returns:
        The dataset as a `Labels` object.
    """
    from sleap_io.io import coco

    return coco.read_labels(
        json_path,
        dataset_root=dataset_root,
        grayscale=grayscale,
        segmentation_format=segmentation_format,
        category_as_track=category_as_track,
    )

`sleap_io.io.main.save_coco(labels, json_path, image_filenames=None, visibility_encoding='ternary')` ¶

Save a SLEAP dataset to COCO-style JSON annotation format.

Parameters:

Name	Type	Description	Default
`labels`	`Labels`	A SLEAP `Labels` object.	required
`json_path`	`str`	Path to save the COCO annotation JSON file.	required
`image_filenames`	`str \| list[str] \| None`	Optional image filenames to use in the COCO JSON. If provided, must be a single string (for single-frame videos) or a list of strings matching the number of labeled frames. If None, generates filenames from video filenames and frame indices.	`None`
`visibility_encoding`	`str`	Visibility encoding to use. Either "binary" (0/1) or "ternary" (0/½). Default is "ternary".	`'ternary'`

Notes

This function only writes the JSON annotation file. It does not save images.
The generated JSON can be used with mmpose and other COCO-compatible tools.
For saving images along with annotations, you would need to extract and save frames separately.

Source code in sleap_io/io/main.py

def save_coco(
    labels: Labels,
    json_path: str,
    image_filenames: str | list[str] | None = None,
    visibility_encoding: str = "ternary",
):
    """Save a SLEAP dataset to COCO-style JSON annotation format.

    Args:
        labels: A SLEAP `Labels` object.
        json_path: Path to save the COCO annotation JSON file.
        image_filenames: Optional image filenames to use in the COCO JSON. If
                        provided, must be a single string (for single-frame videos) or
                        a list of strings matching the number of labeled frames. If
                        None, generates filenames from video filenames and frame
                        indices.
        visibility_encoding: Visibility encoding to use. Either "binary" (0/1) or
                           "ternary" (0/1/2). Default is "ternary".

    Notes:
        - This function only writes the JSON annotation file. It does not save images.
        - The generated JSON can be used with mmpose and other COCO-compatible tools.
        - For saving images along with annotations, you would need to extract and save
          frames separately.
    """
    from sleap_io.io import coco

    coco.write_labels(labels, json_path, image_filenames, visibility_encoding)

COCO Format (.json)¶

Segmentation handling¶

Categories as identities¶

sleap_io.io.main.load_coco(json_path, dataset_root=None, grayscale=False, segmentation_format='mask', category_as_track=False, **kwargs) ¶

sleap_io.io.main.save_coco(labels, json_path, image_filenames=None, visibility_encoding='ternary') ¶

`sleap_io.io.main.load_coco(json_path, dataset_root=None, grayscale=False, segmentation_format='mask', category_as_track=False, **kwargs)` ¶

`sleap_io.io.main.save_coco(labels, json_path, image_filenames=None, visibility_encoding='ternary')` ¶