labeled_frame
sleap_io.model.labeled_frame
¶
Data structures for data contained within a single video frame.
The LabeledFrame class is a data structure that contains Instances and
PredictedInstances that are associated with a single frame within a video.
Classes:
| Name | Description |
|---|---|
Instance |
This class represents a ground truth instance such as an animal. |
LabeledFrame |
Labeled data for a single frame of a video. |
PredictedInstance |
A |
Video |
|
Attributes:
| Name | Type | Description |
|---|---|---|
TYPE_CHECKING |
Returns True when the argument is true, False otherwise. |
|
__cached__ |
str(object='') -> str |
|
__doc__ |
str(object='') -> str |
|
__file__ |
str(object='') -> str |
|
__name__ |
str(object='') -> str |
|
__package__ |
str(object='') -> str |
TYPE_CHECKING = False
module-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__cached__ = '/home/runner/work/sleap-io/sleap-io/sleap_io/model/__pycache__/labeled_frame.cpython-313.pyc'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__doc__ = 'Data structures for data contained within a single video frame.\n\nThe `LabeledFrame` class is a data structure that contains `Instance`s and\n`PredictedInstance`s that are associated with a single frame within a video.\n'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__file__ = '/home/runner/work/sleap-io/sleap-io/sleap_io/model/labeled_frame.py'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__name__ = 'sleap_io.model.labeled_frame'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__package__ = 'sleap_io.model'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
Instance
¶
This class represents a ground truth instance such as an animal.
An Instance has a set of landmarks (points) that correspond to a Skeleton. Each
point is associated with a Node in the skeleton. The points are stored in a
structured numpy array with columns for x, y, visible, complete and name.
The Instance may also be associated with a Track which links multiple instances
together across frames or videos.
Attributes:
| Name | Type | Description |
|---|---|---|
points |
A numpy structured array with columns for xy, visible and complete. The
array should have shape |
|
skeleton |
The |
|
track |
An optional |
|
tracking_score |
The score associated with the |
|
identity |
An optional |
|
identity_score |
The score associated with the |
|
from_predicted |
The |
|
identity_embedding |
An optional |
|
category |
An optional |
|
category_score |
The score associated with the |
|
category_embedding |
An optional |
Methods:
| Name | Description |
|---|---|
__attrs_post_init__ |
Convert the points array after initialization. |
__getitem__ |
Return the point associated with a node. |
__init__ |
Method generated by attrs for class Instance. |
__len__ |
Return the number of points in the instance. |
__repr__ |
Return a readable representation of the instance. |
__setattr__ |
Method generated by attrs for class Instance. |
__setitem__ |
Set the point associated with a node. |
bounding_box |
Get the bounding box of visible points. |
empty |
Create an empty instance with no points. |
from_numpy |
Create an instance object from a numpy array. |
numpy |
Return the instance points as a |
overlaps_with |
Check if this instance overlaps with another based on bounding box IoU. |
replace_skeleton |
Replace the skeleton associated with the instance. |
same_identity_as |
Check if this instance has the same identity as another instance. |
same_pose_as |
Check if this instance has the same pose as another instance. |
to_bbox |
Create a bounding box from this instance. |
to_centroid |
Create a |
to_mask |
Rasterize this instance's ROI geometry into a segmentation mask. |
to_roi |
Create a region-of-interest geometry from this instance. |
update_skeleton |
Update or replace the skeleton associated with the instance. |
Source code in sleap_io/model/instance.py
@attrs.define(auto_attribs=True, slots=True, eq=False)
class Instance:
"""This class represents a ground truth instance such as an animal.
An `Instance` has a set of landmarks (points) that correspond to a `Skeleton`. Each
point is associated with a `Node` in the skeleton. The points are stored in a
structured numpy array with columns for x, y, visible, complete and name.
The `Instance` may also be associated with a `Track` which links multiple instances
together across frames or videos.
Attributes:
points: A numpy structured array with columns for xy, visible and complete. The
array should have shape `(n_nodes,)`. This representation is useful for
performance efficiency when working with large datasets.
skeleton: The `Skeleton` that describes the `Node`s and `Edge`s associated with
this instance.
track: An optional `Track` associated with a unique animal/object across frames
or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity assignment.
This is `None` if the instance is not associated with a track or if the
track was assigned manually.
identity: An optional `Identity` representing the global, ground-truth animal
this instance belongs to (persistent across videos/sessions). Unlike
`track` (an ephemeral, video-local tracklet), `Identity` is the cross-file
re-identification key. `None` if no global identity is assigned.
identity_score: The score associated with the `identity` assignment (e.g. the
cosine similarity to a re-ID gallery prototype). This is `None` if the
instance has no identity or the identity was assigned manually. Kept
separate from `tracking_score` (short-term tracklet vs long-term identity).
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
identity_embedding: An optional `Embedding` describing this instance's
appearance for re-identification (e.g. a vector produced by a re-ID
model). ``None`` by default.
category: An optional `Category` representing the *class* this instance
belongs to (e.g. ``"female_fly"``, ``"fur_shaved"``), typically
assigned by classification or re-ID. Mirrors `identity` but groups by
class rather than individual. `None` if no category is assigned.
category_score: The score associated with the `category` assignment (e.g.
the classifier confidence). `None` if the instance has no category or
the category was assigned manually.
category_embedding: An optional `Embedding` describing this instance's
appearance for classification (the vector the `category` was
classified from). ``None`` by default.
"""
points: PointsArray = attrs.field(eq=attrs.cmp_using(eq=np.array_equal))
skeleton: Skeleton
track: Track | None = None
tracking_score: float | None = None
identity: Identity | None = None
identity_score: float | None = None
category: Category | None = attrs.field(default=None, converter=to_category)
category_score: float | None = None
from_predicted: "PredictedInstance | None" = None
identity_embedding: Embedding | None = attrs.field(default=None, repr=False)
category_embedding: Embedding | None = attrs.field(default=None, repr=False)
@classmethod
def empty(
cls,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an empty instance with no points.
Args:
skeleton: The `Skeleton` that this `Instance` is associated with.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` with an empty numpy array of shape `(n_nodes,)`.
"""
points = PointsArray.empty(len(skeleton))
points["name"] = skeleton.node_names
return cls(
points=points,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
@classmethod
def _convert_points(
cls, points_data: np.ndarray | dict | list, skeleton: Skeleton
) -> PointsArray:
"""Convert points to a structured numpy array if needed."""
if isinstance(points_data, dict):
return PointsArray.from_dict(points_data, skeleton)
elif isinstance(points_data, (list, np.ndarray)):
if isinstance(points_data, list):
points_data = np.array(points_data)
points = PointsArray.from_array(points_data)
points["name"] = skeleton.node_names
return points
else:
raise ValueError("points must be a numpy array or dictionary.")
@classmethod
def from_numpy(
cls,
points_data: np.ndarray,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an instance object from a numpy array.
Args:
points_data: A numpy array of shape `(n_nodes, D)` corresponding to the
points of the skeleton. Values of `np.nan` indicate "missing" nodes and
will be reflected in the "visible" field.
If `D == 2`, the array should have columns for x and y.
If `D == 3`, the array should have columns for x, y and visible.
If `D == 4`, the array should have columns for x, y, visible and
complete.
If this is provided as a structured array, it will be used without copy
if it has the correct dtype. Otherwise, a new structured array will be
created reusing the provided data.
skeleton: The `Skeleton` that this `Instance` is associated with. It should
have `n_nodes` nodes.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` object with the specified points.
"""
return cls(
points=points_data,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
def __attrs_post_init__(self):
"""Convert the points array after initialization."""
if not isinstance(self.points, PointsArray):
self.points = self._convert_points(self.points, self.skeleton)
# Ensure points have node names
if "name" in self.points.dtype.names and not all(self.points["name"]):
self.points["name"] = self.skeleton.node_names
def numpy(
self,
invisible_as_nan: bool = True,
) -> np.ndarray:
"""Return the instance points as a `(n_nodes, 2)` numpy array.
Args:
invisible_as_nan: If `True` (the default), points that are not visible will
be set to `np.nan`. If `False`, they will be whatever the stored value
of `Instance.points["xy"]` is.
Returns:
A numpy array of shape `(n_nodes, 2)` corresponding to the points of the
skeleton. Values of `np.nan` indicate "missing" nodes.
Notes:
This will always return a copy of the array.
If you need to avoid making a copy, just access the `Instance.points["xy"]`
attribute directly. This will not replace invisible points with `np.nan`.
"""
if invisible_as_nan:
return np.where(
self.points["visible"].reshape(-1, 1), self.points["xy"], np.nan
)
else:
return self.points["xy"].copy()
@property
def centroid_xy(self) -> tuple[float, float] | None:
"""Mean of visible point coordinates as ``(x, y)``, or ``None``.
Returns:
A tuple ``(x, y)`` representing the center of mass of all visible
points, or ``None`` if no points are visible.
"""
pts = self.numpy(invisible_as_nan=True)
visible = ~np.isnan(pts[:, 0])
if not visible.any():
return None
return float(pts[visible, 0].mean()), float(pts[visible, 1].mean())
def to_centroid(
self,
method: str = "center_of_mass",
node: int | str | None = None,
fallback: str | None = None,
error_on_empty: bool = False,
**kwargs,
) -> "Centroid":
"""Create a ``Centroid`` from this instance.
Delegates to ``Centroid.from_pose()``. A ``PredictedInstance`` yields a
``PredictedCentroid`` carrying its ``score``; any other instance yields a
``UserCentroid``. Metadata (``track``, ``tracking_score``, ``identity``,
``identity_score``, ``identity_embedding``, ``category``,
``category_score``, ``category_embedding``, ``instance=self``) is
propagated.
Args:
method: Computation method (``"center_of_mass"``, ``"bbox_center"``,
``"geometric_median"``, or ``"anchor"``).
node: Node specification for the ``"anchor"`` method. Can be a node
name (str) or index (int).
fallback: For the ``"anchor"`` method, a non-anchor method to fall
back to when the anchor node is occluded.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) centroid.
**kwargs: Additional keyword arguments passed to the centroid
constructor.
Returns:
A ``UserCentroid`` or ``PredictedCentroid`` depending on the
instance type.
Raises:
ValueError: For an unknown ``method``, a missing ``node`` for the
``"anchor"`` method, an invalid ``node`` type, or (when
``error_on_empty`` is ``True``) when there are no visible points.
"""
from sleap_io.model.centroid import Centroid
return Centroid.from_pose(
self,
method=method,
node=node,
fallback=fallback,
error_on_empty=error_on_empty,
**kwargs,
)
def to_bbox(
self,
mode: str = "tight",
size: float | tuple[float, float] | None = None,
padding: float | tuple[float, float] = 0.0,
node: int | str | None = None,
center_method: str = "center_of_mass",
rotated: bool = False,
error_on_empty: bool = False,
) -> "BoundingBox":
"""Create a bounding box from this instance.
A ``PredictedInstance`` yields a ``PredictedBoundingBox`` carrying its
``score``; any other instance yields a ``UserBoundingBox``. Metadata
(``track``, ``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
mode: ``"tight"`` to fit the visible points, or ``"centered"`` to
build a fixed-``size`` box centered on a computed centroid.
size: Box size for ``mode="centered"``. A scalar yields a square box;
a ``(w, h)`` tuple sets width and height independently. Required
for ``mode="centered"``.
padding: Amount to inflate the box outward. Scalar applies to both
axes; a ``(px, py)`` tuple applies per-axis. Negative values
shrink the box.
node: Node specification passed to the centroid computation for
``mode="centered"`` with ``center_method="anchor"``.
center_method: Centroid method used to locate the box center for
``mode="centered"`` (see :meth:`to_centroid`).
rotated: For ``mode="tight"``, if ``True`` fit a minimum-area oriented
box from the convex hull of visible points; otherwise fit an
axis-aligned box.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) box.
Returns:
A ``BoundingBox`` enclosing the instance (or NaN corners if empty).
Raises:
ValueError: For an unknown ``mode``, a missing ``size`` for
``mode="centered"``, or (when ``error_on_empty`` is ``True``)
when there are no visible points.
"""
from sleap_io.model.bbox import PredictedBoundingBox, UserBoundingBox
from sleap_io.model.roi import (
_apply_padding,
_geometry_to_bbox_coords,
_pose_to_geometry,
)
nan = float("nan")
angle = 0.0
if mode == "tight":
pts = self.numpy(invisible_as_nan=True)
visible = ~np.isnan(pts[:, 0])
if not visible.any():
if error_on_empty:
raise ValueError("No visible points to compute bounding box.")
x1 = y1 = x2 = y2 = nan
elif rotated:
hull = _pose_to_geometry(
pts, self.skeleton.edge_inds, method="convex_hull"
)
x1, y1, x2, y2, angle = _geometry_to_bbox_coords(hull, rotated=True)
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
vis = pts[visible]
x1 = float(vis[:, 0].min())
y1 = float(vis[:, 1].min())
x2 = float(vis[:, 0].max())
y2 = float(vis[:, 1].max())
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
elif mode == "centered":
if size is None:
raise ValueError("'size' is required for mode='centered'.")
centroid = self.to_centroid(
method=center_method, node=node, error_on_empty=error_on_empty
)
if centroid.is_empty:
x1 = y1 = x2 = y2 = nan
else:
cx, cy = centroid.xy
if isinstance(size, (tuple, list)):
w, h = size
else:
w = h = size
x1 = cx - w / 2
y1 = cy - h / 2
x2 = cx + w / 2
y2 = cy + h / 2
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
raise ValueError(f"Unknown mode {mode!r}. Expected 'tight' or 'centered'.")
kwargs = dict(
x1=x1,
y1=y1,
x2=x2,
y2=y2,
angle=angle,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedBoundingBox(score=self.score, **kwargs)
return UserBoundingBox(**kwargs)
def to_roi(
self,
method: str = "shapes",
node_radius: float = 0.0,
edge_radius: float = 0.0,
radius: float = 0.0,
quad_segs: int = 8,
error_on_empty: bool = False,
) -> "ROI":
"""Create a region-of-interest geometry from this instance.
A ``PredictedInstance`` yields a ``PredictedROI`` carrying its ``score``;
any other instance yields a ``UserROI``. Metadata (``track``,
``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
method: ``"shapes"`` to union buffered node points and/or edge
segments, or ``"convex_hull"`` to take the convex hull of the
visible points.
node_radius: Buffer radius around each visible node (``"shapes"``
only).
edge_radius: Buffer radius around each fully-visible edge segment
(``"shapes"`` only).
radius: Optional buffer applied to the convex hull
(``"convex_hull"`` only).
quad_segs: Number of segments used to approximate a quarter circle
when buffering.
error_on_empty: If ``True``, raise ``ValueError`` when the resulting
geometry is empty instead of returning an empty-geometry ROI.
Returns:
A ``ROI`` whose geometry encloses the instance (an empty ``Polygon``
if there are no visible points).
Raises:
ValueError: If ``method="shapes"`` with both ``node_radius`` and
``edge_radius`` equal to 0 (a misconfiguration, always raised),
for an unknown ``method``, or (when ``error_on_empty`` is
``True``) when the resulting geometry is empty.
"""
from sleap_io.model.roi import PredictedROI, UserROI, _pose_to_geometry
# Misconfiguration: raise before the empty-points check so that an empty
# instance still surfaces the error.
if method == "shapes" and node_radius == 0 and edge_radius == 0:
raise ValueError(
"method='shapes' requires at least one of node_radius or "
"edge_radius to be > 0."
)
geom = _pose_to_geometry(
self.numpy(invisible_as_nan=True),
self.skeleton.edge_inds,
method=method,
node_radius=node_radius,
edge_radius=edge_radius,
radius=radius,
quad_segs=quad_segs,
)
if geom.is_empty and error_on_empty:
raise ValueError("No visible points to compute ROI geometry.")
kwargs = dict(
geometry=geom,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedROI(score=self.score, **kwargs)
return UserROI(**kwargs)
def to_mask(self, height: int, width: int, **roi_kwargs) -> "SegmentationMask":
"""Rasterize this instance's ROI geometry into a segmentation mask.
Equivalent to ``self.to_roi(**roi_kwargs).to_mask(height, width)``,
except that a zero-area hull (``method="convex_hull"`` over fewer than
three visible points yields a ``Point`` or ``LineString``) rasterizes to
an all-background mask here instead of raising. A ``PredictedInstance``
yields a ``PredictedSegmentationMask`` carrying its ``score``; any other
instance yields a ``UserSegmentationMask``. Metadata is propagated.
Args:
height: Height of the output mask in pixels.
width: Width of the output mask in pixels.
**roi_kwargs: Keyword arguments forwarded to :meth:`to_roi` (e.g.
``method``, ``node_radius``, ``edge_radius``, ``radius``,
``quad_segs``, ``error_on_empty``).
Returns:
A ``SegmentationMask`` with the rasterized geometry (all background
if the geometry is empty or has zero area).
Raises:
ValueError: Propagated from :meth:`to_roi` for a ``"shapes"``
misconfiguration, an unknown method, or (when
``error_on_empty`` is ``True``) an empty geometry.
"""
from shapely.geometry import MultiPolygon, Polygon
error_on_empty = roi_kwargs.pop("error_on_empty", False)
roi = self.to_roi(error_on_empty=error_on_empty, **roi_kwargs)
# A non-empty but non-fillable geometry (e.g. convex_hull of <3 visible
# points -> Point/LineString) has zero area; rasterize it as all
# background rather than letting _rasterize_geometry raise a TypeError.
rasterizable = isinstance(roi.geometry, (Polygon, MultiPolygon))
if roi.geometry.is_empty or not rasterizable:
from sleap_io.model.mask import (
PredictedSegmentationMask,
UserSegmentationMask,
)
empty = np.zeros((height, width), dtype=bool)
kwargs = dict(
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
category=self.category,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedSegmentationMask.from_numpy(
empty, score=self.score, **kwargs
)
return UserSegmentationMask.from_numpy(empty, **kwargs)
return roi.to_mask(height, width)
def __getitem__(self, node: int | str | Node) -> np.ndarray:
"""Return the point associated with a node."""
if type(node) is not int:
node = self.skeleton.index(node)
return self.points[node]
def __setitem__(self, node: int | str | Node, value):
"""Set the point associated with a node.
Args:
node: The node to set the point for. Can be an integer index, string name,
or Node object.
value: A tuple or array-like of length 2 containing (x, y) coordinates.
Notes:
This sets the point coordinates and marks the point as visible.
"""
if type(node) is not int:
node = self.skeleton.index(node)
if len(value) < 2:
raise ValueError("Value must have at least 2 elements (x, y)")
self.points[node]["xy"] = value[:2]
self.points[node]["visible"] = True
def __len__(self) -> int:
"""Return the number of points in the instance."""
return len(self.points)
def __repr__(self) -> str:
"""Return a readable representation of the instance."""
pts = self.numpy().tolist()
track = f'"{self.track.name}"' if self.track is not None else self.track
return f"Instance(points={pts}, track={track})"
@property
def n_visible(self) -> int:
"""Return the number of visible points in the instance."""
return sum(self.points["visible"])
@property
def is_empty(self) -> bool:
"""Return `True` if no points are visible on the instance."""
return ~(self.points["visible"].any())
def update_skeleton(self, names_only: bool = False):
"""Update or replace the skeleton associated with the instance.
Args:
names_only: If `True`, only update the node names in the points array. If
`False`, the points array will be updated to match the new skeleton.
"""
if names_only:
# Update the node names.
self.points["name"] = self.skeleton.node_names
return
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(self.points["name"])
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
new_points["name"] = self.skeleton.node_names
self.points = new_points
def replace_skeleton(
self,
new_skeleton: Skeleton,
node_names_map: dict[str, str] | None = None,
):
"""Replace the skeleton associated with the instance.
Args:
new_skeleton: The new `Skeleton` to associate with the instance.
node_names_map: Dictionary mapping nodes in the old skeleton to nodes in the
new skeleton. Keys and values should be specified as lists of strings.
If not provided, only nodes with identical names will be mapped. Points
associated with unmapped nodes will be removed.
Notes:
This method will update the `Instance.skeleton` attribute and the
`Instance.points` attribute in place (a copy is made of the points array).
It is recommended to use `Labels.replace_skeleton` instead of this method if
more flexible node mapping is required.
"""
# Update skeleton object.
# old_skeleton = self.skeleton
self.skeleton = new_skeleton
# Get node names with replacements from node map if possible.
# old_node_names = old_skeleton.node_names
old_node_names = self.points["name"].tolist()
if node_names_map is not None:
old_node_names = [node_names_map.get(node, node) for node in old_node_names]
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(old_node_names)
# old_node_inds = np.array(old_node_inds).reshape(-1, 1)
# new_node_inds = np.array(new_node_inds).reshape(-1, 1)
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
self.points = new_points
self.points["name"] = self.skeleton.node_names
def same_pose_as(self, other: "Instance", tolerance: float = None) -> bool:
"""Check if this instance has the same pose as another instance.
Args:
other: Another instance to compare with.
tolerance: Maximum distance (in pixels) between corresponding points
for them to be considered the same. If None (default), uses exact
comparison including proper NaN handling.
Returns:
True if the instances have the same pose within tolerance, False otherwise.
Notes:
Two instances are considered to have the same pose if:
- They have the same skeleton structure
- When tolerance is None: All coordinates match exactly (including NaN)
- When tolerance is specified: All visible points are within tolerance
distance and NaN patterns match exactly
"""
# Check skeleton compatibility
if not self.skeleton.matches(other.skeleton):
return False
if tolerance is None:
# Exact comparison using numpy arrays with proper NaN handling
return np.array_equal(self.numpy(), other.numpy(), equal_nan=True)
else:
# Tolerance-based comparison with proper NaN handling
self_array = self.numpy()
other_array = other.numpy()
# First, check if NaN patterns match exactly
self_nan_mask = np.isnan(self_array)
other_nan_mask = np.isnan(other_array)
if not np.array_equal(self_nan_mask, other_nan_mask):
return False
# Get mask for non-NaN values
non_nan_mask = ~self_nan_mask
# If all values are NaN, they're considered equal
if not non_nan_mask.any():
return True
# Calculate distances only for non-NaN points
self_pts = self_array[non_nan_mask]
other_pts = other_array[non_nan_mask]
# Reshape to handle the coordinate pairs properly
self_pts = self_pts.reshape(-1, 2)
other_pts = other_pts.reshape(-1, 2)
distances = np.linalg.norm(self_pts - other_pts, axis=1)
return np.all(distances <= tolerance)
def same_identity_as(self, other: "Instance") -> bool:
"""Check if this instance has the same identity as another instance.
Args:
other: Another instance to compare with.
Returns:
True if both instances share the same identity, False otherwise.
Notes:
Global `Identity` takes precedence: if both instances carry an
`Identity`, they match when their `name`s match (which survives
serialization and cross-file merges). Otherwise this falls back to
the ephemeral `Track`, where instances match only when they share the
same `Track` object (by object identity, not just by name).
"""
if self.identity is not None and other.identity is not None:
return self.identity.matches(other.identity, method="name")
if self.track is None or other.track is None:
return False
return self.track is other.track
def overlaps_with(self, other: "Instance", iou_threshold: float = 0.5) -> bool:
"""Check if this instance overlaps with another based on bounding box IoU.
Args:
other: Another instance to compare with.
iou_threshold: Minimum IoU (Intersection over Union) value to consider
the instances as overlapping.
Returns:
True if the instances overlap above the threshold, False otherwise.
Notes:
Overlap is computed using the bounding boxes of visible points.
If either instance has no visible points, they don't overlap.
"""
# Get visible points for both instances
self_visible = self.points["visible"]
other_visible = other.points["visible"]
if not self_visible.any() or not other_visible.any():
return False
# Calculate bounding boxes
self_pts = self.points["xy"][self_visible]
other_pts = other.points["xy"][other_visible]
self_bbox = np.array(
[
[np.min(self_pts[:, 0]), np.min(self_pts[:, 1])], # min x, y
[np.max(self_pts[:, 0]), np.max(self_pts[:, 1])], # max x, y
]
)
other_bbox = np.array(
[
[np.min(other_pts[:, 0]), np.min(other_pts[:, 1])],
[np.max(other_pts[:, 0]), np.max(other_pts[:, 1])],
]
)
# Calculate intersection
intersection_min = np.maximum(self_bbox[0], other_bbox[0])
intersection_max = np.minimum(self_bbox[1], other_bbox[1])
if np.any(intersection_min >= intersection_max):
# No intersection
return False
intersection_area = np.prod(intersection_max - intersection_min)
# Calculate union
self_area = np.prod(self_bbox[1] - self_bbox[0])
other_area = np.prod(other_bbox[1] - other_bbox[0])
union_area = self_area + other_area - intersection_area
# Calculate IoU
iou = intersection_area / union_area if union_area > 0 else 0
return iou >= iou_threshold
def bounding_box(self) -> np.ndarray | None:
"""Get the bounding box of visible points.
Returns:
A numpy array of shape (2, 2) with [[min_x, min_y], [max_x, max_y]],
or None if there are no visible points.
"""
visible = self.points["visible"]
if not visible.any():
return None
pts = self.points["xy"][visible]
return np.array(
[
[np.min(pts[:, 0]), np.min(pts[:, 1])],
[np.max(pts[:, 0]), np.max(pts[:, 1])],
]
)
__annotations__ = {'points': 'PointsArray', 'skeleton': 'Skeleton', 'track': 'Track | None', 'tracking_score': 'float | None', 'identity': 'Identity | None', 'identity_score': 'float | None', 'category': 'Category | None', 'category_score': 'float | None', 'from_predicted': "'PredictedInstance | None'", 'identity_embedding': 'Embedding | None', 'category_embedding': 'Embedding | None'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'This class represents a ground truth instance such as an animal.\n\nAn `Instance` has a set of landmarks (points) that correspond to a `Skeleton`. Each\npoint is associated with a `Node` in the skeleton. The points are stored in a\nstructured numpy array with columns for x, y, visible, complete and name.\n\nThe `Instance` may also be associated with a `Track` which links multiple instances\ntogether across frames or videos.\n\nAttributes:\n points: A numpy structured array with columns for xy, visible and complete. The\n array should have shape `(n_nodes,)`. This representation is useful for\n performance efficiency when working with large datasets.\n skeleton: The `Skeleton` that describes the `Node`s and `Edge`s associated with\n this instance.\n track: An optional `Track` associated with a unique animal/object across frames\n or videos.\n tracking_score: The score associated with the `Track` assignment. This is\n typically the value from the score matrix used in an identity assignment.\n This is `None` if the instance is not associated with a track or if the\n track was assigned manually.\n identity: An optional `Identity` representing the global, ground-truth animal\n this instance belongs to (persistent across videos/sessions). Unlike\n `track` (an ephemeral, video-local tracklet), `Identity` is the cross-file\n re-identification key. `None` if no global identity is assigned.\n identity_score: The score associated with the `identity` assignment (e.g. the\n cosine similarity to a re-ID gallery prototype). This is `None` if the\n instance has no identity or the identity was assigned manually. Kept\n separate from `tracking_score` (short-term tracklet vs long-term identity).\n from_predicted: The `PredictedInstance` (if any) that this instance was\n initialized from. This is used with human-in-the-loop workflows.\n identity_embedding: An optional `Embedding` describing this instance\'s\n appearance for re-identification (e.g. a vector produced by a re-ID\n model). ``None`` by default.\n category: An optional `Category` representing the *class* this instance\n belongs to (e.g. ``"female_fly"``, ``"fur_shaved"``), typically\n assigned by classification or re-ID. Mirrors `identity` but groups by\n class rather than individual. `None` if no category is assigned.\n category_score: The score associated with the `category` assignment (e.g.\n the classifier confidence). `None` if the instance has no category or\n the category was assigned manually.\n category_embedding: An optional `Embedding` describing this instance\'s\n appearance for classification (the vector the `category` was\n classified from). ``None`` by default.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 397
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('points', 'skeleton', 'track', 'tracking_score', 'identity', 'identity_score', 'category', 'category_score', 'from_predicted', 'identity_embedding', 'category_embedding')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.instance'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('points', 'skeleton', 'track', 'tracking_score', 'identity', 'identity_score', 'category', 'category_score', 'from_predicted', 'identity_embedding', 'category_embedding', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('points', 'skeleton')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
centroid_xy
property
¶
Mean of visible point coordinates as (x, y), or None.
Returns:
| Type | Description |
|---|---|
|
A tuple |
is_empty
property
¶
Return True if no points are visible on the instance.
n_visible
property
¶
Return the number of visible points in the instance.
__attrs_post_init__()
¶
Convert the points array after initialization.
Source code in sleap_io/model/instance.py
def __attrs_post_init__(self):
"""Convert the points array after initialization."""
if not isinstance(self.points, PointsArray):
self.points = self._convert_points(self.points, self.skeleton)
# Ensure points have node names
if "name" in self.points.dtype.names and not all(self.points["name"]):
self.points["name"] = self.skeleton.node_names
__getitem__(node)
¶
__init__(points, skeleton, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, from_predicted=None, identity_embedding=None, category_embedding=None)
¶
Method generated by attrs for class Instance.
Source code in sleap_io/model/instance.py
"""Data structures for data associated with a single instance such as an animal.
The `Instance` class is a SLEAP data structure that contains a collection of points that
correspond to landmarks within a `Skeleton`.
`PredictedInstance` additionally contains metadata associated with how the instance was
estimated, such as confidence scores.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import attrs
__len__()
¶
__repr__()
¶
Return a readable representation of the instance.
__setattr__(name, val)
¶
Method generated by attrs for class Instance.
Source code in sleap_io/model/instance.py
__setitem__(node, value)
¶
Set the point associated with a node.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
int | str | Node
|
The node to set the point for. Can be an integer index, string name, or Node object. |
required |
value
|
A tuple or array-like of length 2 containing (x, y) coordinates. |
required |
Notes
This sets the point coordinates and marks the point as visible.
Source code in sleap_io/model/instance.py
def __setitem__(self, node: int | str | Node, value):
"""Set the point associated with a node.
Args:
node: The node to set the point for. Can be an integer index, string name,
or Node object.
value: A tuple or array-like of length 2 containing (x, y) coordinates.
Notes:
This sets the point coordinates and marks the point as visible.
"""
if type(node) is not int:
node = self.skeleton.index(node)
if len(value) < 2:
raise ValueError("Value must have at least 2 elements (x, y)")
self.points[node]["xy"] = value[:2]
self.points[node]["visible"] = True
bounding_box()
¶
Get the bounding box of visible points.
Returns:
| Type | Description |
|---|---|
ndarray | None
|
A numpy array of shape (2, 2) with [[min_x, min_y], [max_x, max_y]], or None if there are no visible points. |
Source code in sleap_io/model/instance.py
def bounding_box(self) -> np.ndarray | None:
"""Get the bounding box of visible points.
Returns:
A numpy array of shape (2, 2) with [[min_x, min_y], [max_x, max_y]],
or None if there are no visible points.
"""
visible = self.points["visible"]
if not visible.any():
return None
pts = self.points["xy"][visible]
return np.array(
[
[np.min(pts[:, 0]), np.min(pts[:, 1])],
[np.max(pts[:, 0]), np.max(pts[:, 1])],
]
)
empty(skeleton, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, identity_embedding=None, category_embedding=None, from_predicted=None)
classmethod
¶
Create an empty instance with no points.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
skeleton
|
Skeleton
|
The |
required |
track
|
Track | None
|
An optional |
None
|
tracking_score
|
float | None
|
The score associated with the |
None
|
identity
|
Identity | None
|
An optional global |
None
|
identity_score
|
float | None
|
The score associated with the |
None
|
category
|
Category | None
|
An optional |
None
|
category_score
|
float | None
|
The score associated with the |
None
|
identity_embedding
|
Embedding | None
|
An optional re-ID |
None
|
category_embedding
|
Embedding | None
|
An optional classification |
None
|
from_predicted
|
PredictedInstance | None
|
The |
None
|
Returns:
| Type | Description |
|---|---|
Instance
|
An |
Source code in sleap_io/model/instance.py
@classmethod
def empty(
cls,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an empty instance with no points.
Args:
skeleton: The `Skeleton` that this `Instance` is associated with.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` with an empty numpy array of shape `(n_nodes,)`.
"""
points = PointsArray.empty(len(skeleton))
points["name"] = skeleton.node_names
return cls(
points=points,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
from_numpy(points_data, skeleton, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, identity_embedding=None, category_embedding=None, from_predicted=None)
classmethod
¶
Create an instance object from a numpy array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
points_data
|
ndarray
|
A numpy array of shape If If this is provided as a structured array, it will be used without copy if it has the correct dtype. Otherwise, a new structured array will be created reusing the provided data. |
required |
skeleton
|
Skeleton
|
The |
required |
track
|
Track | None
|
An optional |
None
|
tracking_score
|
float | None
|
The score associated with the |
None
|
identity
|
Identity | None
|
An optional global |
None
|
identity_score
|
float | None
|
The score associated with the |
None
|
category
|
Category | None
|
An optional |
None
|
category_score
|
float | None
|
The score associated with the |
None
|
identity_embedding
|
Embedding | None
|
An optional re-ID |
None
|
category_embedding
|
Embedding | None
|
An optional classification |
None
|
from_predicted
|
PredictedInstance | None
|
The |
None
|
Returns:
| Type | Description |
|---|---|
Instance
|
An |
Source code in sleap_io/model/instance.py
@classmethod
def from_numpy(
cls,
points_data: np.ndarray,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an instance object from a numpy array.
Args:
points_data: A numpy array of shape `(n_nodes, D)` corresponding to the
points of the skeleton. Values of `np.nan` indicate "missing" nodes and
will be reflected in the "visible" field.
If `D == 2`, the array should have columns for x and y.
If `D == 3`, the array should have columns for x, y and visible.
If `D == 4`, the array should have columns for x, y, visible and
complete.
If this is provided as a structured array, it will be used without copy
if it has the correct dtype. Otherwise, a new structured array will be
created reusing the provided data.
skeleton: The `Skeleton` that this `Instance` is associated with. It should
have `n_nodes` nodes.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` object with the specified points.
"""
return cls(
points=points_data,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
numpy(invisible_as_nan=True)
¶
Return the instance points as a (n_nodes, 2) numpy array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
invisible_as_nan
|
bool
|
If |
True
|
Returns:
| Type | Description |
|---|---|
ndarray
|
A numpy array of shape |
Notes
This will always return a copy of the array.
If you need to avoid making a copy, just access the Instance.points["xy"]
attribute directly. This will not replace invisible points with np.nan.
Source code in sleap_io/model/instance.py
def numpy(
self,
invisible_as_nan: bool = True,
) -> np.ndarray:
"""Return the instance points as a `(n_nodes, 2)` numpy array.
Args:
invisible_as_nan: If `True` (the default), points that are not visible will
be set to `np.nan`. If `False`, they will be whatever the stored value
of `Instance.points["xy"]` is.
Returns:
A numpy array of shape `(n_nodes, 2)` corresponding to the points of the
skeleton. Values of `np.nan` indicate "missing" nodes.
Notes:
This will always return a copy of the array.
If you need to avoid making a copy, just access the `Instance.points["xy"]`
attribute directly. This will not replace invisible points with `np.nan`.
"""
if invisible_as_nan:
return np.where(
self.points["visible"].reshape(-1, 1), self.points["xy"], np.nan
)
else:
return self.points["xy"].copy()
overlaps_with(other, iou_threshold=0.5)
¶
Check if this instance overlaps with another based on bounding box IoU.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Instance
|
Another instance to compare with. |
required |
iou_threshold
|
float
|
Minimum IoU (Intersection over Union) value to consider the instances as overlapping. |
0.5
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the instances overlap above the threshold, False otherwise. |
Notes
Overlap is computed using the bounding boxes of visible points. If either instance has no visible points, they don't overlap.
Source code in sleap_io/model/instance.py
def overlaps_with(self, other: "Instance", iou_threshold: float = 0.5) -> bool:
"""Check if this instance overlaps with another based on bounding box IoU.
Args:
other: Another instance to compare with.
iou_threshold: Minimum IoU (Intersection over Union) value to consider
the instances as overlapping.
Returns:
True if the instances overlap above the threshold, False otherwise.
Notes:
Overlap is computed using the bounding boxes of visible points.
If either instance has no visible points, they don't overlap.
"""
# Get visible points for both instances
self_visible = self.points["visible"]
other_visible = other.points["visible"]
if not self_visible.any() or not other_visible.any():
return False
# Calculate bounding boxes
self_pts = self.points["xy"][self_visible]
other_pts = other.points["xy"][other_visible]
self_bbox = np.array(
[
[np.min(self_pts[:, 0]), np.min(self_pts[:, 1])], # min x, y
[np.max(self_pts[:, 0]), np.max(self_pts[:, 1])], # max x, y
]
)
other_bbox = np.array(
[
[np.min(other_pts[:, 0]), np.min(other_pts[:, 1])],
[np.max(other_pts[:, 0]), np.max(other_pts[:, 1])],
]
)
# Calculate intersection
intersection_min = np.maximum(self_bbox[0], other_bbox[0])
intersection_max = np.minimum(self_bbox[1], other_bbox[1])
if np.any(intersection_min >= intersection_max):
# No intersection
return False
intersection_area = np.prod(intersection_max - intersection_min)
# Calculate union
self_area = np.prod(self_bbox[1] - self_bbox[0])
other_area = np.prod(other_bbox[1] - other_bbox[0])
union_area = self_area + other_area - intersection_area
# Calculate IoU
iou = intersection_area / union_area if union_area > 0 else 0
return iou >= iou_threshold
replace_skeleton(new_skeleton, node_names_map=None)
¶
Replace the skeleton associated with the instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_skeleton
|
Skeleton
|
The new |
required |
node_names_map
|
dict[str, str] | None
|
Dictionary mapping nodes in the old skeleton to nodes in the new skeleton. Keys and values should be specified as lists of strings. If not provided, only nodes with identical names will be mapped. Points associated with unmapped nodes will be removed. |
None
|
Notes
This method will update the Instance.skeleton attribute and the
Instance.points attribute in place (a copy is made of the points array).
It is recommended to use Labels.replace_skeleton instead of this method if
more flexible node mapping is required.
Source code in sleap_io/model/instance.py
def replace_skeleton(
self,
new_skeleton: Skeleton,
node_names_map: dict[str, str] | None = None,
):
"""Replace the skeleton associated with the instance.
Args:
new_skeleton: The new `Skeleton` to associate with the instance.
node_names_map: Dictionary mapping nodes in the old skeleton to nodes in the
new skeleton. Keys and values should be specified as lists of strings.
If not provided, only nodes with identical names will be mapped. Points
associated with unmapped nodes will be removed.
Notes:
This method will update the `Instance.skeleton` attribute and the
`Instance.points` attribute in place (a copy is made of the points array).
It is recommended to use `Labels.replace_skeleton` instead of this method if
more flexible node mapping is required.
"""
# Update skeleton object.
# old_skeleton = self.skeleton
self.skeleton = new_skeleton
# Get node names with replacements from node map if possible.
# old_node_names = old_skeleton.node_names
old_node_names = self.points["name"].tolist()
if node_names_map is not None:
old_node_names = [node_names_map.get(node, node) for node in old_node_names]
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(old_node_names)
# old_node_inds = np.array(old_node_inds).reshape(-1, 1)
# new_node_inds = np.array(new_node_inds).reshape(-1, 1)
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
self.points = new_points
self.points["name"] = self.skeleton.node_names
same_identity_as(other)
¶
Check if this instance has the same identity as another instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Instance
|
Another instance to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if both instances share the same identity, False otherwise. |
Notes
Global Identity takes precedence: if both instances carry an
Identity, they match when their names match (which survives
serialization and cross-file merges). Otherwise this falls back to
the ephemeral Track, where instances match only when they share the
same Track object (by object identity, not just by name).
Source code in sleap_io/model/instance.py
def same_identity_as(self, other: "Instance") -> bool:
"""Check if this instance has the same identity as another instance.
Args:
other: Another instance to compare with.
Returns:
True if both instances share the same identity, False otherwise.
Notes:
Global `Identity` takes precedence: if both instances carry an
`Identity`, they match when their `name`s match (which survives
serialization and cross-file merges). Otherwise this falls back to
the ephemeral `Track`, where instances match only when they share the
same `Track` object (by object identity, not just by name).
"""
if self.identity is not None and other.identity is not None:
return self.identity.matches(other.identity, method="name")
if self.track is None or other.track is None:
return False
return self.track is other.track
same_pose_as(other, tolerance=None)
¶
Check if this instance has the same pose as another instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Instance
|
Another instance to compare with. |
required |
tolerance
|
float
|
Maximum distance (in pixels) between corresponding points for them to be considered the same. If None (default), uses exact comparison including proper NaN handling. |
None
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the instances have the same pose within tolerance, False otherwise. |
Notes
Two instances are considered to have the same pose if: - They have the same skeleton structure - When tolerance is None: All coordinates match exactly (including NaN) - When tolerance is specified: All visible points are within tolerance distance and NaN patterns match exactly
Source code in sleap_io/model/instance.py
def same_pose_as(self, other: "Instance", tolerance: float = None) -> bool:
"""Check if this instance has the same pose as another instance.
Args:
other: Another instance to compare with.
tolerance: Maximum distance (in pixels) between corresponding points
for them to be considered the same. If None (default), uses exact
comparison including proper NaN handling.
Returns:
True if the instances have the same pose within tolerance, False otherwise.
Notes:
Two instances are considered to have the same pose if:
- They have the same skeleton structure
- When tolerance is None: All coordinates match exactly (including NaN)
- When tolerance is specified: All visible points are within tolerance
distance and NaN patterns match exactly
"""
# Check skeleton compatibility
if not self.skeleton.matches(other.skeleton):
return False
if tolerance is None:
# Exact comparison using numpy arrays with proper NaN handling
return np.array_equal(self.numpy(), other.numpy(), equal_nan=True)
else:
# Tolerance-based comparison with proper NaN handling
self_array = self.numpy()
other_array = other.numpy()
# First, check if NaN patterns match exactly
self_nan_mask = np.isnan(self_array)
other_nan_mask = np.isnan(other_array)
if not np.array_equal(self_nan_mask, other_nan_mask):
return False
# Get mask for non-NaN values
non_nan_mask = ~self_nan_mask
# If all values are NaN, they're considered equal
if not non_nan_mask.any():
return True
# Calculate distances only for non-NaN points
self_pts = self_array[non_nan_mask]
other_pts = other_array[non_nan_mask]
# Reshape to handle the coordinate pairs properly
self_pts = self_pts.reshape(-1, 2)
other_pts = other_pts.reshape(-1, 2)
distances = np.linalg.norm(self_pts - other_pts, axis=1)
return np.all(distances <= tolerance)
to_bbox(mode='tight', size=None, padding=0.0, node=None, center_method='center_of_mass', rotated=False, error_on_empty=False)
¶
Create a bounding box from this instance.
A PredictedInstance yields a PredictedBoundingBox carrying its
score; any other instance yields a UserBoundingBox. Metadata
(track, tracking_score, identity, identity_score,
identity_embedding, category, category_score,
category_embedding, instance=self) is propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mode
|
str
|
|
'tight'
|
size
|
float | tuple[float, float] | None
|
Box size for |
None
|
padding
|
float | tuple[float, float]
|
Amount to inflate the box outward. Scalar applies to both
axes; a |
0.0
|
node
|
int | str | None
|
Node specification passed to the centroid computation for
|
None
|
center_method
|
str
|
Centroid method used to locate the box center for
|
'center_of_mass'
|
rotated
|
bool
|
For |
False
|
error_on_empty
|
bool
|
If |
False
|
Returns:
| Type | Description |
|---|---|
BoundingBox
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
For an unknown |
Source code in sleap_io/model/instance.py
def to_bbox(
self,
mode: str = "tight",
size: float | tuple[float, float] | None = None,
padding: float | tuple[float, float] = 0.0,
node: int | str | None = None,
center_method: str = "center_of_mass",
rotated: bool = False,
error_on_empty: bool = False,
) -> "BoundingBox":
"""Create a bounding box from this instance.
A ``PredictedInstance`` yields a ``PredictedBoundingBox`` carrying its
``score``; any other instance yields a ``UserBoundingBox``. Metadata
(``track``, ``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
mode: ``"tight"`` to fit the visible points, or ``"centered"`` to
build a fixed-``size`` box centered on a computed centroid.
size: Box size for ``mode="centered"``. A scalar yields a square box;
a ``(w, h)`` tuple sets width and height independently. Required
for ``mode="centered"``.
padding: Amount to inflate the box outward. Scalar applies to both
axes; a ``(px, py)`` tuple applies per-axis. Negative values
shrink the box.
node: Node specification passed to the centroid computation for
``mode="centered"`` with ``center_method="anchor"``.
center_method: Centroid method used to locate the box center for
``mode="centered"`` (see :meth:`to_centroid`).
rotated: For ``mode="tight"``, if ``True`` fit a minimum-area oriented
box from the convex hull of visible points; otherwise fit an
axis-aligned box.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) box.
Returns:
A ``BoundingBox`` enclosing the instance (or NaN corners if empty).
Raises:
ValueError: For an unknown ``mode``, a missing ``size`` for
``mode="centered"``, or (when ``error_on_empty`` is ``True``)
when there are no visible points.
"""
from sleap_io.model.bbox import PredictedBoundingBox, UserBoundingBox
from sleap_io.model.roi import (
_apply_padding,
_geometry_to_bbox_coords,
_pose_to_geometry,
)
nan = float("nan")
angle = 0.0
if mode == "tight":
pts = self.numpy(invisible_as_nan=True)
visible = ~np.isnan(pts[:, 0])
if not visible.any():
if error_on_empty:
raise ValueError("No visible points to compute bounding box.")
x1 = y1 = x2 = y2 = nan
elif rotated:
hull = _pose_to_geometry(
pts, self.skeleton.edge_inds, method="convex_hull"
)
x1, y1, x2, y2, angle = _geometry_to_bbox_coords(hull, rotated=True)
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
vis = pts[visible]
x1 = float(vis[:, 0].min())
y1 = float(vis[:, 1].min())
x2 = float(vis[:, 0].max())
y2 = float(vis[:, 1].max())
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
elif mode == "centered":
if size is None:
raise ValueError("'size' is required for mode='centered'.")
centroid = self.to_centroid(
method=center_method, node=node, error_on_empty=error_on_empty
)
if centroid.is_empty:
x1 = y1 = x2 = y2 = nan
else:
cx, cy = centroid.xy
if isinstance(size, (tuple, list)):
w, h = size
else:
w = h = size
x1 = cx - w / 2
y1 = cy - h / 2
x2 = cx + w / 2
y2 = cy + h / 2
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
raise ValueError(f"Unknown mode {mode!r}. Expected 'tight' or 'centered'.")
kwargs = dict(
x1=x1,
y1=y1,
x2=x2,
y2=y2,
angle=angle,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedBoundingBox(score=self.score, **kwargs)
return UserBoundingBox(**kwargs)
to_centroid(method='center_of_mass', node=None, fallback=None, error_on_empty=False, **kwargs)
¶
Create a Centroid from this instance.
Delegates to Centroid.from_pose(). A PredictedInstance yields a
PredictedCentroid carrying its score; any other instance yields a
UserCentroid. Metadata (track, tracking_score, identity,
identity_score, identity_embedding, category,
category_score, category_embedding, instance=self) is
propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
str
|
Computation method ( |
'center_of_mass'
|
node
|
int | str | None
|
Node specification for the |
None
|
fallback
|
str | None
|
For the |
None
|
error_on_empty
|
bool
|
If |
False
|
**kwargs
|
Additional keyword arguments passed to the centroid constructor. |
required |
Returns:
| Type | Description |
|---|---|
Centroid
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
For an unknown |
Source code in sleap_io/model/instance.py
def to_centroid(
self,
method: str = "center_of_mass",
node: int | str | None = None,
fallback: str | None = None,
error_on_empty: bool = False,
**kwargs,
) -> "Centroid":
"""Create a ``Centroid`` from this instance.
Delegates to ``Centroid.from_pose()``. A ``PredictedInstance`` yields a
``PredictedCentroid`` carrying its ``score``; any other instance yields a
``UserCentroid``. Metadata (``track``, ``tracking_score``, ``identity``,
``identity_score``, ``identity_embedding``, ``category``,
``category_score``, ``category_embedding``, ``instance=self``) is
propagated.
Args:
method: Computation method (``"center_of_mass"``, ``"bbox_center"``,
``"geometric_median"``, or ``"anchor"``).
node: Node specification for the ``"anchor"`` method. Can be a node
name (str) or index (int).
fallback: For the ``"anchor"`` method, a non-anchor method to fall
back to when the anchor node is occluded.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) centroid.
**kwargs: Additional keyword arguments passed to the centroid
constructor.
Returns:
A ``UserCentroid`` or ``PredictedCentroid`` depending on the
instance type.
Raises:
ValueError: For an unknown ``method``, a missing ``node`` for the
``"anchor"`` method, an invalid ``node`` type, or (when
``error_on_empty`` is ``True``) when there are no visible points.
"""
from sleap_io.model.centroid import Centroid
return Centroid.from_pose(
self,
method=method,
node=node,
fallback=fallback,
error_on_empty=error_on_empty,
**kwargs,
)
to_mask(height, width, **roi_kwargs)
¶
Rasterize this instance's ROI geometry into a segmentation mask.
Equivalent to self.to_roi(**roi_kwargs).to_mask(height, width),
except that a zero-area hull (method="convex_hull" over fewer than
three visible points yields a Point or LineString) rasterizes to
an all-background mask here instead of raising. A PredictedInstance
yields a PredictedSegmentationMask carrying its score; any other
instance yields a UserSegmentationMask. Metadata is propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
height
|
int
|
Height of the output mask in pixels. |
required |
width
|
int
|
Width of the output mask in pixels. |
required |
**roi_kwargs
|
Keyword arguments forwarded to :meth: |
required |
Returns:
| Type | Description |
|---|---|
SegmentationMask
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
Propagated from :meth: |
Source code in sleap_io/model/instance.py
def to_mask(self, height: int, width: int, **roi_kwargs) -> "SegmentationMask":
"""Rasterize this instance's ROI geometry into a segmentation mask.
Equivalent to ``self.to_roi(**roi_kwargs).to_mask(height, width)``,
except that a zero-area hull (``method="convex_hull"`` over fewer than
three visible points yields a ``Point`` or ``LineString``) rasterizes to
an all-background mask here instead of raising. A ``PredictedInstance``
yields a ``PredictedSegmentationMask`` carrying its ``score``; any other
instance yields a ``UserSegmentationMask``. Metadata is propagated.
Args:
height: Height of the output mask in pixels.
width: Width of the output mask in pixels.
**roi_kwargs: Keyword arguments forwarded to :meth:`to_roi` (e.g.
``method``, ``node_radius``, ``edge_radius``, ``radius``,
``quad_segs``, ``error_on_empty``).
Returns:
A ``SegmentationMask`` with the rasterized geometry (all background
if the geometry is empty or has zero area).
Raises:
ValueError: Propagated from :meth:`to_roi` for a ``"shapes"``
misconfiguration, an unknown method, or (when
``error_on_empty`` is ``True``) an empty geometry.
"""
from shapely.geometry import MultiPolygon, Polygon
error_on_empty = roi_kwargs.pop("error_on_empty", False)
roi = self.to_roi(error_on_empty=error_on_empty, **roi_kwargs)
# A non-empty but non-fillable geometry (e.g. convex_hull of <3 visible
# points -> Point/LineString) has zero area; rasterize it as all
# background rather than letting _rasterize_geometry raise a TypeError.
rasterizable = isinstance(roi.geometry, (Polygon, MultiPolygon))
if roi.geometry.is_empty or not rasterizable:
from sleap_io.model.mask import (
PredictedSegmentationMask,
UserSegmentationMask,
)
empty = np.zeros((height, width), dtype=bool)
kwargs = dict(
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
category=self.category,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedSegmentationMask.from_numpy(
empty, score=self.score, **kwargs
)
return UserSegmentationMask.from_numpy(empty, **kwargs)
return roi.to_mask(height, width)
to_roi(method='shapes', node_radius=0.0, edge_radius=0.0, radius=0.0, quad_segs=8, error_on_empty=False)
¶
Create a region-of-interest geometry from this instance.
A PredictedInstance yields a PredictedROI carrying its score;
any other instance yields a UserROI. Metadata (track,
tracking_score, identity, identity_score,
identity_embedding, category, category_score,
category_embedding, instance=self) is propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
str
|
|
'shapes'
|
node_radius
|
float
|
Buffer radius around each visible node ( |
0.0
|
edge_radius
|
float
|
Buffer radius around each fully-visible edge segment
( |
0.0
|
radius
|
float
|
Optional buffer applied to the convex hull
( |
0.0
|
quad_segs
|
int
|
Number of segments used to approximate a quarter circle when buffering. |
8
|
error_on_empty
|
bool
|
If |
False
|
Returns:
| Type | Description |
|---|---|
ROI
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in sleap_io/model/instance.py
def to_roi(
self,
method: str = "shapes",
node_radius: float = 0.0,
edge_radius: float = 0.0,
radius: float = 0.0,
quad_segs: int = 8,
error_on_empty: bool = False,
) -> "ROI":
"""Create a region-of-interest geometry from this instance.
A ``PredictedInstance`` yields a ``PredictedROI`` carrying its ``score``;
any other instance yields a ``UserROI``. Metadata (``track``,
``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
method: ``"shapes"`` to union buffered node points and/or edge
segments, or ``"convex_hull"`` to take the convex hull of the
visible points.
node_radius: Buffer radius around each visible node (``"shapes"``
only).
edge_radius: Buffer radius around each fully-visible edge segment
(``"shapes"`` only).
radius: Optional buffer applied to the convex hull
(``"convex_hull"`` only).
quad_segs: Number of segments used to approximate a quarter circle
when buffering.
error_on_empty: If ``True``, raise ``ValueError`` when the resulting
geometry is empty instead of returning an empty-geometry ROI.
Returns:
A ``ROI`` whose geometry encloses the instance (an empty ``Polygon``
if there are no visible points).
Raises:
ValueError: If ``method="shapes"`` with both ``node_radius`` and
``edge_radius`` equal to 0 (a misconfiguration, always raised),
for an unknown ``method``, or (when ``error_on_empty`` is
``True``) when the resulting geometry is empty.
"""
from sleap_io.model.roi import PredictedROI, UserROI, _pose_to_geometry
# Misconfiguration: raise before the empty-points check so that an empty
# instance still surfaces the error.
if method == "shapes" and node_radius == 0 and edge_radius == 0:
raise ValueError(
"method='shapes' requires at least one of node_radius or "
"edge_radius to be > 0."
)
geom = _pose_to_geometry(
self.numpy(invisible_as_nan=True),
self.skeleton.edge_inds,
method=method,
node_radius=node_radius,
edge_radius=edge_radius,
radius=radius,
quad_segs=quad_segs,
)
if geom.is_empty and error_on_empty:
raise ValueError("No visible points to compute ROI geometry.")
kwargs = dict(
geometry=geom,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedROI(score=self.score, **kwargs)
return UserROI(**kwargs)
update_skeleton(names_only=False)
¶
Update or replace the skeleton associated with the instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
names_only
|
bool
|
If |
False
|
Source code in sleap_io/model/instance.py
def update_skeleton(self, names_only: bool = False):
"""Update or replace the skeleton associated with the instance.
Args:
names_only: If `True`, only update the node names in the points array. If
`False`, the points array will be updated to match the new skeleton.
"""
if names_only:
# Update the node names.
self.points["name"] = self.skeleton.node_names
return
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(self.points["name"])
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
new_points["name"] = self.skeleton.node_names
self.points = new_points
LabeledFrame
¶
Labeled data for a single frame of a video.
Attributes:
| Name | Type | Description |
|---|---|---|
video |
The |
|
frame_idx |
The index of the |
|
instances |
List of |
|
is_negative |
If True, this frame is explicitly marked as containing no instances (a "negative" or background frame for training). This is distinct from frames that are simply empty (e.g., instances were deleted). |
|
centroids |
List of |
|
bboxes |
List of |
|
masks |
List of |
|
label_images |
List of |
|
rois |
List of |
Notes
Instances of this class are hashed by identity, not by value. This means that
two LabeledFrame instances with the same attributes will NOT be considered
equal in a set or dict.
Methods:
| Name | Description |
|---|---|
__getitem__ |
Return the |
__init__ |
Method generated by attrs for class LabeledFrame. |
__iter__ |
Iterate over |
__len__ |
Return the number of instances in the frame. |
__repr__ |
Method generated by attrs for class LabeledFrame. |
__setattr__ |
Method generated by attrs for class LabeledFrame. |
append |
Append an annotation to the appropriate frame-level container. |
convert |
Convert annotations between detection modalities. |
matches |
Check if this frame matches another frame's identity. |
merge |
Merge instances from another frame into this frame. |
numpy |
Return all instances in the frame as a numpy array. |
remove_empty_instances |
Remove all instances with no visible points. |
remove_predictions |
Remove all predicted instances and annotations from the frame. |
similarity_to |
Calculate instance overlap metrics with another frame. |
Source code in sleap_io/model/labeled_frame.py
@define(eq=False)
class LabeledFrame:
"""Labeled data for a single frame of a video.
Attributes:
video: The `Video` associated with this `LabeledFrame`.
frame_idx: The index of the `LabeledFrame` in the `Video`.
instances: List of `Instance` objects associated with this `LabeledFrame`.
is_negative: If True, this frame is explicitly marked as containing no
instances (a "negative" or background frame for training). This is
distinct from frames that are simply empty (e.g., instances were deleted).
centroids: List of `Centroid` annotations for this frame.
bboxes: List of `BoundingBox` annotations for this frame.
masks: List of `SegmentationMask` annotations for this frame.
label_images: List of `LabelImage` annotations for this frame.
rois: List of `ROI` annotations for this frame.
Notes:
Instances of this class are hashed by identity, not by value. This means that
two `LabeledFrame` instances with the same attributes will NOT be considered
equal in a set or dict.
"""
video: Video
frame_idx: int = field(converter=int)
instances: list[Instance | PredictedInstance] = field(factory=list)
is_negative: bool = field(default=False)
centroids: "list[Centroid]" = field(factory=list)
bboxes: "list[BoundingBox]" = field(factory=list)
masks: "list[SegmentationMask]" = field(factory=list)
label_images: "list[LabelImage]" = field(factory=list)
rois: "list[ROI]" = field(factory=list)
def append(
self,
annotation: (
"Instance | PredictedInstance | Centroid"
" | BoundingBox | SegmentationMask | LabelImage | ROI"
),
) -> None:
"""Append an annotation to the appropriate frame-level container.
Routes the annotation to the correct list based on its type:
``Instance``/``PredictedInstance`` → ``instances``,
``Centroid`` → ``centroids``, ``BoundingBox`` → ``bboxes``,
``SegmentationMask`` → ``masks``, ``LabelImage`` → ``label_images``,
``ROI`` → ``rois``.
Args:
annotation: The annotation object to add.
Raises:
TypeError: If the annotation type is not recognized.
"""
from sleap_io.model.bbox import BoundingBox
from sleap_io.model.centroid import Centroid
from sleap_io.model.label_image import LabelImage
from sleap_io.model.mask import SegmentationMask
from sleap_io.model.roi import ROI
if isinstance(annotation, (Instance, PredictedInstance)):
self.instances.append(annotation)
elif isinstance(annotation, Centroid):
self.centroids.append(annotation)
elif isinstance(annotation, BoundingBox):
self.bboxes.append(annotation)
elif isinstance(annotation, SegmentationMask):
self.masks.append(annotation)
elif isinstance(annotation, LabelImage):
self.label_images.append(annotation)
elif isinstance(annotation, ROI):
self.rois.append(annotation)
else:
raise TypeError(
f"Cannot append {type(annotation).__name__} to LabeledFrame. "
f"Expected one of: Instance, PredictedInstance, Centroid, "
f"BoundingBox, SegmentationMask, LabelImage, ROI."
)
def __len__(self) -> int:
"""Return the number of instances in the frame."""
return len(self.instances)
def __getitem__(self, key: int) -> Instance | PredictedInstance:
"""Return the `Instance` at `key` index in the `instances` list."""
return self.instances[key]
def __iter__(self):
"""Iterate over `Instance`s in `instances` list."""
return iter(self.instances)
@property
def user_instances(self) -> list[Instance]:
"""Frame instances that are user-labeled (`Instance` objects)."""
return [inst for inst in self.instances if type(inst) is Instance]
@property
def has_user_instances(self) -> bool:
"""Return True if the frame has any user-labeled instances."""
for inst in self.instances:
if type(inst) is Instance:
return True
return False
@property
def is_user_labeled(self) -> bool:
"""Return True if frame has user instances/annotations OR is negative.
This property indicates whether the frame represents intentional user
annotation, either through labeled instances, user annotations
(centroids, bboxes, ROIs, masks, label images), or explicit marking as a
negative/background frame.
"""
from sleap_io.model.label_image import PredictedLabelImage
from sleap_io.model.mask import PredictedSegmentationMask
return (
self.has_user_instances
or self.is_negative
or any(not c.is_predicted for c in self.centroids)
or any(not b.is_predicted for b in self.bboxes)
or any(not r.is_predicted for r in self.rois)
or any(not isinstance(m, PredictedSegmentationMask) for m in self.masks)
or any(not isinstance(li, PredictedLabelImage) for li in self.label_images)
)
@property
def predicted_instances(self) -> list[Instance]:
"""Frame instances that are predicted by a model (`PredictedInstance`)."""
return [inst for inst in self.instances if type(inst) is PredictedInstance]
@property
def has_predicted_instances(self) -> bool:
"""Return True if the frame has any predicted instances."""
for inst in self.instances:
if type(inst) is PredictedInstance:
return True
return False
def numpy(self) -> np.ndarray:
"""Return all instances in the frame as a numpy array.
Returns:
Points as a numpy array of shape `(n_instances, n_nodes, 2)`.
Note that the order of the instances is arbitrary.
"""
n_instances = len(self.instances)
n_nodes = len(self.instances[0]) if n_instances > 0 else 0
pts = np.full((n_instances, n_nodes, 2), np.nan)
for i, inst in enumerate(self.instances):
pts[i] = inst.numpy()[:, 0:2]
return pts
@property
def image(self) -> np.ndarray:
"""Return the image of the frame as a numpy array."""
return self.video[self.frame_idx]
@property
def unused_predictions(self) -> list[Instance]:
"""Return a list of "unused" `PredictedInstance` objects in frame.
This is all of the `PredictedInstance` objects which do not have a corresponding
`Instance` in the same track in the same frame.
"""
unused_predictions = []
any_tracks = [inst.track for inst in self.instances if inst.track is not None]
if len(any_tracks):
# Use tracks to determine which predicted instances have been used
used_tracks = [
inst.track
for inst in self.instances
if type(inst) is Instance and inst.track is not None
]
unused_predictions = [
inst
for inst in self.instances
if inst.track not in used_tracks and type(inst) is PredictedInstance
]
else:
# Use from_predicted to determine which predicted instances have been used
# TODO: should we always do this instead of using tracks?
used_instances = [
inst.from_predicted
for inst in self.instances
if inst.from_predicted is not None
]
unused_predictions = [
inst
for inst in self.instances
if type(inst) is PredictedInstance and inst not in used_instances
]
return unused_predictions
@property
def unused_predicted_masks(self) -> list["SegmentationMask"]:
"""Return predicted masks in this frame not yet adopted by a user mask.
A `PredictedSegmentationMask` is considered *adopted* (and so excluded
from the result) when some `UserSegmentationMask` in the same frame
either links to it via `from_predicted` (checked first) or, lacking an
explicit link, spatially overlaps it (bbox-centroid distance within 5 px,
the auto-merge default). This mirrors the link-first, spatial-fallback
precedence used by the auto-merge cascade and supports the
"retrain only what a human corrected" workflow.
This is the segmentation-mask analogue of `unused_predictions` (which
covers `PredictedInstance` objects).
Returns:
The `PredictedSegmentationMask` objects with no adopting user mask.
"""
from sleap_io.model.mask import PredictedSegmentationMask
predicted = [m for m in self.masks if isinstance(m, PredictedSegmentationMask)]
if not predicted:
return []
user_masks = [m for m in self.masks if not m.is_predicted]
adopted: set[int] = set()
# Link-first: predicted masks explicitly adopted via from_predicted.
for u in user_masks:
src = getattr(u, "from_predicted", None)
if src is not None:
adopted.add(id(src))
# Spatial fallback: a user mask overlaps a still-unadopted prediction.
remaining = [m for m in predicted if id(m) not in adopted]
if remaining and user_masks:
for self_idx, _other_idx, _score in _find_annotation_matches(
remaining, user_masks, "masks", 5.0
):
adopted.add(id(remaining[self_idx]))
return [m for m in predicted if id(m) not in adopted]
def remove_predictions(self):
"""Remove all predicted instances and annotations from the frame."""
from sleap_io.model.bbox import PredictedBoundingBox
from sleap_io.model.centroid import PredictedCentroid
from sleap_io.model.label_image import PredictedLabelImage
from sleap_io.model.mask import PredictedSegmentationMask
from sleap_io.model.roi import PredictedROI
self.instances = [inst for inst in self.instances if type(inst) is Instance]
self.centroids = [
c for c in self.centroids if not isinstance(c, PredictedCentroid)
]
self.bboxes = [
b for b in self.bboxes if not isinstance(b, PredictedBoundingBox)
]
self.masks = [
m for m in self.masks if not isinstance(m, PredictedSegmentationMask)
]
self.label_images = [
li for li in self.label_images if not isinstance(li, PredictedLabelImage)
]
self.rois = [r for r in self.rois if not isinstance(r, PredictedROI)]
def remove_empty_instances(self):
"""Remove all instances with no visible points."""
self.instances = [inst for inst in self.instances if not inst.is_empty]
def convert(
self,
to: str,
source: str = "pose",
inplace: bool = False,
**kwargs,
) -> list:
"""Convert annotations between detection modalities.
Reads every annotation of the ``source`` modality from this frame and
converts each one to the ``to`` modality by dispatching to the matching
per-object verb (``to_centroid``, ``to_bbox``, ``to_mask``, ``to_roi`` or
``to_pose``). Keyword arguments are forwarded unchanged to the per-object
verb (e.g. ``height``/``width`` for ``to="mask"``).
Args:
to: Target modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``.
source: Source modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``. Reads from the matching frame list
(``instances``, ``centroids``, ``bboxes``, ``masks`` or ``rois``).
inplace: If ``True``, append each produced annotation to this frame
(via `append`) in addition to returning them. If ``False``
(default), the frame is left unmodified.
**kwargs: Forwarded to the per-object conversion verb.
Returns:
A list of the produced annotations (one per source annotation), of the
``to`` modality.
Raises:
ValueError: If ``to`` or ``source`` is not a recognized modality, if
``to="pose"`` is requested from a non-centroid source (only
``centroid`` → ``pose`` is defined), or if a source annotation
lacks the target conversion verb.
"""
modalities = {
"pose": "instances",
"centroid": "centroids",
"bbox": "bboxes",
"mask": "masks",
"roi": "rois",
}
if to not in modalities:
raise ValueError(
f"Unknown target modality {to!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if source not in modalities:
raise ValueError(
f"Unknown source modality {source!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if to == "pose" and source != "centroid":
raise ValueError(
f"Conversion from {source!r} to 'pose' is not supported; only "
"'centroid' -> 'pose' is defined."
)
verb = "to_pose" if to == "pose" else f"to_{to}"
sources = getattr(self, modalities[source])
results = []
for obj in sources:
method = getattr(obj, verb, None)
if method is None:
raise ValueError(
f"Cannot convert {source!r} to {to!r}: "
f"{type(obj).__name__} has no {verb}() method."
)
result = method(**kwargs)
results.append(result)
if inplace:
self.append(result)
return results
def matches(self, other: "LabeledFrame", video_must_match: bool = True) -> bool:
"""Check if this frame matches another frame's identity.
Args:
other: Another LabeledFrame to compare with.
video_must_match: If True, frames must be from the same video.
If False, only frame index needs to match.
Returns:
True if the frames have the same identity, False otherwise.
Notes:
Frame identity is determined by video and frame index.
This does not compare the instances within the frame.
"""
if self.frame_idx != other.frame_idx:
return False
if video_must_match:
# Check if videos are the same object
if self.video is other.video:
return True
# Check if videos have matching paths
return self.video.matches_path(other.video, strict=False)
return True
def similarity_to(self, other: "LabeledFrame") -> dict[str, any]:
"""Calculate instance overlap metrics with another frame.
Args:
other: Another LabeledFrame to compare with.
Returns:
A dictionary with similarity metrics:
- 'n_user_self': Number of user instances in this frame
- 'n_user_other': Number of user instances in the other frame
- 'n_pred_self': Number of predicted instances in this frame
- 'n_pred_other': Number of predicted instances in the other frame
- 'n_overlapping': Number of instances that overlap (by IoU)
- 'mean_pose_distance': Mean distance between matching poses
"""
metrics = {
"n_user_self": len(self.user_instances),
"n_user_other": len(other.user_instances),
"n_pred_self": len(self.predicted_instances),
"n_pred_other": len(other.predicted_instances),
"n_overlapping": 0,
"mean_pose_distance": None,
}
# Count overlapping instances and compute pose distances
pose_distances = []
for inst1 in self.instances:
for inst2 in other.instances:
# Check if instances overlap
if inst1.overlaps_with(inst2, iou_threshold=0.1):
metrics["n_overlapping"] += 1
# If they have the same skeleton, compute pose distance
if inst1.skeleton.matches(inst2.skeleton):
# Get visible points for both
pts1 = inst1.numpy()
pts2 = inst2.numpy()
# Compute distances for visible points in both
valid = ~(np.isnan(pts1[:, 0]) | np.isnan(pts2[:, 0]))
if valid.any():
distances = np.linalg.norm(
pts1[valid] - pts2[valid], axis=1
)
pose_distances.extend(distances.tolist())
if pose_distances:
metrics["mean_pose_distance"] = np.mean(pose_distances)
return metrics
def merge(
self,
other: "LabeledFrame",
instance: "InstanceMatcher | None" = None,
frame: str = "auto",
) -> tuple[list[Instance], list[tuple[Instance, Instance, str]]]:
"""Merge instances from another frame into this frame.
Args:
other: Another LabeledFrame to merge instances from.
instance: Matcher to use for finding duplicate instances.
If None, uses default spatial matching with 5px tolerance.
frame: Merge strategy:
- "auto": Keep user labels, update predictions only if no user label
- "keep_original": Keep all original instances, ignore new ones
- "keep_new": Replace with new instances
- "keep_both": Keep all instances from both frames
- "update_tracks": Update track and score of the original instances
from the new instances.
- "replace_predictions": Keep all user instances from original frame,
remove all predictions from original frame, add only predictions
from the incoming frame. No spatial matching is performed.
Returns:
A tuple of (merged_instances, conflicts) where:
- merged_instances: List of instances after merging
- conflicts: List of (original, new, resolution) tuples for conflicts
Notes:
The merged instance list is returned (not assigned back) so the
caller can decide what to do with it. Frame-level annotations
(centroids, bboxes, masks, label images, rois) and the
``is_negative`` flag are updated on this frame in place.
"""
from sleap_io.model.matching import InstanceMatcher, InstanceMatchMethod
if instance is None:
instance_matcher = InstanceMatcher(
method=InstanceMatchMethod.SPATIAL, threshold=5.0
)
else:
instance_matcher = instance
conflicts = []
if frame == "keep_original":
self._merge_annotations(other, strategy="keep_original")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances.copy(), conflicts
elif frame == "keep_new":
self._merge_annotations(other, strategy="keep_new")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, other.instances
)
return other.instances.copy(), conflicts
elif frame == "keep_both":
self._merge_annotations(other, strategy="keep_both")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances + other.instances
)
return self.instances + other.instances, conflicts
elif frame == "update_tracks":
# match instances and update .track and tracking score of the old instances
matches = instance_matcher.find_matches(self.instances, other.instances)
for self_idx, other_idx, score in matches:
self.instances[self_idx].track = other.instances[other_idx].track
self.instances[self_idx].tracking_score = other.instances[
other_idx
].tracking_score
self._merge_annotations(
other,
strategy="update_tracks",
threshold=instance_matcher.threshold,
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances, conflicts
elif frame == "replace_predictions":
# Keep all user instances from original frame
merged = [inst for inst in self.instances if type(inst) is Instance]
# Add only predictions from incoming frame (not user instances)
merged.extend(
inst for inst in other.instances if type(inst) is PredictedInstance
)
self._merge_annotations(other, strategy="replace_predictions")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged
)
# No instance conflicts to report - this is a clean replacement
return merged, []
# Auto merging strategy
merged_instances = []
used_indices = set()
# First, keep all user instances from self
for inst in self.instances:
if type(inst) is Instance:
merged_instances.append(inst)
# Find matches between instances
matches = instance_matcher.find_matches(self.instances, other.instances)
# Group matches by instance in other frame
other_to_self = {}
for self_idx, other_idx, score in matches:
if other_idx not in other_to_self or score > other_to_self[other_idx][1]:
other_to_self[other_idx] = (self_idx, score)
# Process instances from other frame
for other_idx, other_inst in enumerate(other.instances):
if other_idx in other_to_self:
self_idx, score = other_to_self[other_idx]
self_inst = self.instances[self_idx]
# Check for conflicts
if type(self_inst) is Instance and type(other_inst) is Instance:
# Both are user instances - conflict
conflicts.append((self_inst, other_inst, "kept_original"))
used_indices.add(self_idx)
elif (
type(self_inst) is PredictedInstance
and type(other_inst) is Instance
):
# Replace prediction with user instance
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
elif (
type(self_inst) is Instance
and type(other_inst) is PredictedInstance
):
# Keep user instance, ignore prediction
conflicts.append((self_inst, other_inst, "kept_user"))
used_indices.add(self_idx)
else:
# Both are predictions - keep the new one
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
else:
# No match found, add new instance
merged_instances.append(other_inst)
# Add remaining instances from self that weren't matched
for self_idx, self_inst in enumerate(self.instances):
if type(self_inst) is PredictedInstance and self_idx not in used_indices:
# Check if this prediction should be kept
# NOTE: This defensive logic should be unreachable under normal
# circumstances since all matched instances should have been added to
# used_indices above. However, we keep this as a safety net for edge
# cases or future changes.
keep = True
for other_idx, (matched_self_idx, _) in other_to_self.items():
if matched_self_idx == self_idx:
keep = False
break
if keep:
merged_instances.append(self_inst)
# Merge annotations from the other frame (spatial matching + resolution)
self._merge_annotations(
other, strategy="auto", threshold=instance_matcher.threshold
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged_instances
)
return merged_instances, conflicts
def _merge_annotations(
self,
other: "LabeledFrame",
strategy: str = "keep_both",
threshold: float = 5.0,
):
"""Merge annotation lists from another frame into this frame.
Shallow-copies annotations from the other frame to avoid mutating the
source when references are later remapped. Video and track references
are preserved so that ``_remap_frame_annotations`` can find them in
the mapping dicts.
Args:
other: The frame to merge annotations from.
strategy: The merge strategy, matching the ``frame`` parameter of
``merge()``. Controls which annotations are kept:
- ``"keep_original"``: Keep self only.
- ``"keep_new"``: Replace with other's annotations.
- ``"keep_both"``: Keep self + add other's (default).
- ``"replace_predictions"``: Keep user from self, replace
predicted with other's predicted.
- ``"auto"``: Spatial matching + user-vs-predicted resolution
cascade (mirrors instance auto-merge logic).
- ``"update_tracks"``: Spatial matching, then update track
assignments on matched self annotations.
threshold: Maximum centroid distance (pixels) for spatial matching
in ``"auto"`` and ``"update_tracks"`` strategies.
"""
attrs = ("centroids", "bboxes", "masks", "label_images", "rois")
if strategy == "keep_original":
return
if strategy == "keep_new":
for attr in attrs:
memo: dict[int, Any] = {}
new_list = [
_copy_with_memo(item, memo) for item in getattr(other, attr)
]
_relink_from_predicted(new_list, memo)
setattr(self, attr, new_list)
return
if strategy == "replace_predictions":
for attr in attrs:
memo = {}
kept = [a for a in getattr(self, attr) if not a.is_predicted]
for item in getattr(other, attr):
if item.is_predicted:
kept.append(_copy_with_memo(item, memo))
_relink_from_predicted(kept, memo)
setattr(self, attr, kept)
return
if strategy == "auto":
for attr in attrs:
setattr(
self,
attr,
_resolve_annotation_auto(
getattr(self, attr), getattr(other, attr), attr, threshold
),
)
return
if strategy == "update_tracks":
for attr in attrs:
_resolve_annotation_update_tracks(
getattr(self, attr), getattr(other, attr), attr, threshold
)
return
# "keep_both" (default)
for attr in attrs:
memo = {}
target = getattr(self, attr)
existing_ids = set(id(x) for x in target)
for item in getattr(other, attr):
if id(item) not in existing_ids:
target.append(_copy_with_memo(item, memo))
_relink_from_predicted(target, memo)
__annotations__ = {'video': 'Video', 'frame_idx': 'int', 'instances': 'list[Instance | PredictedInstance]', 'is_negative': 'bool', 'centroids': "'list[Centroid]'", 'bboxes': "'list[BoundingBox]'", 'masks': "'list[SegmentationMask]'", 'label_images': "'list[LabelImage]'", 'rois': "'list[ROI]'"}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=True, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'Labeled data for a single frame of a video.\n\nAttributes:\n video: The `Video` associated with this `LabeledFrame`.\n frame_idx: The index of the `LabeledFrame` in the `Video`.\n instances: List of `Instance` objects associated with this `LabeledFrame`.\n is_negative: If True, this frame is explicitly marked as containing no\n instances (a "negative" or background frame for training). This is\n distinct from frames that are simply empty (e.g., instances were deleted).\n centroids: List of `Centroid` annotations for this frame.\n bboxes: List of `BoundingBox` annotations for this frame.\n masks: List of `SegmentationMask` annotations for this frame.\n label_images: List of `LabelImage` annotations for this frame.\n rois: List of `ROI` annotations for this frame.\n\nNotes:\n Instances of this class are hashed by identity, not by value. This means that\n two `LabeledFrame` instances with the same attributes will NOT be considered\n equal in a set or dict.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 329
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('video', 'frame_idx', 'instances', 'is_negative', 'centroids', 'bboxes', 'masks', 'label_images', 'rois')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.labeled_frame'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('video', 'frame_idx', 'instances', 'is_negative', 'centroids', 'bboxes', 'masks', 'label_images', 'rois', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('bboxes', 'centroids', 'instances', 'is_negative', 'label_images', 'masks', 'rois')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
has_predicted_instances
property
¶
Return True if the frame has any predicted instances.
has_user_instances
property
¶
Return True if the frame has any user-labeled instances.
image
property
¶
Return the image of the frame as a numpy array.
is_user_labeled
property
¶
Return True if frame has user instances/annotations OR is negative.
This property indicates whether the frame represents intentional user annotation, either through labeled instances, user annotations (centroids, bboxes, ROIs, masks, label images), or explicit marking as a negative/background frame.
predicted_instances
property
¶
Frame instances that are predicted by a model (PredictedInstance).
unused_predicted_masks
property
¶
Return predicted masks in this frame not yet adopted by a user mask.
A PredictedSegmentationMask is considered adopted (and so excluded
from the result) when some UserSegmentationMask in the same frame
either links to it via from_predicted (checked first) or, lacking an
explicit link, spatially overlaps it (bbox-centroid distance within 5 px,
the auto-merge default). This mirrors the link-first, spatial-fallback
precedence used by the auto-merge cascade and supports the
"retrain only what a human corrected" workflow.
This is the segmentation-mask analogue of unused_predictions (which
covers PredictedInstance objects).
Returns:
| Type | Description |
|---|---|
|
The |
unused_predictions
property
¶
Return a list of "unused" PredictedInstance objects in frame.
This is all of the PredictedInstance objects which do not have a corresponding
Instance in the same track in the same frame.
user_instances
property
¶
Frame instances that are user-labeled (Instance objects).
__getitem__(key)
¶
__init__(video, frame_idx, instances=NOTHING, is_negative=False, centroids=NOTHING, bboxes=NOTHING, masks=NOTHING, label_images=NOTHING, rois=NOTHING)
¶
Method generated by attrs for class LabeledFrame.
Source code in sleap_io/model/labeled_frame.py
from sleap_io.model.instance import Instance, PredictedInstance
from sleap_io.model.video import Video
if TYPE_CHECKING:
from sleap_io.model.bbox import BoundingBox
from sleap_io.model.centroid import Centroid
from sleap_io.model.label_image import LabelImage
from sleap_io.model.mask import SegmentationMask
from sleap_io.model.matching import InstanceMatcher
from sleap_io.model.roi import ROI
def _annotation_centroid_xy(annotation: Any, attr: str) -> tuple[float, float] | None:
"""Extract centroid (x, y) from an annotation based on its modality.
Args:
annotation: An annotation object (Centroid, BoundingBox, etc.).
attr: The attribute name indicating the modality.
Returns:
A tuple of (x, y) coordinates, or ``None`` if the centroid cannot be
computed (e.g., empty mask or empty ROI geometry).
"""
if attr == "centroids":
return (annotation.x, annotation.y)
elif attr == "bboxes":
return annotation.centroid_xy
elif attr == "rois":
if annotation.geometry.is_empty:
__iter__()
¶
__len__()
¶
__repr__()
¶
Method generated by attrs for class LabeledFrame.
Source code in sleap_io/model/labeled_frame.py
"""Data structures for data contained within a single video frame.
The `LabeledFrame` class is a data structure that contains `Instance`s and
`PredictedInstance`s that are associated with a single frame within a video.
"""
from __future__ import annotations
import math
from copy import copy
from typing import TYPE_CHECKING, Any
import numpy as np
from attrs import define, field
__setattr__(name, val)
¶
Method generated by attrs for class LabeledFrame.
append(annotation)
¶
Append an annotation to the appropriate frame-level container.
Routes the annotation to the correct list based on its type:
Instance/PredictedInstance → instances,
Centroid → centroids, BoundingBox → bboxes,
SegmentationMask → masks, LabelImage → label_images,
ROI → rois.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
annotation
|
Instance | PredictedInstance | Centroid | BoundingBox | SegmentationMask | LabelImage | ROI
|
The annotation object to add. |
required |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the annotation type is not recognized. |
Source code in sleap_io/model/labeled_frame.py
def append(
self,
annotation: (
"Instance | PredictedInstance | Centroid"
" | BoundingBox | SegmentationMask | LabelImage | ROI"
),
) -> None:
"""Append an annotation to the appropriate frame-level container.
Routes the annotation to the correct list based on its type:
``Instance``/``PredictedInstance`` → ``instances``,
``Centroid`` → ``centroids``, ``BoundingBox`` → ``bboxes``,
``SegmentationMask`` → ``masks``, ``LabelImage`` → ``label_images``,
``ROI`` → ``rois``.
Args:
annotation: The annotation object to add.
Raises:
TypeError: If the annotation type is not recognized.
"""
from sleap_io.model.bbox import BoundingBox
from sleap_io.model.centroid import Centroid
from sleap_io.model.label_image import LabelImage
from sleap_io.model.mask import SegmentationMask
from sleap_io.model.roi import ROI
if isinstance(annotation, (Instance, PredictedInstance)):
self.instances.append(annotation)
elif isinstance(annotation, Centroid):
self.centroids.append(annotation)
elif isinstance(annotation, BoundingBox):
self.bboxes.append(annotation)
elif isinstance(annotation, SegmentationMask):
self.masks.append(annotation)
elif isinstance(annotation, LabelImage):
self.label_images.append(annotation)
elif isinstance(annotation, ROI):
self.rois.append(annotation)
else:
raise TypeError(
f"Cannot append {type(annotation).__name__} to LabeledFrame. "
f"Expected one of: Instance, PredictedInstance, Centroid, "
f"BoundingBox, SegmentationMask, LabelImage, ROI."
)
convert(to, source='pose', inplace=False, **kwargs)
¶
Convert annotations between detection modalities.
Reads every annotation of the source modality from this frame and
converts each one to the to modality by dispatching to the matching
per-object verb (to_centroid, to_bbox, to_mask, to_roi or
to_pose). Keyword arguments are forwarded unchanged to the per-object
verb (e.g. height/width for to="mask").
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
to
|
str
|
Target modality, one of |
required |
source
|
str
|
Source modality, one of |
'pose'
|
inplace
|
bool
|
If |
False
|
**kwargs
|
Forwarded to the per-object conversion verb. |
required |
Returns:
| Type | Description |
|---|---|
list
|
A list of the produced annotations (one per source annotation), of the
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in sleap_io/model/labeled_frame.py
def convert(
self,
to: str,
source: str = "pose",
inplace: bool = False,
**kwargs,
) -> list:
"""Convert annotations between detection modalities.
Reads every annotation of the ``source`` modality from this frame and
converts each one to the ``to`` modality by dispatching to the matching
per-object verb (``to_centroid``, ``to_bbox``, ``to_mask``, ``to_roi`` or
``to_pose``). Keyword arguments are forwarded unchanged to the per-object
verb (e.g. ``height``/``width`` for ``to="mask"``).
Args:
to: Target modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``.
source: Source modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``. Reads from the matching frame list
(``instances``, ``centroids``, ``bboxes``, ``masks`` or ``rois``).
inplace: If ``True``, append each produced annotation to this frame
(via `append`) in addition to returning them. If ``False``
(default), the frame is left unmodified.
**kwargs: Forwarded to the per-object conversion verb.
Returns:
A list of the produced annotations (one per source annotation), of the
``to`` modality.
Raises:
ValueError: If ``to`` or ``source`` is not a recognized modality, if
``to="pose"`` is requested from a non-centroid source (only
``centroid`` → ``pose`` is defined), or if a source annotation
lacks the target conversion verb.
"""
modalities = {
"pose": "instances",
"centroid": "centroids",
"bbox": "bboxes",
"mask": "masks",
"roi": "rois",
}
if to not in modalities:
raise ValueError(
f"Unknown target modality {to!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if source not in modalities:
raise ValueError(
f"Unknown source modality {source!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if to == "pose" and source != "centroid":
raise ValueError(
f"Conversion from {source!r} to 'pose' is not supported; only "
"'centroid' -> 'pose' is defined."
)
verb = "to_pose" if to == "pose" else f"to_{to}"
sources = getattr(self, modalities[source])
results = []
for obj in sources:
method = getattr(obj, verb, None)
if method is None:
raise ValueError(
f"Cannot convert {source!r} to {to!r}: "
f"{type(obj).__name__} has no {verb}() method."
)
result = method(**kwargs)
results.append(result)
if inplace:
self.append(result)
return results
matches(other, video_must_match=True)
¶
Check if this frame matches another frame's identity.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
LabeledFrame
|
Another LabeledFrame to compare with. |
required |
video_must_match
|
bool
|
If True, frames must be from the same video. If False, only frame index needs to match. |
True
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the frames have the same identity, False otherwise. |
Notes
Frame identity is determined by video and frame index. This does not compare the instances within the frame.
Source code in sleap_io/model/labeled_frame.py
def matches(self, other: "LabeledFrame", video_must_match: bool = True) -> bool:
"""Check if this frame matches another frame's identity.
Args:
other: Another LabeledFrame to compare with.
video_must_match: If True, frames must be from the same video.
If False, only frame index needs to match.
Returns:
True if the frames have the same identity, False otherwise.
Notes:
Frame identity is determined by video and frame index.
This does not compare the instances within the frame.
"""
if self.frame_idx != other.frame_idx:
return False
if video_must_match:
# Check if videos are the same object
if self.video is other.video:
return True
# Check if videos have matching paths
return self.video.matches_path(other.video, strict=False)
return True
merge(other, instance=None, frame='auto')
¶
Merge instances from another frame into this frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
LabeledFrame
|
Another LabeledFrame to merge instances from. |
required |
instance
|
InstanceMatcher | None
|
Matcher to use for finding duplicate instances. If None, uses default spatial matching with 5px tolerance. |
None
|
frame
|
str
|
Merge strategy: - "auto": Keep user labels, update predictions only if no user label - "keep_original": Keep all original instances, ignore new ones - "keep_new": Replace with new instances - "keep_both": Keep all instances from both frames - "update_tracks": Update track and score of the original instances from the new instances. - "replace_predictions": Keep all user instances from original frame, remove all predictions from original frame, add only predictions from the incoming frame. No spatial matching is performed. |
'auto'
|
Returns:
| Type | Description |
|---|---|
tuple[list[Instance], list[tuple[Instance, Instance, str]]]
|
A tuple of (merged_instances, conflicts) where: - merged_instances: List of instances after merging - conflicts: List of (original, new, resolution) tuples for conflicts |
Notes
The merged instance list is returned (not assigned back) so the
caller can decide what to do with it. Frame-level annotations
(centroids, bboxes, masks, label images, rois) and the
is_negative flag are updated on this frame in place.
Source code in sleap_io/model/labeled_frame.py
def merge(
self,
other: "LabeledFrame",
instance: "InstanceMatcher | None" = None,
frame: str = "auto",
) -> tuple[list[Instance], list[tuple[Instance, Instance, str]]]:
"""Merge instances from another frame into this frame.
Args:
other: Another LabeledFrame to merge instances from.
instance: Matcher to use for finding duplicate instances.
If None, uses default spatial matching with 5px tolerance.
frame: Merge strategy:
- "auto": Keep user labels, update predictions only if no user label
- "keep_original": Keep all original instances, ignore new ones
- "keep_new": Replace with new instances
- "keep_both": Keep all instances from both frames
- "update_tracks": Update track and score of the original instances
from the new instances.
- "replace_predictions": Keep all user instances from original frame,
remove all predictions from original frame, add only predictions
from the incoming frame. No spatial matching is performed.
Returns:
A tuple of (merged_instances, conflicts) where:
- merged_instances: List of instances after merging
- conflicts: List of (original, new, resolution) tuples for conflicts
Notes:
The merged instance list is returned (not assigned back) so the
caller can decide what to do with it. Frame-level annotations
(centroids, bboxes, masks, label images, rois) and the
``is_negative`` flag are updated on this frame in place.
"""
from sleap_io.model.matching import InstanceMatcher, InstanceMatchMethod
if instance is None:
instance_matcher = InstanceMatcher(
method=InstanceMatchMethod.SPATIAL, threshold=5.0
)
else:
instance_matcher = instance
conflicts = []
if frame == "keep_original":
self._merge_annotations(other, strategy="keep_original")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances.copy(), conflicts
elif frame == "keep_new":
self._merge_annotations(other, strategy="keep_new")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, other.instances
)
return other.instances.copy(), conflicts
elif frame == "keep_both":
self._merge_annotations(other, strategy="keep_both")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances + other.instances
)
return self.instances + other.instances, conflicts
elif frame == "update_tracks":
# match instances and update .track and tracking score of the old instances
matches = instance_matcher.find_matches(self.instances, other.instances)
for self_idx, other_idx, score in matches:
self.instances[self_idx].track = other.instances[other_idx].track
self.instances[self_idx].tracking_score = other.instances[
other_idx
].tracking_score
self._merge_annotations(
other,
strategy="update_tracks",
threshold=instance_matcher.threshold,
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances, conflicts
elif frame == "replace_predictions":
# Keep all user instances from original frame
merged = [inst for inst in self.instances if type(inst) is Instance]
# Add only predictions from incoming frame (not user instances)
merged.extend(
inst for inst in other.instances if type(inst) is PredictedInstance
)
self._merge_annotations(other, strategy="replace_predictions")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged
)
# No instance conflicts to report - this is a clean replacement
return merged, []
# Auto merging strategy
merged_instances = []
used_indices = set()
# First, keep all user instances from self
for inst in self.instances:
if type(inst) is Instance:
merged_instances.append(inst)
# Find matches between instances
matches = instance_matcher.find_matches(self.instances, other.instances)
# Group matches by instance in other frame
other_to_self = {}
for self_idx, other_idx, score in matches:
if other_idx not in other_to_self or score > other_to_self[other_idx][1]:
other_to_self[other_idx] = (self_idx, score)
# Process instances from other frame
for other_idx, other_inst in enumerate(other.instances):
if other_idx in other_to_self:
self_idx, score = other_to_self[other_idx]
self_inst = self.instances[self_idx]
# Check for conflicts
if type(self_inst) is Instance and type(other_inst) is Instance:
# Both are user instances - conflict
conflicts.append((self_inst, other_inst, "kept_original"))
used_indices.add(self_idx)
elif (
type(self_inst) is PredictedInstance
and type(other_inst) is Instance
):
# Replace prediction with user instance
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
elif (
type(self_inst) is Instance
and type(other_inst) is PredictedInstance
):
# Keep user instance, ignore prediction
conflicts.append((self_inst, other_inst, "kept_user"))
used_indices.add(self_idx)
else:
# Both are predictions - keep the new one
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
else:
# No match found, add new instance
merged_instances.append(other_inst)
# Add remaining instances from self that weren't matched
for self_idx, self_inst in enumerate(self.instances):
if type(self_inst) is PredictedInstance and self_idx not in used_indices:
# Check if this prediction should be kept
# NOTE: This defensive logic should be unreachable under normal
# circumstances since all matched instances should have been added to
# used_indices above. However, we keep this as a safety net for edge
# cases or future changes.
keep = True
for other_idx, (matched_self_idx, _) in other_to_self.items():
if matched_self_idx == self_idx:
keep = False
break
if keep:
merged_instances.append(self_inst)
# Merge annotations from the other frame (spatial matching + resolution)
self._merge_annotations(
other, strategy="auto", threshold=instance_matcher.threshold
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged_instances
)
return merged_instances, conflicts
numpy()
¶
Return all instances in the frame as a numpy array.
Returns:
| Type | Description |
|---|---|
ndarray
|
Points as a numpy array of shape Note that the order of the instances is arbitrary. |
Source code in sleap_io/model/labeled_frame.py
def numpy(self) -> np.ndarray:
"""Return all instances in the frame as a numpy array.
Returns:
Points as a numpy array of shape `(n_instances, n_nodes, 2)`.
Note that the order of the instances is arbitrary.
"""
n_instances = len(self.instances)
n_nodes = len(self.instances[0]) if n_instances > 0 else 0
pts = np.full((n_instances, n_nodes, 2), np.nan)
for i, inst in enumerate(self.instances):
pts[i] = inst.numpy()[:, 0:2]
return pts
remove_empty_instances()
¶
remove_predictions()
¶
Remove all predicted instances and annotations from the frame.
Source code in sleap_io/model/labeled_frame.py
def remove_predictions(self):
"""Remove all predicted instances and annotations from the frame."""
from sleap_io.model.bbox import PredictedBoundingBox
from sleap_io.model.centroid import PredictedCentroid
from sleap_io.model.label_image import PredictedLabelImage
from sleap_io.model.mask import PredictedSegmentationMask
from sleap_io.model.roi import PredictedROI
self.instances = [inst for inst in self.instances if type(inst) is Instance]
self.centroids = [
c for c in self.centroids if not isinstance(c, PredictedCentroid)
]
self.bboxes = [
b for b in self.bboxes if not isinstance(b, PredictedBoundingBox)
]
self.masks = [
m for m in self.masks if not isinstance(m, PredictedSegmentationMask)
]
self.label_images = [
li for li in self.label_images if not isinstance(li, PredictedLabelImage)
]
self.rois = [r for r in self.rois if not isinstance(r, PredictedROI)]
similarity_to(other)
¶
Calculate instance overlap metrics with another frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
LabeledFrame
|
Another LabeledFrame to compare with. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, any]
|
A dictionary with similarity metrics: - 'n_user_self': Number of user instances in this frame - 'n_user_other': Number of user instances in the other frame - 'n_pred_self': Number of predicted instances in this frame - 'n_pred_other': Number of predicted instances in the other frame - 'n_overlapping': Number of instances that overlap (by IoU) - 'mean_pose_distance': Mean distance between matching poses |
Source code in sleap_io/model/labeled_frame.py
def similarity_to(self, other: "LabeledFrame") -> dict[str, any]:
"""Calculate instance overlap metrics with another frame.
Args:
other: Another LabeledFrame to compare with.
Returns:
A dictionary with similarity metrics:
- 'n_user_self': Number of user instances in this frame
- 'n_user_other': Number of user instances in the other frame
- 'n_pred_self': Number of predicted instances in this frame
- 'n_pred_other': Number of predicted instances in the other frame
- 'n_overlapping': Number of instances that overlap (by IoU)
- 'mean_pose_distance': Mean distance between matching poses
"""
metrics = {
"n_user_self": len(self.user_instances),
"n_user_other": len(other.user_instances),
"n_pred_self": len(self.predicted_instances),
"n_pred_other": len(other.predicted_instances),
"n_overlapping": 0,
"mean_pose_distance": None,
}
# Count overlapping instances and compute pose distances
pose_distances = []
for inst1 in self.instances:
for inst2 in other.instances:
# Check if instances overlap
if inst1.overlaps_with(inst2, iou_threshold=0.1):
metrics["n_overlapping"] += 1
# If they have the same skeleton, compute pose distance
if inst1.skeleton.matches(inst2.skeleton):
# Get visible points for both
pts1 = inst1.numpy()
pts2 = inst2.numpy()
# Compute distances for visible points in both
valid = ~(np.isnan(pts1[:, 0]) | np.isnan(pts2[:, 0]))
if valid.any():
distances = np.linalg.norm(
pts1[valid] - pts2[valid], axis=1
)
pose_distances.extend(distances.tolist())
if pose_distances:
metrics["mean_pose_distance"] = np.mean(pose_distances)
return metrics
PredictedInstance
¶
Bases: sleap_io.model.instance.Instance
A PredictedInstance is an Instance that was predicted using a model.
Attributes:
| Name | Type | Description |
|---|---|---|
skeleton |
The |
|
points |
A dictionary where keys are |
|
track |
An optional |
|
from_predicted |
Not applicable in |
|
score |
The instance detection or part grouping prediction score. This is a scalar that represents the confidence with which this entire instance was predicted. This may not always be applicable depending on the model type. |
|
tracking_score |
The score associated with the |
|
identity |
An optional global |
|
identity_score |
The score associated with the |
|
identity_embedding |
An optional re-ID |
|
category |
An optional |
|
category_score |
The score associated with the |
|
category_embedding |
An optional classification |
Methods:
| Name | Description |
|---|---|
__getitem__ |
Return the point associated with a node. |
__init__ |
Method generated by attrs for class PredictedInstance. |
__repr__ |
Return a readable representation of the instance. |
__setattr__ |
Method generated by attrs for class PredictedInstance. |
__setitem__ |
Set the point associated with a node. |
empty |
Create an empty instance with no points. |
from_numpy |
Create a predicted instance object from a numpy array. |
numpy |
Return the instance points as a |
replace_skeleton |
Replace the skeleton associated with the instance. |
update_skeleton |
Update or replace the skeleton associated with the instance. |
Source code in sleap_io/model/instance.py
@attrs.define(eq=False)
class PredictedInstance(Instance):
"""A `PredictedInstance` is an `Instance` that was predicted using a model.
Attributes:
skeleton: The `Skeleton` that this `Instance` is associated with.
points: A dictionary where keys are `Skeleton` nodes and values are `Point`s.
track: An optional `Track` associated with a unique animal/object across frames
or videos.
from_predicted: Not applicable in `PredictedInstance`s (must be set to `None`).
score: The instance detection or part grouping prediction score. This is a
scalar that represents the confidence with which this entire instance was
predicted. This may not always be applicable depending on the model type.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity assignment.
identity: An optional global `Identity` (see `Instance.identity`).
identity_score: The score associated with the `identity` assignment (see
`Instance.identity_score`).
identity_embedding: An optional re-ID `Embedding` (see
`Instance.identity_embedding`).
category: An optional `Category` (class) (see `Instance.category`).
category_score: The score associated with the `category` assignment (see
`Instance.category_score`).
category_embedding: An optional classification `Embedding` (see
`Instance.category_embedding`).
"""
points: PredictedPointsArray = attrs.field(eq=attrs.cmp_using(eq=np.array_equal))
skeleton: Skeleton
score: float = 0.0
track: Track | None = None
tracking_score: float | None = 0
identity: Identity | None = None
identity_score: float | None = None
category: Category | None = attrs.field(default=None, converter=to_category)
category_score: float | None = None
from_predicted: "PredictedInstance | None" = None
identity_embedding: Embedding | None = attrs.field(default=None, repr=False)
category_embedding: Embedding | None = attrs.field(default=None, repr=False)
def __repr__(self) -> str:
"""Return a readable representation of the instance."""
pts = self.numpy().tolist()
track = f'"{self.track.name}"' if self.track is not None else self.track
score = str(self.score) if self.score is None else f"{self.score:.2f}"
tracking_score = (
str(self.tracking_score)
if self.tracking_score is None
else f"{self.tracking_score:.2f}"
)
return (
f"PredictedInstance(points={pts}, track={track}, "
f"score={score}, tracking_score={tracking_score})"
)
@classmethod
def empty(
cls,
skeleton: Skeleton,
score: float = 0.0,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "PredictedInstance":
"""Create an empty instance with no points."""
points = PredictedPointsArray.empty(len(skeleton))
points["name"] = skeleton.node_names
return cls(
points=points,
skeleton=skeleton,
score=score,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
@classmethod
def _convert_points(
cls, points_data: np.ndarray | dict | list, skeleton: Skeleton
) -> PredictedPointsArray:
"""Convert points to a structured numpy array if needed."""
if isinstance(points_data, dict):
return PredictedPointsArray.from_dict(points_data, skeleton)
elif isinstance(points_data, (list, np.ndarray)):
if isinstance(points_data, list):
points_data = np.array(points_data)
points = PredictedPointsArray.from_array(points_data)
points["name"] = skeleton.node_names
return points
else:
raise ValueError("points must be a numpy array or dictionary.")
@classmethod
def from_numpy(
cls,
points_data: np.ndarray,
skeleton: Skeleton,
point_scores: np.ndarray | None = None,
score: float = 0.0,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "PredictedInstance":
"""Create a predicted instance object from a numpy array."""
points = cls._convert_points(points_data, skeleton)
if point_scores is not None:
points["score"] = point_scores
return cls(
points=points,
skeleton=skeleton,
score=score,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
def numpy(
self,
invisible_as_nan: bool = True,
scores: bool = False,
) -> np.ndarray:
"""Return the instance points as a `(n_nodes, 2)` numpy array.
Args:
invisible_as_nan: If `True` (the default), points that are not visible will
be set to `np.nan`. If `False`, they will be whatever the stored value
of `PredictedInstance.points["xy"]` is.
scores: If `True`, the score associated with each point will be
included in the output.
Returns:
A numpy array of shape `(n_nodes, 2)` corresponding to the points of the
skeleton. Values of `np.nan` indicate "missing" nodes.
If `scores` is `True`, the array will have shape `(n_nodes, 3)` with the
third column containing the score associated with each point.
Notes:
This will always return a copy of the array.
If you need to avoid making a copy, just access the
`PredictedInstance.points["xy"]` attribute directly. This will not replace
invisible points with `np.nan`.
"""
if invisible_as_nan:
pts = np.where(
self.points["visible"].reshape(-1, 1), self.points["xy"], np.nan
)
else:
pts = self.points["xy"].copy()
if scores:
return np.column_stack((pts, self.points["score"]))
else:
return pts
def update_skeleton(self, names_only: bool = False):
"""Update or replace the skeleton associated with the instance.
Args:
names_only: If `True`, only update the node names in the points array. If
`False`, the points array will be updated to match the new skeleton.
"""
if names_only:
# Update the node names.
self.points["name"] = self.skeleton.node_names
return
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(self.points["name"])
# Update the points.
new_points = PredictedPointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
new_points["name"] = self.skeleton.node_names
self.points = new_points
def replace_skeleton(
self,
new_skeleton: Skeleton,
node_names_map: dict[str, str] | None = None,
):
"""Replace the skeleton associated with the instance.
Args:
new_skeleton: The new `Skeleton` to associate with the instance.
node_names_map: Dictionary mapping nodes in the old skeleton to nodes in the
new skeleton. Keys and values should be specified as lists of strings.
If not provided, only nodes with identical names will be mapped. Points
associated with unmapped nodes will be removed.
Notes:
This method will update the `PredictedInstance.skeleton` attribute and the
`PredictedInstance.points` attribute in place (a copy is made of the points
array).
It is recommended to use `Labels.replace_skeleton` instead of this method if
more flexible node mapping is required.
"""
# Update skeleton object.
self.skeleton = new_skeleton
# Get node names with replacements from node map if possible.
old_node_names = self.points["name"].tolist()
if node_names_map is not None:
old_node_names = [node_names_map.get(node, node) for node in old_node_names]
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(old_node_names)
# Update the points.
new_points = PredictedPointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
self.points = new_points
self.points["name"] = self.skeleton.node_names
def __getitem__(self, node: int | str | Node) -> np.ndarray:
"""Return the point associated with a node."""
# Inherit from Instance.__getitem__
return super().__getitem__(node)
def __setitem__(self, node: int | str | Node, value):
"""Set the point associated with a node.
Args:
node: The node to set the point for. Can be an integer index, string name,
or Node object.
value: A tuple or array-like of length 2 or 3 containing (x, y) coordinates
and optionally a confidence score. If the score is not provided, it
defaults to 1.0.
Notes:
This sets the point coordinates, score, and marks the point as visible.
"""
if type(node) is not int:
node = self.skeleton.index(node)
if len(value) < 2:
raise ValueError("Value must have at least 2 elements (x, y)")
self.points[node]["xy"] = value[:2]
# Set score if provided, otherwise default to 1.0
if len(value) >= 3:
self.points[node]["score"] = value[2]
else:
self.points[node]["score"] = 1.0
self.points[node]["visible"] = True
__annotations__ = {'points': 'PredictedPointsArray', 'skeleton': 'Skeleton', 'score': 'float', 'track': 'Track | None', 'tracking_score': 'float | None', 'identity': 'Identity | None', 'identity_score': 'float | None', 'category': 'Category | None', 'category_score': 'float | None', 'from_predicted': "'PredictedInstance | None'", 'identity_embedding': 'Embedding | None', 'category_embedding': 'Embedding | None'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'A `PredictedInstance` is an `Instance` that was predicted using a model.\n\nAttributes:\n skeleton: The `Skeleton` that this `Instance` is associated with.\n points: A dictionary where keys are `Skeleton` nodes and values are `Point`s.\n track: An optional `Track` associated with a unique animal/object across frames\n or videos.\n from_predicted: Not applicable in `PredictedInstance`s (must be set to `None`).\n score: The instance detection or part grouping prediction score. This is a\n scalar that represents the confidence with which this entire instance was\n predicted. This may not always be applicable depending on the model type.\n tracking_score: The score associated with the `Track` assignment. This is\n typically the value from the score matrix used in an identity assignment.\n identity: An optional global `Identity` (see `Instance.identity`).\n identity_score: The score associated with the `identity` assignment (see\n `Instance.identity_score`).\n identity_embedding: An optional re-ID `Embedding` (see\n `Instance.identity_embedding`).\n category: An optional `Category` (class) (see `Instance.category`).\n category_score: The score associated with the `category` assignment (see\n `Instance.category_score`).\n category_embedding: An optional classification `Embedding` (see\n `Instance.category_embedding`).\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 1218
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('points', 'skeleton', 'score', 'track', 'tracking_score', 'identity', 'identity_score', 'category', 'category_score', 'from_predicted', 'identity_embedding', 'category_embedding')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.instance'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('score',)
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('points', 'skeleton')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__getitem__(node)
¶
__init__(points, skeleton, score=0.0, track=None, tracking_score=0, identity=None, identity_score=None, category=None, category_score=None, from_predicted=None, identity_embedding=None, category_embedding=None)
¶
Method generated by attrs for class PredictedInstance.
Source code in sleap_io/model/instance.py
"""Data structures for data associated with a single instance such as an animal.
The `Instance` class is a SLEAP data structure that contains a collection of points that
correspond to landmarks within a `Skeleton`.
`PredictedInstance` additionally contains metadata associated with how the instance was
estimated, such as confidence scores.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import attrs
import numpy as np
__repr__()
¶
Return a readable representation of the instance.
Source code in sleap_io/model/instance.py
def __repr__(self) -> str:
"""Return a readable representation of the instance."""
pts = self.numpy().tolist()
track = f'"{self.track.name}"' if self.track is not None else self.track
score = str(self.score) if self.score is None else f"{self.score:.2f}"
tracking_score = (
str(self.tracking_score)
if self.tracking_score is None
else f"{self.tracking_score:.2f}"
)
return (
f"PredictedInstance(points={pts}, track={track}, "
f"score={score}, tracking_score={tracking_score})"
)
__setattr__(name, val)
¶
Method generated by attrs for class PredictedInstance.
Source code in sleap_io/model/instance.py
__setitem__(node, value)
¶
Set the point associated with a node.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
int | str | Node
|
The node to set the point for. Can be an integer index, string name, or Node object. |
required |
value
|
A tuple or array-like of length 2 or 3 containing (x, y) coordinates and optionally a confidence score. If the score is not provided, it defaults to 1.0. |
required |
Notes
This sets the point coordinates, score, and marks the point as visible.
Source code in sleap_io/model/instance.py
def __setitem__(self, node: int | str | Node, value):
"""Set the point associated with a node.
Args:
node: The node to set the point for. Can be an integer index, string name,
or Node object.
value: A tuple or array-like of length 2 or 3 containing (x, y) coordinates
and optionally a confidence score. If the score is not provided, it
defaults to 1.0.
Notes:
This sets the point coordinates, score, and marks the point as visible.
"""
if type(node) is not int:
node = self.skeleton.index(node)
if len(value) < 2:
raise ValueError("Value must have at least 2 elements (x, y)")
self.points[node]["xy"] = value[:2]
# Set score if provided, otherwise default to 1.0
if len(value) >= 3:
self.points[node]["score"] = value[2]
else:
self.points[node]["score"] = 1.0
self.points[node]["visible"] = True
empty(skeleton, score=0.0, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, identity_embedding=None, category_embedding=None, from_predicted=None)
classmethod
¶
Create an empty instance with no points.
Source code in sleap_io/model/instance.py
@classmethod
def empty(
cls,
skeleton: Skeleton,
score: float = 0.0,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "PredictedInstance":
"""Create an empty instance with no points."""
points = PredictedPointsArray.empty(len(skeleton))
points["name"] = skeleton.node_names
return cls(
points=points,
skeleton=skeleton,
score=score,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
from_numpy(points_data, skeleton, point_scores=None, score=0.0, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, identity_embedding=None, category_embedding=None, from_predicted=None)
classmethod
¶
Create a predicted instance object from a numpy array.
Source code in sleap_io/model/instance.py
@classmethod
def from_numpy(
cls,
points_data: np.ndarray,
skeleton: Skeleton,
point_scores: np.ndarray | None = None,
score: float = 0.0,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "PredictedInstance":
"""Create a predicted instance object from a numpy array."""
points = cls._convert_points(points_data, skeleton)
if point_scores is not None:
points["score"] = point_scores
return cls(
points=points,
skeleton=skeleton,
score=score,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
numpy(invisible_as_nan=True, scores=False)
¶
Return the instance points as a (n_nodes, 2) numpy array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
invisible_as_nan
|
bool
|
If |
True
|
scores
|
bool
|
If |
False
|
Returns:
| Type | Description |
|---|---|
ndarray
|
A numpy array of shape If |
Notes
This will always return a copy of the array.
If you need to avoid making a copy, just access the
PredictedInstance.points["xy"] attribute directly. This will not replace
invisible points with np.nan.
Source code in sleap_io/model/instance.py
def numpy(
self,
invisible_as_nan: bool = True,
scores: bool = False,
) -> np.ndarray:
"""Return the instance points as a `(n_nodes, 2)` numpy array.
Args:
invisible_as_nan: If `True` (the default), points that are not visible will
be set to `np.nan`. If `False`, they will be whatever the stored value
of `PredictedInstance.points["xy"]` is.
scores: If `True`, the score associated with each point will be
included in the output.
Returns:
A numpy array of shape `(n_nodes, 2)` corresponding to the points of the
skeleton. Values of `np.nan` indicate "missing" nodes.
If `scores` is `True`, the array will have shape `(n_nodes, 3)` with the
third column containing the score associated with each point.
Notes:
This will always return a copy of the array.
If you need to avoid making a copy, just access the
`PredictedInstance.points["xy"]` attribute directly. This will not replace
invisible points with `np.nan`.
"""
if invisible_as_nan:
pts = np.where(
self.points["visible"].reshape(-1, 1), self.points["xy"], np.nan
)
else:
pts = self.points["xy"].copy()
if scores:
return np.column_stack((pts, self.points["score"]))
else:
return pts
replace_skeleton(new_skeleton, node_names_map=None)
¶
Replace the skeleton associated with the instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_skeleton
|
Skeleton
|
The new |
required |
node_names_map
|
dict[str, str] | None
|
Dictionary mapping nodes in the old skeleton to nodes in the new skeleton. Keys and values should be specified as lists of strings. If not provided, only nodes with identical names will be mapped. Points associated with unmapped nodes will be removed. |
None
|
Notes
This method will update the PredictedInstance.skeleton attribute and the
PredictedInstance.points attribute in place (a copy is made of the points
array).
It is recommended to use Labels.replace_skeleton instead of this method if
more flexible node mapping is required.
Source code in sleap_io/model/instance.py
def replace_skeleton(
self,
new_skeleton: Skeleton,
node_names_map: dict[str, str] | None = None,
):
"""Replace the skeleton associated with the instance.
Args:
new_skeleton: The new `Skeleton` to associate with the instance.
node_names_map: Dictionary mapping nodes in the old skeleton to nodes in the
new skeleton. Keys and values should be specified as lists of strings.
If not provided, only nodes with identical names will be mapped. Points
associated with unmapped nodes will be removed.
Notes:
This method will update the `PredictedInstance.skeleton` attribute and the
`PredictedInstance.points` attribute in place (a copy is made of the points
array).
It is recommended to use `Labels.replace_skeleton` instead of this method if
more flexible node mapping is required.
"""
# Update skeleton object.
self.skeleton = new_skeleton
# Get node names with replacements from node map if possible.
old_node_names = self.points["name"].tolist()
if node_names_map is not None:
old_node_names = [node_names_map.get(node, node) for node in old_node_names]
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(old_node_names)
# Update the points.
new_points = PredictedPointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
self.points = new_points
self.points["name"] = self.skeleton.node_names
update_skeleton(names_only=False)
¶
Update or replace the skeleton associated with the instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
names_only
|
bool
|
If |
False
|
Source code in sleap_io/model/instance.py
def update_skeleton(self, names_only: bool = False):
"""Update or replace the skeleton associated with the instance.
Args:
names_only: If `True`, only update the node names in the points array. If
`False`, the points array will be updated to match the new skeleton.
"""
if names_only:
# Update the node names.
self.points["name"] = self.skeleton.node_names
return
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(self.points["name"])
# Update the points.
new_points = PredictedPointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
new_points["name"] = self.skeleton.node_names
self.points = new_points
Video
¶
Video class used by sleap to represent videos and data associated with them.
This class is used to store information regarding a video and its components.
It is used to store the video's filename, shape, and the video's backend.
To create a Video object, use the from_filename method which will select the
backend appropriately.
Attributes:
| Name | Type | Description |
|---|---|---|
filename |
The filename(s) of the video. Supported extensions: "mp4", "avi", "mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif", "tiff", "bmp", "seq". If the filename is a list, a list of image filenames are expected. If filename is a folder, it will be searched for images. |
|
backend |
An object that implements the basic methods for reading and manipulating frames of a specific video type. |
|
backend_metadata |
A dictionary of metadata specific to the backend. This is useful for storing metadata that requires an open backend (e.g., shape information) without having access to the video file itself. |
|
source_video |
The source video object if this is a proxy video. This is present when the video contains an embedded subset of frames from another video. |
|
open_backend |
Whether to open the backend when the video is available. If |
|
_exists_cache |
Per-instance TTL cache for the result of |
Notes
Instances of this class are hashed by identity, not by value. This means that
two Video instances with the same attributes will NOT be considered equal in a
set or dict.
Media Video Plugin Support
For media files (mp4, avi, etc.), the following plugins are supported: - "opencv": Uses OpenCV (cv2) for video reading - "FFMPEG": Uses imageio-ffmpeg for video reading - "pyav": Uses PyAV for video reading
Plugin aliases (case-insensitive): - opencv: "opencv", "cv", "cv2", "ocv" - FFMPEG: "FFMPEG", "ffmpeg", "imageio-ffmpeg", "imageio_ffmpeg" - pyav: "pyav", "av"
Plugin selection priority: 1. Explicitly specified plugin parameter 2. Backend metadata plugin value 3. Global default (set via sio.set_default_video_plugin) 4. Auto-detection based on available packages
See Also
VideoBackend: The backend interface for reading video data. sleap_io.set_default_video_plugin: Set global default plugin. sleap_io.get_default_video_plugin: Get current default plugin.
Methods:
| Name | Description |
|---|---|
__attrs_post_init__ |
Post init syntactic sugar. |
__deepcopy__ |
Deep copy the video object. |
__getitem__ |
Return the frames of the video at the given indices. |
__init__ |
Method generated by attrs for class Video. |
__len__ |
Return the length of the video as the number of frames. |
__repr__ |
Informal string representation (for print or format). |
__str__ |
Informal string representation (for print or format). |
apply_crop |
Bake this video's virtual crop into a new physical video file. |
close |
Close the video backend. |
crop |
Return a virtual, on-read cropped view of this video. |
deduplicate_with |
Create a new video with duplicate images removed. |
exists |
Check if the video file exists and is accessible. |
frame_to_seconds |
Convert a frame index to timestamp in seconds. |
from_crop |
Open |
from_filename |
Create a Video from a filename. |
has_overlapping_images |
Check if this video has overlapping images with another video. |
matches_content |
Check if this video has the same content as another video. |
matches_path |
Check if this video has the same path as another video. |
matches_shape |
Check if this video has the same shape as another video. |
merge_with |
Merge another video's images into this one. |
open |
Open the video backend for reading. |
replace_filename |
Update the filename of the video, optionally opening the backend. |
save |
Save video frames to a new video file. |
seconds_to_frame |
Convert a timestamp in seconds to frame index. |
set_video_plugin |
Set the video plugin and reopen the video. |
to_crop_coords |
Map source-frame |
to_source_coords |
Map cropped-frame |
Source code in sleap_io/model/video.py
@attrs.define(eq=False)
class Video:
"""`Video` class used by sleap to represent videos and data associated with them.
This class is used to store information regarding a video and its components.
It is used to store the video's `filename`, `shape`, and the video's `backend`.
To create a `Video` object, use the `from_filename` method which will select the
backend appropriately.
Attributes:
filename: The filename(s) of the video. Supported extensions: "mp4", "avi",
"mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",
"tiff", "bmp", "seq". If the filename is a list, a list of image filenames
are expected. If filename is a folder, it will be searched for images.
backend: An object that implements the basic methods for reading and
manipulating frames of a specific video type.
backend_metadata: A dictionary of metadata specific to the backend. This is
useful for storing metadata that requires an open backend (e.g., shape
information) without having access to the video file itself.
source_video: The source video object if this is a proxy video. This is present
when the video contains an embedded subset of frames from another video.
open_backend: Whether to open the backend when the video is available. If `True`
(the default), the backend will be automatically opened if the video exists.
Set this to `False` when you want to manually open the backend, or when the
you know the video file does not exist and you want to avoid trying to open
the file.
_exists_cache: Per-instance TTL cache for the result of `exists()` when the
`filename` is a remote URL. Keyed by `(filename, dataset)` and storing
`(exists_bool, monotonic_timestamp)`. This avoids issuing a network probe
on every call (e.g. from the `is_open` property, which GUIs poll on each
render). The TTL defaults to 60 seconds and can be overridden via the
`SLEAP_IO_EXISTS_TTL` environment variable. The cache is cleared on
`replace_filename`.
Notes:
Instances of this class are hashed by identity, not by value. This means that
two `Video` instances with the same attributes will NOT be considered equal in a
set or dict.
Media Video Plugin Support:
For media files (mp4, avi, etc.), the following plugins are supported:
- "opencv": Uses OpenCV (cv2) for video reading
- "FFMPEG": Uses imageio-ffmpeg for video reading
- "pyav": Uses PyAV for video reading
Plugin aliases (case-insensitive):
- opencv: "opencv", "cv", "cv2", "ocv"
- FFMPEG: "FFMPEG", "ffmpeg", "imageio-ffmpeg", "imageio_ffmpeg"
- pyav: "pyav", "av"
Plugin selection priority:
1. Explicitly specified plugin parameter
2. Backend metadata plugin value
3. Global default (set via sio.set_default_video_plugin)
4. Auto-detection based on available packages
See Also:
VideoBackend: The backend interface for reading video data.
sleap_io.set_default_video_plugin: Set global default plugin.
sleap_io.get_default_video_plugin: Get current default plugin.
"""
filename: str | list[str]
backend: VideoBackend | None = None
backend_metadata: dict[str, any] = attrs.field(factory=dict)
source_video: "Video | None" = None
open_backend: bool = True
_exists_cache: dict[tuple[str, str | None], tuple[bool, float]] = attrs.field(
init=False, factory=dict, repr=False, eq=False
)
# URL auth context, threaded in by `make_video` for remote loads. Persisted
# on the Video (not just the backend) so existence probes and a later
# `open()` reconstruction stay authenticated after the backend is closed.
_url_headers: dict[str, str] | None = attrs.field(
init=False, default=None, repr=False, eq=False
)
_url_stream_mode: str = attrs.field(
init=False, default="blockcache", repr=False, eq=False
)
EXTS = MediaVideo.EXTS + HDF5Video.EXTS + ImageVideo.EXTS + ("seq",)
def _backend_url_headers(self) -> dict[str, str] | None:
"""Return the HTTP headers to authenticate remote existence probes.
Prefers the URL auth context stored on this `Video` (set by `make_video`
at load time); falls back to the live backend's headers when present.
Returns `None` for local files and unauthenticated URLs.
"""
if self._url_headers is not None:
return self._url_headers
if isinstance(self.backend, HDF5Video):
return getattr(self.backend, "_url_headers", None)
return None
@property
def original_video(self) -> "Video | None":
"""The root video in the provenance chain.
For embedded videos, this returns the ultimate source video by
traversing the source_video chain. Returns None if this video
has no source_video (i.e., it IS an original).
This property is computed by following the source_video chain to find
the root. For a single-level embedding (A embeds from B), original_video
returns B. For multi-level embedding (A <- B <- C), it returns C.
"""
if self.source_video is None:
return None # This IS the original
# Traverse to root
v = self.source_video
while v.source_video is not None:
v = v.source_video
return v
def __attrs_post_init__(self):
"""Post init syntactic sugar."""
if self.open_backend and self.backend is None and self.exists():
try:
self.open()
except Exception:
# If we can't open the backend, just ignore it for now so we don't
# prevent the user from building the Video object entirely.
pass
def __deepcopy__(self, memo):
"""Deep copy the video object."""
if id(self) in memo:
return memo[id(self)]
reopen = False
if self.is_open:
reopen = True
self.close()
new_video = Video(
filename=self.filename,
backend=None,
backend_metadata=self.backend_metadata.copy(),
source_video=self.source_video,
open_backend=self.open_backend,
)
memo[id(self)] = new_video
if reopen:
self.open()
return new_video
@classmethod
def from_filename(
cls,
filename: str | list[str],
dataset: str | None = None,
grayscale: bool | None = None,
keep_open: bool = True,
source_video: "Video | None" = None,
**kwargs,
) -> VideoBackend:
"""Create a Video from a filename.
Args:
filename: The filename(s) of the video. Supported extensions: "mp4", "avi",
"mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",
"tiff", "bmp". If the filename is a list, a list of image filenames are
expected. If filename is a folder, it will be searched for images.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
source_video: The source video object if this is a proxy video. This is
present when the video contains an embedded subset of frames from
another video.
**kwargs: Additional backend-specific arguments passed to
VideoBackend.from_filename. See VideoBackend.from_filename for supported
arguments.
Returns:
Video instance with the appropriate backend instantiated.
"""
backend = VideoBackend.from_filename(
filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
**kwargs,
)
# If filename is a directory, VideoBackend.from_filename will expand it
# to a list of paths to images contained within the directory. In this
# case we want to use the expanded list as filename
return cls(
filename=backend.filename,
backend=backend,
source_video=source_video,
)
def crop(
self,
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
) -> "Video":
"""Return a virtual, on-read cropped view of this video.
Exactly one region spec must be given: ``crop`` (explicit
``(x1, y1, x2, y2)`` rect), ``bbox``, ``roi`` (its axis-aligned bounds +
``margin``), or (``center``, ``size``) for a fixed-size centered/
centroid-following window. The returned ``Video`` shares no pixels with
this one; frames are decoded on read and cropped (byte-identical to
:func:`sleap_io.transform.frame.crop_frame`). Out-of-bounds regions are
pad-filled with ``fill`` (never clamped), so the output shape is always
exactly ``(y2 - y1, x2 - x1)``.
The crop composes (FLATTENS when fills agree and the region is in-bounds)
with any existing crop on this video via
:meth:`CropVideoBackend.wrap`. ``source_video`` is set to this video for
provenance. When ``share_decode`` (the default), the new crop reuses this
video's backend instance as the shared inner so a mosaic of tiles over
one file decodes each source frame once; in that case the new tile does
NOT own the shared decoder (this video does).
Args:
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2``
exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: Any object exposing axis-aligned ``.bounds`` as
``(minx, miny, maxx, maxy)`` (e.g. a shapely geometry).
center: Window center ``(cx, cy)`` (used with ``size``).
size: Fixed output ``(width, height)`` (used with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (the default), reuse this video's backend
as the shared inner so tiles decode each frame once; the new tile
does not own the shared decoder.
Returns:
A new ``Video`` exposing the cropped view.
"""
from sleap_io.io.video_reading import CropVideoBackend
rect = _resolve_crop_rect(crop, bbox, roi, center, size, margin)
if self.backend is None and self.open_backend:
self.open()
if self.backend is None:
raise ValueError(
"Cannot crop a video with no open backend. Open it first (set "
"open_backend=True or call .open()) before cropping."
)
inner = self.backend
cropped_backend = CropVideoBackend.wrap(
inner=inner, crop=rect, fill=fill, owns_inner=not share_decode
)
cropped = Video(
filename=self.filename,
backend=cropped_backend,
source_video=self,
open_backend=self.open_backend,
)
x1, y1, x2, y2 = cropped_backend.crop
src_shape = self.shape
cropped.backend_metadata = {
**self.backend_metadata,
"shape": (src_shape[0], y2 - y1, x2 - x1, src_shape[3])
if src_shape is not None
else None,
# The uncropped source shape, so a closed re-serialize keeps videos_json
# describing the full frame even without a live source_video (D-120/DI-2).
"source_shape": list(src_shape) if src_shape is not None else None,
# COMPOSED source rect from wrap (D-120): keeps open/closed crop keys
# identical and root-canonical, and survives close()->open().
"crop": list(cropped_backend.crop),
"crop_fill": cropped_backend.fill,
}
return cropped
@classmethod
def from_crop(
cls,
video: "str | Path | Video",
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
**kwargs,
) -> "Video":
"""Open ``video`` (path or ``Video``) and return a virtual crop.
Accepts the same region specs as :meth:`crop` (``crop``/``bbox``/``roi``/
``center``+``size``); extra keyword arguments are forwarded to
:meth:`from_filename` when ``video`` is a path (ignored when it is already
a ``Video``).
Args:
video: A path/filename to open, or an existing ``Video`` to crop.
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2`` exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: An object exposing axis-aligned ``.bounds`` (e.g. a shapely
geometry); ``margin`` is applied around it.
center: Window center ``(cx, cy)`` (with ``size``).
size: Fixed output ``(width, height)`` (with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (default), reuse the source decoder.
**kwargs: Forwarded to :meth:`from_filename` for a path input.
Returns:
A new ``Video`` exposing the cropped view.
"""
if isinstance(video, (str, Path)):
video = cls.from_filename(video, **kwargs)
return video.crop(
crop,
bbox=bbox,
roi=roi,
center=center,
size=size,
margin=margin,
fill=fill,
share_decode=share_decode,
)
def _crop_tuple(self) -> tuple[int, int, int, int] | None:
"""Return this video's crop rect ``(x1, y1, x2, y2)`` or ``None``.
Reads ``backend.crop`` when the backend is a ``CropVideoBackend`` (open
path), else ``backend_metadata["crop"]`` (closed path), else ``None``
(uncropped).
"""
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
return tuple(self.backend.crop)
crop = self.backend_metadata.get("crop")
return tuple(crop) if crop is not None else None
def _crop_fill(self) -> int | tuple[int, ...]:
"""Return this video's crop fill value (open: backend; closed: metadata).
Returns ``0`` for an uncropped video. Mirrors :meth:`_crop_tuple`.
"""
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
return self.backend.fill
return self.backend_metadata.get("crop_fill", 0)
@property
def is_cropped(self) -> bool:
"""Whether this video is a virtual crop of another video."""
return self._crop_tuple() is not None
@property
def crop_rect(self) -> tuple[int, int, int, int] | None:
"""Crop rect ``(x1, y1, x2, y2)`` in source coords, or ``None`` if uncropped."""
return self._crop_tuple()
@property
def crop_fill(self) -> int | tuple[int, ...]:
"""The out-of-bounds fill value for this video's crop (``0`` if uncropped)."""
return self._crop_fill()
def to_crop_coords(self, points: np.ndarray) -> np.ndarray:
"""Map source-frame ``(x, y)`` into this video's cropped frame.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated into the cropped frame. If this video is not
cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else crop_points(points, crop)
def to_source_coords(self, points: np.ndarray) -> np.ndarray:
"""Map cropped-frame ``(x, y)`` back to source-frame coordinates.
Inverse of :meth:`to_crop_coords`.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated back to source coordinates. If this video is
not cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else uncrop_points(points, crop)
@property
def shape(self) -> tuple[int, int, int, int] | None:
"""Return the shape of the video as (num_frames, height, width, channels).
If the video backend is not set or it cannot determine the shape of the video,
this will return None.
"""
return self._get_shape()
def _get_shape(self) -> tuple[int, int, int, int] | None:
"""Return the shape of the video as (num_frames, height, width, channels).
This suppresses errors related to querying the backend for the video shape, such
as when it has not been set or when the video file is not found.
"""
try:
return self.backend.shape
except Exception:
if "shape" in self.backend_metadata:
return self.backend_metadata["shape"]
return None
@property
def grayscale(self) -> bool | None:
"""Return whether the video is grayscale.
If the video backend is not set or it cannot determine whether the video is
grayscale, this will return None.
"""
shape = self.shape
if shape is not None:
return shape[-1] == 1
else:
grayscale = None
if "grayscale" in self.backend_metadata:
grayscale = self.backend_metadata["grayscale"]
return grayscale
@grayscale.setter
def grayscale(self, value: bool):
"""Set the grayscale value and adjust the backend."""
if self.backend is not None:
self.backend.grayscale = value
self.backend._cached_shape = None
self.backend_metadata["grayscale"] = value
@property
def fps(self) -> float | None:
"""Return the frames per second of the video.
For MediaVideo backends, this reads FPS from the video container metadata.
For other backends (ImageVideo, HDF5Video, TiffVideo), this returns the
explicitly set value or None if not set.
Returns:
The FPS if known, or None if unavailable/unknown.
"""
if self.backend is not None:
return self.backend.fps
return self.backend_metadata.get("fps")
@fps.setter
def fps(self, value: float | None):
"""Set the frames per second.
Args:
value: Frames per second. Must be positive if not None.
Raises:
ValueError: If value is not positive.
Notes:
For MediaVideo backends, setting FPS overrides the value from container
metadata. For other backends, this sets the FPS directly.
"""
if value is not None and value <= 0:
raise ValueError(f"FPS must be positive, got {value}")
if self.backend is not None:
self.backend.fps = value
self.backend_metadata["fps"] = value
def frame_to_seconds(self, frame_idx: int) -> float | None:
"""Convert a frame index to timestamp in seconds.
Args:
frame_idx: Zero-indexed frame number.
Returns:
Time in seconds, or None if FPS is unknown.
Notes:
This assumes constant frame rate. For variable frame rate videos,
the returned timestamp may be approximate.
"""
if self.fps is None or self.fps <= 0:
return None
return frame_idx / self.fps
def seconds_to_frame(self, seconds: float) -> int | None:
"""Convert a timestamp in seconds to frame index.
Args:
seconds: Time in seconds from video start.
Returns:
Zero-indexed frame number (rounded down), or None if FPS unknown.
"""
if self.fps is None or self.fps <= 0:
return None
return int(seconds * self.fps)
def __len__(self) -> int:
"""Return the length of the video as the number of frames."""
shape = self.shape
return 0 if shape is None else shape[0]
def __repr__(self) -> str:
"""Informal string representation (for print or format)."""
dataset = (
f"dataset={self.backend.dataset}, "
if getattr(self.backend, "dataset", "")
else ""
)
return (
"Video("
f'filename="{self.filename}", '
f"shape={self.shape}, "
f"{dataset}"
f"backend={type(self.backend).__name__}"
")"
)
def __str__(self) -> str:
"""Informal string representation (for print or format)."""
return self.__repr__()
def __getitem__(self, inds: int | list[int] | slice) -> np.ndarray:
"""Return the frames of the video at the given indices.
Args:
inds: Index or list of indices of frames to read.
Returns:
Frame or frames as a numpy array of shape `(height, width, channels)` if a
scalar index is provided, or `(frames, height, width, channels)` if a list
of indices is provided.
See also: VideoBackend.get_frame, VideoBackend.get_frames
"""
if not self.is_open:
if self.open_backend:
self.open()
else:
raise ValueError(
"Video backend is not open. Call video.open() or set "
"video.open_backend to True to do automatically on frame read."
)
return self.backend[inds]
def exists(self, check_all: bool = False, dataset: str | None = None) -> bool:
"""Check if the video file exists and is accessible.
Args:
check_all: If `True`, check that all filenames in a list exist. If `False`
(the default), check that the first filename exists.
dataset: Name of dataset in HDF5 file. If specified, this will function will
return `False` if the dataset does not exist.
Returns:
`True` if the file exists and is accessible, `False` otherwise.
"""
if isinstance(self.filename, list):
if check_all:
for f in self.filename:
if not is_file_accessible(f):
return False
return True
else:
return is_file_accessible(self.filename[0])
# URL fast path: must run BEFORE `is_file_accessible`, which treats the
# filename as a local path and would spuriously return False for a URL.
from sleap_io.io._remote import _is_url
if _is_url(self.filename):
return self._url_exists(dataset)
file_is_accessible = is_file_accessible(self.filename)
if not file_is_accessible:
# Check if it's a directory (ImageVideo source)
if Path(self.filename).is_dir():
return True
return False
if dataset is None or dataset == "":
dataset = self.backend_metadata.get("dataset", None)
if dataset is not None and dataset != "":
has_dataset = False
if (
self.backend is not None
and type(self.backend) is HDF5Video
and self.backend._open_reader is not None
):
has_dataset = dataset in self.backend._open_reader
else:
with h5py.File(self.filename, "r") as f:
has_dataset = dataset in f
return has_dataset
return True
def _url_exists(self, dataset: str | None) -> bool:
"""Check whether a remote URL `filename` exists, with a TTL cache.
Args:
dataset: Name of dataset in the (remote) HDF5 file. If specified (or
derivable from `backend_metadata`), existence additionally requires
that the dataset be present in the file.
Returns:
`True` if the URL is reachable (and, if a dataset was requested, the
dataset exists), `False` otherwise.
Notes:
Results are cached per instance keyed by `(filename, dataset)` for a
TTL (default 60s, overridable via the `SLEAP_IO_EXISTS_TTL` env var) so
repeated calls (e.g. from the `is_open` property in a GUI render loop)
do not issue a network probe each time.
"""
from sleap_io.io._remote import _head_or_range_probe
key = (self.filename, dataset)
try:
ttl = float(os.environ.get("SLEAP_IO_EXISTS_TTL", "60"))
except ValueError:
# A malformed env value must not break the never-raise bool
# contract of exists()/is_open; fall back to the 60s default.
ttl = 60.0
cached = self._exists_cache.get(key)
if cached is not None and (time.monotonic() - cached[1]) < ttl:
return cached[0]
try:
if not _head_or_range_probe(
self.filename, headers=self._backend_url_headers()
):
result = False
else:
if dataset is None or dataset == "":
dataset = self.backend_metadata.get("dataset", None)
if dataset is None or dataset == "":
result = True
else:
result = self._url_dataset_exists(dataset)
except Exception:
result = False
self._exists_cache[key] = (result, time.monotonic())
return result
def _url_dataset_exists(self, dataset: str) -> bool:
"""Check whether `dataset` is present in the remote HDF5 file.
Reuses the backend's already-open HDF5 reader when available; otherwise
opens the remote file via fsspec for a single membership check.
Args:
dataset: Name of dataset in the remote HDF5 file.
Returns:
`True` if the dataset is present, `False` otherwise.
"""
if (
self.backend is not None
and type(self.backend) is HDF5Video
and self.backend._open_reader is not None
):
return dataset in self.backend._open_reader
from sleap_io.io._remote import open_remote_h5
url_file = open_remote_h5(self.filename, headers=self._backend_url_headers())
try:
with h5py.File(url_file, "r") as f:
return dataset in f
finally:
url_file.close()
@property
def is_open(self) -> bool:
"""Check if the video backend is open."""
return self.exists() and self.backend is not None
def open(
self,
filename: str | None = None,
dataset: str | None = None,
grayscale: str | None = None,
keep_open: bool = True,
plugin: str | None = None,
):
"""Open the video backend for reading.
Args:
filename: Filename to open. If not specified, will use the filename set on
the video object.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
plugin: Video plugin to use for MediaVideo files. One of "opencv",
"FFMPEG", or "pyav". Also accepts aliases (case-insensitive).
If not specified, uses the backend metadata, global default,
or auto-detection in that order.
Notes:
This is useful for opening the video backend to read frames and then closing
it after reading all the necessary frames.
If the backend was already open, it will be closed before opening a new one.
Values for the HDF5 dataset and grayscale will be remembered if not
specified.
"""
if filename is not None:
self.replace_filename(filename, open=False)
# Try to remember values from previous backend if available and not specified.
if self.backend is not None:
if dataset is None:
dataset = getattr(self.backend, "dataset", None)
if grayscale is None:
grayscale = getattr(self.backend, "grayscale", None)
else:
if dataset is None and "dataset" in self.backend_metadata:
dataset = self.backend_metadata["dataset"]
if grayscale is None:
if "grayscale" in self.backend_metadata:
grayscale = self.backend_metadata["grayscale"]
elif "shape" in self.backend_metadata:
grayscale = self.backend_metadata["shape"][-1] == 1
if not self.exists(dataset=dataset):
from sleap_io.io._remote import _is_url, _redact_url
# Redact credential-bearing URLs (e.g. presigned ``?token=`` links)
# so they never surface in tracebacks/logs. Local paths are shown
# verbatim.
name = (
_redact_url(self.filename)
if isinstance(self.filename, str) and _is_url(self.filename)
else self.filename
)
msg = f"Video does not exist or cannot be opened for reading: {name}"
if dataset is not None:
msg += f" (dataset: {dataset})"
raise FileNotFoundError(msg)
# Close previous backend if open.
self.close()
# Handle plugin parameter
backend_kwargs = {}
if plugin is not None:
from sleap_io.io.video_reading import normalize_plugin_name
plugin = normalize_plugin_name(plugin)
self.backend_metadata["plugin"] = plugin
if "plugin" in self.backend_metadata:
backend_kwargs["plugin"] = self.backend_metadata["plugin"]
# Create new backend. Forward the URL auth context so a reopened remote
# HDF5Video stays authenticated (the previous backend, and its headers,
# were dropped by self.close() above).
self.backend = VideoBackend.from_filename(
self.filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
url_headers=self._url_headers,
url_stream_mode=self._url_stream_mode,
**backend_kwargs,
)
# Re-wrap as a crop view if this video records a crop in its metadata.
# The rebuilt backend above is always a plain backend, so this wraps
# exactly once (idempotent across close()->open() and deepcopy).
if "crop" in self.backend_metadata:
from sleap_io.io.video_reading import CropVideoBackend
self.backend = CropVideoBackend.wrap(
inner=self.backend,
crop=tuple(self.backend_metadata["crop"]),
fill=self.backend_metadata.get("crop_fill", 0),
)
def close(self):
"""Close the video backend."""
if self.backend is not None:
# Try to remember values from previous backend if available and not
# specified.
try:
self.backend_metadata["dataset"] = getattr(
self.backend, "dataset", None
)
self.backend_metadata["grayscale"] = getattr(
self.backend, "grayscale", None
)
self.backend_metadata["shape"] = getattr(self.backend, "shape", None)
self.backend_metadata["fps"] = getattr(self.backend, "fps", None)
# Persist the crop so a Video cropped in-memory (never loaded
# from disk) survives a close()->open() and deepcopy: open()
# re-wraps from these keys (the closed-path shape above is
# already the cropped shape).
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
self.backend_metadata["crop"] = list(self.backend.crop)
self.backend_metadata["crop_fill"] = self.backend.fill
except Exception:
pass
# Deterministically release the backend's open handles (the cached
# reader and, for a remote HDF5Video, the fsspec URL file-like)
# rather than relying on garbage collection.
try:
self.backend.close()
except Exception:
pass
del self.backend
self.backend = None
def replace_filename(
self, new_filename: str | Path | list[str] | list[Path], open: bool = True
):
"""Update the filename of the video, optionally opening the backend.
Args:
new_filename: New filename to set for the video.
open: If `True` (the default), open the backend with the new filename. If
the new filename does not exist, no error is raised.
"""
if isinstance(new_filename, Path):
new_filename = new_filename.as_posix()
if isinstance(new_filename, list):
new_filename = [
p.as_posix() if isinstance(p, Path) else p for p in new_filename
]
# A relink to a different file makes the recorded shape/grayscale/fps in
# ``backend_metadata`` stale: they describe the OLD file but the new file
# may have a different resolution/channels/frame rate. They must not be
# serialized under the new filename (regression from #483, where
# ``save_slp(prefer_metadata=True)`` prefers these recorded values), so
# invalidate them on a real relink and let them be recomputed from the new
# backend. The no-relink path leaves metadata untouched so golden
# byte-identical saves stay byte-identical.
filename_changed = new_filename != self.filename
self.filename = new_filename
self.backend_metadata["filename"] = new_filename
# Invalidate any cached URL existence results for the previous filename.
self._exists_cache.clear()
if open:
if self.exists():
self.open()
else:
self.close()
# Drop stale metadata AFTER (re)opening: ``open()`` internally calls
# ``close()``, which would otherwise re-stamp the OLD backend's
# shape/grayscale/fps back into ``backend_metadata``.
if filename_changed:
for key in ("shape", "grayscale", "fps"):
self.backend_metadata.pop(key, None)
def matches_path(self, other: "Video", strict: bool = False) -> bool:
"""Check if this video has the same path as another video.
Args:
other: Another video to compare with.
strict: If True, require exact path match. If False, consider videos
with the same filename (basename) as matching.
Returns:
True if the videos have matching paths, False otherwise.
Notes:
For HDF5 video backends (e.g., embedded videos in .pkg.slp files),
matching prioritizes the source_filename attribute since multiple
videos can share the same HDF5 file path but reference different
source videos. Falls back to dataset name matching if source_filename
is not available.
"""
# Handle HDF5 backends specially - prioritize source_filename matching
self_is_hdf5 = isinstance(self.backend, HDF5Video)
other_is_hdf5 = isinstance(other.backend, HDF5Video)
if self_is_hdf5 and other_is_hdf5:
# Both are HDF5 videos - must match by BOTH source_filename AND dataset
# to distinguish different videos embedded in the same pkg.slp file
self_source = self.backend.source_filename
other_source = other.backend.source_filename
self_dataset = self.backend.dataset
other_dataset = other.backend.dataset
# If both have datasets, they must match
if self_dataset is not None and other_dataset is not None:
if self_dataset != other_dataset:
return False # Different datasets = different videos
# If both have source_filenames, compare them
if self_source is not None and other_source is not None:
if strict:
# For HDF5 videos, just compare normalized path strings
# (avoid slow resolve() on network paths)
return Path(self_source).as_posix() == Path(other_source).as_posix()
else:
return Path(self_source).name == Path(other_source).name
# If only datasets available (no source_filename), they must match
if self_dataset is not None and other_dataset is not None:
return self_dataset == other_dataset
# If neither source_filename nor dataset available, cannot match
return False
if isinstance(self.filename, list) and isinstance(other.filename, list):
# Both are image sequences
if strict:
return self.filename == other.filename
else:
# Compare basenames
self_basenames = [Path(f).name for f in self.filename]
other_basenames = [Path(f).name for f in other.filename]
return self_basenames == other_basenames
elif isinstance(self.filename, list) or isinstance(other.filename, list):
# One is image sequence, other is single file
return False
else:
# Both are single files - use resolve() for symlink handling
if strict:
p1, p2 = Path(self.filename), Path(other.filename)
# Fast string comparison first
if p1.as_posix() == p2.as_posix():
return True
# Only resolve if both exist locally (avoid slow network timeouts)
try:
if p1.exists() and p2.exists():
return p1.resolve() == p2.resolve()
except OSError:
pass
return False
else:
return Path(self.filename).name == Path(other.filename).name
def matches_content(self, other: "Video") -> bool:
"""Check if this video has the same content as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same shape and backend type.
Notes:
This compares metadata like shape and backend type, not actual frame data.
"""
# Compare shapes
self_shape = self.shape
other_shape = other.shape
if self_shape != other_shape:
return False
# Compare backend types
if self.backend is None and other.backend is None:
return True
elif self.backend is None or other.backend is None:
return False
return type(self.backend).__name__ == type(other.backend).__name__
def matches_shape(self, other: "Video") -> bool:
"""Check if this video has the same shape as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same height, width, and channels.
Notes:
This only compares spatial dimensions, not the number of frames.
"""
# Try to get shape from backend metadata first if shape is not available
if self.backend is None and "shape" in self.backend_metadata:
self_shape = self.backend_metadata["shape"]
else:
self_shape = self.shape
if other.backend is None and "shape" in other.backend_metadata:
other_shape = other.backend_metadata["shape"]
else:
other_shape = other.shape
# Handle None shapes
if self_shape is None or other_shape is None:
return False
# Compare only height, width, channels (not frames)
return self_shape[1:] == other_shape[1:]
def has_overlapping_images(self, other: "Video") -> bool:
"""Check if this video has overlapping images with another video.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to compare with.
Returns:
True if both are ImageVideo instances with overlapping image files.
False if either video is not an ImageVideo or no overlap exists.
Notes:
Only works with ImageVideo backends where filename is a list.
Compares individual image filenames (basenames only).
"""
# Both must be image sequences
if not (isinstance(self.filename, list) and isinstance(other.filename, list)):
return False
# Get basenames for comparison
self_basenames = set(Path(f).name for f in self.filename)
other_basenames = set(Path(f).name for f in other.filename)
# Check if there's any overlap
return len(self_basenames & other_basenames) > 0
def deduplicate_with(self, other: "Video") -> "Video":
"""Create a new video with duplicate images removed.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to deduplicate against. Must also be ImageVideo.
Returns:
A new Video object with duplicate images removed from this video,
or None if all images were duplicates.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
Images are considered duplicates if they have the same basename.
The returned video contains only images from this video that are
not present in the other video.
"""
if not isinstance(self.filename, list):
raise ValueError("deduplicate_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get basenames from other video
other_basenames = set(Path(f).name for f in other.filename)
# Keep only non-duplicate images
deduplicated_paths = [
f for f in self.filename if Path(f).name not in other_basenames
]
if not deduplicated_paths:
# All images were duplicates
return None
# Create new video with deduplicated images
return Video.from_filename(deduplicated_paths, grayscale=self.grayscale)
def merge_with(self, other: "Video") -> "Video":
"""Merge another video's images into this one.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to merge with. Must also be ImageVideo.
Returns:
A new Video object with unique images from both videos.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
The merged video contains all unique images from both videos,
with automatic deduplication based on image basename.
"""
if not isinstance(self.filename, list):
raise ValueError("merge_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get all unique images (by basename) preserving order
seen_basenames = set()
merged_paths = []
for path in self.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
for path in other.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
# Create new video with merged images
return Video.from_filename(merged_paths, grayscale=self.grayscale)
def save(
self,
save_path: str | Path,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Save video frames to a new video file.
Args:
save_path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to save. Can be specified as a list or array of
frame integers. If not specified, saves all video frames.
fps: Frames per second for the output video. If not specified, uses the
source video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
`sio.save_video` for video compression.
Returns:
A new `Video` object pointing to the new video file.
"""
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
frame_inds = np.arange(len(self)) if frame_inds is None else frame_inds
# Use source video FPS if not explicitly specified
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(save_path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
new_video = Video.from_filename(save_path, grayscale=self.grayscale)
return new_video
def apply_crop(
self,
path: str | Path,
*,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Bake this video's virtual crop into a new physical video file.
Materializes the cropped frames (``self[i]``, already cropped by the
virtual :class:`~sleap_io.io.video_reading.CropVideoBackend`) to ``path``
via :class:`~sleap_io.io.video_writing.VideoWriter`. The crop becomes
physical: the returned video has no ``CropVideoBackend`` / ``/video_crops``
entry. ``baked.shape`` equals this video's cropped shape when the cropped
width and height are multiples of 16; otherwise the H.264 encoder pads the
bottom/right edges up to the next multiple of 16 (the macro-block size),
so ``baked.shape`` may exceed the cropped shape on those edges. The
top-left content is preserved, so coordinates stay aligned regardless.
This operation is coordinate-neutral. A virtual crop already presents
cropped-frame coordinates, so baking the cropped pixels does not change
any point coordinates (unlike ``sio transform --crop``, which applies a
new crop and adjusts coordinates).
Provenance is preserved: the returned video's ``source_video`` is the
uncropped original — ``self.source_video`` (the parent a virtual crop is
created against), or, for a manually-built crop with no parent, an
uncropped view reconstructed from the crop backend's inner. So
``baked.source_video.shape`` is the uncropped shape while ``baked.shape``
is the cropped shape, and ``baked.grayscale`` is carried from this video.
Args:
path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to bake. Can be specified as a list or array
of frame integers. If not specified, bakes all video frames.
fps: Frames per second for the output video. If not specified, uses
this video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
``sio.save_video`` for video compression.
Returns:
A new ``Video`` pointing to the baked file, with ``source_video`` set
to the uncropped original (or this video) and ``grayscale`` carried
from this video.
Raises:
ValueError: If this video has no virtual crop to apply (i.e.,
:meth:`_crop_tuple` returns ``None``). Use :meth:`save` to
re-encode an uncropped video.
"""
if self._crop_tuple() is None:
raise ValueError(
"apply_crop requires a cropped video (a virtual crop created via "
"Video.crop / Video.from_crop), but this video has no crop to "
"apply. Use Video.save to re-encode an uncropped video."
)
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
if frame_inds is None:
# A crop over a SPARSELY embedded video (frame_map keys are not the dense
# range 0..N-1, e.g. {5, 9}) cannot be baked by default: writing the frames
# compacts them to 0..k-1, so any labeled frame referencing a source index
# (5, 9) would dangle. Refuse with a clear error rather than crash or
# silently misalign. An explicit frame_inds bypasses this for advanced use.
inner = getattr(self.backend, "inner", None)
frame_map = getattr(inner, "frame_map", None)
if frame_map:
keys = sorted(frame_map.keys())
if keys != list(range(len(keys))):
raise ValueError(
"Cannot bake a virtual crop over a video with sparsely "
f"embedded frames (frame_map keys {keys}): baking would "
"compact frames to a contiguous range and break frame_idx "
"references. Pass explicit frame_inds to override, or "
"materialize from the original source video."
)
frame_inds = np.arange(len(self))
# Use this video's FPS if not explicitly specified.
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
baked = Video.from_filename(path, grayscale=self.grayscale)
# Provenance: the uncropped original. Walk past any still-virtual crop
# ancestors (a flattened crop-of-crop's source_video may itself be a crop)
# to the first uncropped ancestor. For a manually-built crop with no parent,
# reconstruct an uncropped view from the crop backend's inner, so
# source_video is never a cropped video.
source = self.source_video
while source is not None and source._crop_tuple() is not None:
source = source.source_video
if source is None:
inner = getattr(self.backend, "inner", None)
source = (
Video(filename=inner.filename, backend=inner)
if inner is not None
else self
)
baked.source_video = source
return baked
def set_video_plugin(self, plugin: str) -> None:
"""Set the video plugin and reopen the video.
Args:
plugin: Video plugin to use. One of "opencv", "FFMPEG", or "pyav".
Also accepts aliases (case-insensitive).
Raises:
ValueError: If the video is not a MediaVideo type.
Examples:
>>> video.set_video_plugin("opencv")
>>> video.set_video_plugin("CV2") # Same as "opencv"
"""
from sleap_io.io.video_reading import MediaVideo, normalize_plugin_name
if not self.filename.endswith(MediaVideo.EXTS):
raise ValueError(f"Cannot set plugin for non-media video: {self.filename}")
plugin = normalize_plugin_name(plugin)
# Close current backend if open
was_open = self.is_open
if was_open:
self.close()
# Update backend metadata
self.backend_metadata["plugin"] = plugin
# Reopen with new plugin if it was open
if was_open:
self.open()
EXTS = ('mp4', 'avi', 'mov', 'mj2', 'mkv', 'h5', 'hdf5', 'slp', 'png', 'jpg', 'jpeg', 'tif', 'tiff', 'bmp', 'seq')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__annotations__ = {'filename': 'str | list[str]', 'backend': 'VideoBackend | None', 'backend_metadata': 'dict[str, any]', 'source_video': "'Video | None'", 'open_backend': 'bool', '_exists_cache': 'dict[tuple[str, str | None], tuple[bool, float]]', '_url_headers': 'dict[str, str] | None', '_url_stream_mode': 'str'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = False
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = '`Video` class used by sleap to represent videos and data associated with them.\n\nThis class is used to store information regarding a video and its components.\nIt is used to store the video\'s `filename`, `shape`, and the video\'s `backend`.\n\nTo create a `Video` object, use the `from_filename` method which will select the\nbackend appropriately.\n\nAttributes:\n filename: The filename(s) of the video. Supported extensions: "mp4", "avi",\n "mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",\n "tiff", "bmp", "seq". If the filename is a list, a list of image filenames\n are expected. If filename is a folder, it will be searched for images.\n backend: An object that implements the basic methods for reading and\n manipulating frames of a specific video type.\n backend_metadata: A dictionary of metadata specific to the backend. This is\n useful for storing metadata that requires an open backend (e.g., shape\n information) without having access to the video file itself.\n source_video: The source video object if this is a proxy video. This is present\n when the video contains an embedded subset of frames from another video.\n open_backend: Whether to open the backend when the video is available. If `True`\n (the default), the backend will be automatically opened if the video exists.\n Set this to `False` when you want to manually open the backend, or when the\n you know the video file does not exist and you want to avoid trying to open\n the file.\n _exists_cache: Per-instance TTL cache for the result of `exists()` when the\n `filename` is a remote URL. Keyed by `(filename, dataset)` and storing\n `(exists_bool, monotonic_timestamp)`. This avoids issuing a network probe\n on every call (e.g. from the `is_open` property, which GUIs poll on each\n render). The TTL defaults to 60 seconds and can be overridden via the\n `SLEAP_IO_EXISTS_TTL` environment variable. The cache is cleared on\n `replace_filename`.\n\nNotes:\n Instances of this class are hashed by identity, not by value. This means that\n two `Video` instances with the same attributes will NOT be considered equal in a\n set or dict.\n\nMedia Video Plugin Support:\n For media files (mp4, avi, etc.), the following plugins are supported:\n - "opencv": Uses OpenCV (cv2) for video reading\n - "FFMPEG": Uses imageio-ffmpeg for video reading\n - "pyav": Uses PyAV for video reading\n\n Plugin aliases (case-insensitive):\n - opencv: "opencv", "cv", "cv2", "ocv"\n - FFMPEG: "FFMPEG", "ffmpeg", "imageio-ffmpeg", "imageio_ffmpeg"\n - pyav: "pyav", "av"\n\n Plugin selection priority:\n 1. Explicitly specified plugin parameter\n 2. Backend metadata plugin value\n 3. Global default (set via sio.set_default_video_plugin)\n 4. Auto-detection based on available packages\n\nSee Also:\n VideoBackend: The backend interface for reading video data.\n sleap_io.set_default_video_plugin: Set global default plugin.\n sleap_io.get_default_video_plugin: Get current default plugin.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 102
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('filename', 'backend', 'backend_metadata', 'source_video', 'open_backend')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.video'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('filename', 'backend', 'backend_metadata', 'source_video', 'open_backend', '_exists_cache', '_url_headers', '_url_stream_mode', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('backend', 'filename')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
crop_fill
property
¶
The out-of-bounds fill value for this video's crop (0 if uncropped).
crop_rect
property
¶
Crop rect (x1, y1, x2, y2) in source coords, or None if uncropped.
fps
property
¶
Return the frames per second of the video.
For MediaVideo backends, this reads FPS from the video container metadata. For other backends (ImageVideo, HDF5Video, TiffVideo), this returns the explicitly set value or None if not set.
Returns:
| Type | Description |
|---|---|
|
The FPS if known, or None if unavailable/unknown. |
grayscale
property
¶
Return whether the video is grayscale.
If the video backend is not set or it cannot determine whether the video is grayscale, this will return None.
is_cropped
property
¶
Whether this video is a virtual crop of another video.
is_open
property
¶
Check if the video backend is open.
original_video
property
¶
The root video in the provenance chain.
For embedded videos, this returns the ultimate source video by traversing the source_video chain. Returns None if this video has no source_video (i.e., it IS an original).
This property is computed by following the source_video chain to find the root. For a single-level embedding (A embeds from B), original_video returns B. For multi-level embedding (A <- B <- C), it returns C.
shape
property
¶
Return the shape of the video as (num_frames, height, width, channels).
If the video backend is not set or it cannot determine the shape of the video, this will return None.
__attrs_post_init__()
¶
Post init syntactic sugar.
Source code in sleap_io/model/video.py
__deepcopy__(memo)
¶
Deep copy the video object.
Source code in sleap_io/model/video.py
def __deepcopy__(self, memo):
"""Deep copy the video object."""
if id(self) in memo:
return memo[id(self)]
reopen = False
if self.is_open:
reopen = True
self.close()
new_video = Video(
filename=self.filename,
backend=None,
backend_metadata=self.backend_metadata.copy(),
source_video=self.source_video,
open_backend=self.open_backend,
)
memo[id(self)] = new_video
if reopen:
self.open()
return new_video
__getitem__(inds)
¶
Return the frames of the video at the given indices.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inds
|
int | list[int] | slice
|
Index or list of indices of frames to read. |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Frame or frames as a numpy array of shape |
See also: VideoBackend.get_frame, VideoBackend.get_frames
Source code in sleap_io/model/video.py
def __getitem__(self, inds: int | list[int] | slice) -> np.ndarray:
"""Return the frames of the video at the given indices.
Args:
inds: Index or list of indices of frames to read.
Returns:
Frame or frames as a numpy array of shape `(height, width, channels)` if a
scalar index is provided, or `(frames, height, width, channels)` if a list
of indices is provided.
See also: VideoBackend.get_frame, VideoBackend.get_frames
"""
if not self.is_open:
if self.open_backend:
self.open()
else:
raise ValueError(
"Video backend is not open. Call video.open() or set "
"video.open_backend to True to do automatically on frame read."
)
return self.backend[inds]
__init__(filename, backend=None, backend_metadata=NOTHING, source_video=None, open_backend=True)
¶
Method generated by attrs for class Video.
__len__()
¶
__repr__()
¶
Informal string representation (for print or format).
Source code in sleap_io/model/video.py
def __repr__(self) -> str:
"""Informal string representation (for print or format)."""
dataset = (
f"dataset={self.backend.dataset}, "
if getattr(self.backend, "dataset", "")
else ""
)
return (
"Video("
f'filename="{self.filename}", '
f"shape={self.shape}, "
f"{dataset}"
f"backend={type(self.backend).__name__}"
")"
)
__str__()
¶
apply_crop(path, *, frame_inds=None, fps=None, video_kwargs=None)
¶
Bake this video's virtual crop into a new physical video file.
Materializes the cropped frames (self[i], already cropped by the
virtual :class:~sleap_io.io.video_reading.CropVideoBackend) to path
via :class:~sleap_io.io.video_writing.VideoWriter. The crop becomes
physical: the returned video has no CropVideoBackend / /video_crops
entry. baked.shape equals this video's cropped shape when the cropped
width and height are multiples of 16; otherwise the H.264 encoder pads the
bottom/right edges up to the next multiple of 16 (the macro-block size),
so baked.shape may exceed the cropped shape on those edges. The
top-left content is preserved, so coordinates stay aligned regardless.
This operation is coordinate-neutral. A virtual crop already presents
cropped-frame coordinates, so baking the cropped pixels does not change
any point coordinates (unlike sio transform --crop, which applies a
new crop and adjusts coordinates).
Provenance is preserved: the returned video's source_video is the
uncropped original — self.source_video (the parent a virtual crop is
created against), or, for a manually-built crop with no parent, an
uncropped view reconstructed from the crop backend's inner. So
baked.source_video.shape is the uncropped shape while baked.shape
is the cropped shape, and baked.grayscale is carried from this video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Path to the new video file. Should end in MP4. |
required |
frame_inds
|
list[int] | ndarray | None
|
Frame indices to bake. Can be specified as a list or array of frame integers. If not specified, bakes all video frames. |
None
|
fps
|
float | None
|
Frames per second for the output video. If not specified, uses this video's FPS if available, otherwise defaults to 30. |
None
|
video_kwargs
|
dict[str, Any] | None
|
A dictionary of keyword arguments to provide to
|
None
|
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Raises:
| Type | Description |
|---|---|
ValueError
|
If this video has no virtual crop to apply (i.e.,
:meth: |
Source code in sleap_io/model/video.py
def apply_crop(
self,
path: str | Path,
*,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Bake this video's virtual crop into a new physical video file.
Materializes the cropped frames (``self[i]``, already cropped by the
virtual :class:`~sleap_io.io.video_reading.CropVideoBackend`) to ``path``
via :class:`~sleap_io.io.video_writing.VideoWriter`. The crop becomes
physical: the returned video has no ``CropVideoBackend`` / ``/video_crops``
entry. ``baked.shape`` equals this video's cropped shape when the cropped
width and height are multiples of 16; otherwise the H.264 encoder pads the
bottom/right edges up to the next multiple of 16 (the macro-block size),
so ``baked.shape`` may exceed the cropped shape on those edges. The
top-left content is preserved, so coordinates stay aligned regardless.
This operation is coordinate-neutral. A virtual crop already presents
cropped-frame coordinates, so baking the cropped pixels does not change
any point coordinates (unlike ``sio transform --crop``, which applies a
new crop and adjusts coordinates).
Provenance is preserved: the returned video's ``source_video`` is the
uncropped original — ``self.source_video`` (the parent a virtual crop is
created against), or, for a manually-built crop with no parent, an
uncropped view reconstructed from the crop backend's inner. So
``baked.source_video.shape`` is the uncropped shape while ``baked.shape``
is the cropped shape, and ``baked.grayscale`` is carried from this video.
Args:
path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to bake. Can be specified as a list or array
of frame integers. If not specified, bakes all video frames.
fps: Frames per second for the output video. If not specified, uses
this video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
``sio.save_video`` for video compression.
Returns:
A new ``Video`` pointing to the baked file, with ``source_video`` set
to the uncropped original (or this video) and ``grayscale`` carried
from this video.
Raises:
ValueError: If this video has no virtual crop to apply (i.e.,
:meth:`_crop_tuple` returns ``None``). Use :meth:`save` to
re-encode an uncropped video.
"""
if self._crop_tuple() is None:
raise ValueError(
"apply_crop requires a cropped video (a virtual crop created via "
"Video.crop / Video.from_crop), but this video has no crop to "
"apply. Use Video.save to re-encode an uncropped video."
)
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
if frame_inds is None:
# A crop over a SPARSELY embedded video (frame_map keys are not the dense
# range 0..N-1, e.g. {5, 9}) cannot be baked by default: writing the frames
# compacts them to 0..k-1, so any labeled frame referencing a source index
# (5, 9) would dangle. Refuse with a clear error rather than crash or
# silently misalign. An explicit frame_inds bypasses this for advanced use.
inner = getattr(self.backend, "inner", None)
frame_map = getattr(inner, "frame_map", None)
if frame_map:
keys = sorted(frame_map.keys())
if keys != list(range(len(keys))):
raise ValueError(
"Cannot bake a virtual crop over a video with sparsely "
f"embedded frames (frame_map keys {keys}): baking would "
"compact frames to a contiguous range and break frame_idx "
"references. Pass explicit frame_inds to override, or "
"materialize from the original source video."
)
frame_inds = np.arange(len(self))
# Use this video's FPS if not explicitly specified.
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
baked = Video.from_filename(path, grayscale=self.grayscale)
# Provenance: the uncropped original. Walk past any still-virtual crop
# ancestors (a flattened crop-of-crop's source_video may itself be a crop)
# to the first uncropped ancestor. For a manually-built crop with no parent,
# reconstruct an uncropped view from the crop backend's inner, so
# source_video is never a cropped video.
source = self.source_video
while source is not None and source._crop_tuple() is not None:
source = source.source_video
if source is None:
inner = getattr(self.backend, "inner", None)
source = (
Video(filename=inner.filename, backend=inner)
if inner is not None
else self
)
baked.source_video = source
return baked
close()
¶
Close the video backend.
Source code in sleap_io/model/video.py
def close(self):
"""Close the video backend."""
if self.backend is not None:
# Try to remember values from previous backend if available and not
# specified.
try:
self.backend_metadata["dataset"] = getattr(
self.backend, "dataset", None
)
self.backend_metadata["grayscale"] = getattr(
self.backend, "grayscale", None
)
self.backend_metadata["shape"] = getattr(self.backend, "shape", None)
self.backend_metadata["fps"] = getattr(self.backend, "fps", None)
# Persist the crop so a Video cropped in-memory (never loaded
# from disk) survives a close()->open() and deepcopy: open()
# re-wraps from these keys (the closed-path shape above is
# already the cropped shape).
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
self.backend_metadata["crop"] = list(self.backend.crop)
self.backend_metadata["crop_fill"] = self.backend.fill
except Exception:
pass
# Deterministically release the backend's open handles (the cached
# reader and, for a remote HDF5Video, the fsspec URL file-like)
# rather than relying on garbage collection.
try:
self.backend.close()
except Exception:
pass
del self.backend
self.backend = None
crop(crop=None, *, bbox=None, roi=None, center=None, size=None, margin=0, fill=0, share_decode=True)
¶
Return a virtual, on-read cropped view of this video.
Exactly one region spec must be given: crop (explicit
(x1, y1, x2, y2) rect), bbox, roi (its axis-aligned bounds +
margin), or (center, size) for a fixed-size centered/
centroid-following window. The returned Video shares no pixels with
this one; frames are decoded on read and cropped (byte-identical to
:func:sleap_io.transform.frame.crop_frame). Out-of-bounds regions are
pad-filled with fill (never clamped), so the output shape is always
exactly (y2 - y1, x2 - x1).
The crop composes (FLATTENS when fills agree and the region is in-bounds)
with any existing crop on this video via
:meth:CropVideoBackend.wrap. source_video is set to this video for
provenance. When share_decode (the default), the new crop reuses this
video's backend instance as the shared inner so a mosaic of tiles over
one file decodes each source frame once; in that case the new tile does
NOT own the shared decoder (this video does).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
crop
|
tuple[int, int, int, int] | None
|
Explicit crop region |
None
|
bbox
|
tuple[float, float, float, float] | None
|
A bounding box |
None
|
roi
|
object | None
|
Any object exposing axis-aligned |
None
|
center
|
tuple[float, float] | None
|
Window center |
None
|
size
|
tuple[int, int] | None
|
Fixed output |
None
|
margin
|
int
|
Pixels added around the |
0
|
fill
|
int | tuple[int, ...]
|
Fill value for out-of-bounds regions. |
0
|
share_decode
|
bool
|
If |
True
|
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Source code in sleap_io/model/video.py
def crop(
self,
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
) -> "Video":
"""Return a virtual, on-read cropped view of this video.
Exactly one region spec must be given: ``crop`` (explicit
``(x1, y1, x2, y2)`` rect), ``bbox``, ``roi`` (its axis-aligned bounds +
``margin``), or (``center``, ``size``) for a fixed-size centered/
centroid-following window. The returned ``Video`` shares no pixels with
this one; frames are decoded on read and cropped (byte-identical to
:func:`sleap_io.transform.frame.crop_frame`). Out-of-bounds regions are
pad-filled with ``fill`` (never clamped), so the output shape is always
exactly ``(y2 - y1, x2 - x1)``.
The crop composes (FLATTENS when fills agree and the region is in-bounds)
with any existing crop on this video via
:meth:`CropVideoBackend.wrap`. ``source_video`` is set to this video for
provenance. When ``share_decode`` (the default), the new crop reuses this
video's backend instance as the shared inner so a mosaic of tiles over
one file decodes each source frame once; in that case the new tile does
NOT own the shared decoder (this video does).
Args:
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2``
exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: Any object exposing axis-aligned ``.bounds`` as
``(minx, miny, maxx, maxy)`` (e.g. a shapely geometry).
center: Window center ``(cx, cy)`` (used with ``size``).
size: Fixed output ``(width, height)`` (used with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (the default), reuse this video's backend
as the shared inner so tiles decode each frame once; the new tile
does not own the shared decoder.
Returns:
A new ``Video`` exposing the cropped view.
"""
from sleap_io.io.video_reading import CropVideoBackend
rect = _resolve_crop_rect(crop, bbox, roi, center, size, margin)
if self.backend is None and self.open_backend:
self.open()
if self.backend is None:
raise ValueError(
"Cannot crop a video with no open backend. Open it first (set "
"open_backend=True or call .open()) before cropping."
)
inner = self.backend
cropped_backend = CropVideoBackend.wrap(
inner=inner, crop=rect, fill=fill, owns_inner=not share_decode
)
cropped = Video(
filename=self.filename,
backend=cropped_backend,
source_video=self,
open_backend=self.open_backend,
)
x1, y1, x2, y2 = cropped_backend.crop
src_shape = self.shape
cropped.backend_metadata = {
**self.backend_metadata,
"shape": (src_shape[0], y2 - y1, x2 - x1, src_shape[3])
if src_shape is not None
else None,
# The uncropped source shape, so a closed re-serialize keeps videos_json
# describing the full frame even without a live source_video (D-120/DI-2).
"source_shape": list(src_shape) if src_shape is not None else None,
# COMPOSED source rect from wrap (D-120): keeps open/closed crop keys
# identical and root-canonical, and survives close()->open().
"crop": list(cropped_backend.crop),
"crop_fill": cropped_backend.fill,
}
return cropped
deduplicate_with(other)
¶
Create a new video with duplicate images removed.
This method is specifically for ImageVideo backends (image sequences).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to deduplicate against. Must also be ImageVideo. |
required |
Returns:
| Type | Description |
|---|---|
Video
|
A new Video object with duplicate images removed from this video, or None if all images were duplicates. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If either video is not an ImageVideo backend. |
Notes
Only works with ImageVideo backends where filename is a list. Images are considered duplicates if they have the same basename. The returned video contains only images from this video that are not present in the other video.
Source code in sleap_io/model/video.py
def deduplicate_with(self, other: "Video") -> "Video":
"""Create a new video with duplicate images removed.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to deduplicate against. Must also be ImageVideo.
Returns:
A new Video object with duplicate images removed from this video,
or None if all images were duplicates.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
Images are considered duplicates if they have the same basename.
The returned video contains only images from this video that are
not present in the other video.
"""
if not isinstance(self.filename, list):
raise ValueError("deduplicate_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get basenames from other video
other_basenames = set(Path(f).name for f in other.filename)
# Keep only non-duplicate images
deduplicated_paths = [
f for f in self.filename if Path(f).name not in other_basenames
]
if not deduplicated_paths:
# All images were duplicates
return None
# Create new video with deduplicated images
return Video.from_filename(deduplicated_paths, grayscale=self.grayscale)
exists(check_all=False, dataset=None)
¶
Check if the video file exists and is accessible.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
check_all
|
bool
|
If |
False
|
dataset
|
str | None
|
Name of dataset in HDF5 file. If specified, this will function will
return |
None
|
Returns:
| Type | Description |
|---|---|
bool
|
|
Source code in sleap_io/model/video.py
def exists(self, check_all: bool = False, dataset: str | None = None) -> bool:
"""Check if the video file exists and is accessible.
Args:
check_all: If `True`, check that all filenames in a list exist. If `False`
(the default), check that the first filename exists.
dataset: Name of dataset in HDF5 file. If specified, this will function will
return `False` if the dataset does not exist.
Returns:
`True` if the file exists and is accessible, `False` otherwise.
"""
if isinstance(self.filename, list):
if check_all:
for f in self.filename:
if not is_file_accessible(f):
return False
return True
else:
return is_file_accessible(self.filename[0])
# URL fast path: must run BEFORE `is_file_accessible`, which treats the
# filename as a local path and would spuriously return False for a URL.
from sleap_io.io._remote import _is_url
if _is_url(self.filename):
return self._url_exists(dataset)
file_is_accessible = is_file_accessible(self.filename)
if not file_is_accessible:
# Check if it's a directory (ImageVideo source)
if Path(self.filename).is_dir():
return True
return False
if dataset is None or dataset == "":
dataset = self.backend_metadata.get("dataset", None)
if dataset is not None and dataset != "":
has_dataset = False
if (
self.backend is not None
and type(self.backend) is HDF5Video
and self.backend._open_reader is not None
):
has_dataset = dataset in self.backend._open_reader
else:
with h5py.File(self.filename, "r") as f:
has_dataset = dataset in f
return has_dataset
return True
frame_to_seconds(frame_idx)
¶
Convert a frame index to timestamp in seconds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame_idx
|
int
|
Zero-indexed frame number. |
required |
Returns:
| Type | Description |
|---|---|
float | None
|
Time in seconds, or None if FPS is unknown. |
Notes
This assumes constant frame rate. For variable frame rate videos, the returned timestamp may be approximate.
Source code in sleap_io/model/video.py
def frame_to_seconds(self, frame_idx: int) -> float | None:
"""Convert a frame index to timestamp in seconds.
Args:
frame_idx: Zero-indexed frame number.
Returns:
Time in seconds, or None if FPS is unknown.
Notes:
This assumes constant frame rate. For variable frame rate videos,
the returned timestamp may be approximate.
"""
if self.fps is None or self.fps <= 0:
return None
return frame_idx / self.fps
from_crop(video, crop=None, *, bbox=None, roi=None, center=None, size=None, margin=0, fill=0, share_decode=True, **kwargs)
classmethod
¶
Open video (path or Video) and return a virtual crop.
Accepts the same region specs as :meth:crop (crop/bbox/roi/
center+size); extra keyword arguments are forwarded to
:meth:from_filename when video is a path (ignored when it is already
a Video).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
video
|
str | Path | Video
|
A path/filename to open, or an existing |
required |
crop
|
tuple[int, int, int, int] | None
|
Explicit crop region |
None
|
bbox
|
tuple[float, float, float, float] | None
|
A bounding box |
None
|
roi
|
object | None
|
An object exposing axis-aligned |
None
|
center
|
tuple[float, float] | None
|
Window center |
None
|
size
|
tuple[int, int] | None
|
Fixed output |
None
|
margin
|
int
|
Pixels added around the |
0
|
fill
|
int | tuple[int, ...]
|
Fill value for out-of-bounds regions. |
0
|
share_decode
|
bool
|
If |
True
|
**kwargs
|
Forwarded to :meth: |
required |
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Source code in sleap_io/model/video.py
@classmethod
def from_crop(
cls,
video: "str | Path | Video",
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
**kwargs,
) -> "Video":
"""Open ``video`` (path or ``Video``) and return a virtual crop.
Accepts the same region specs as :meth:`crop` (``crop``/``bbox``/``roi``/
``center``+``size``); extra keyword arguments are forwarded to
:meth:`from_filename` when ``video`` is a path (ignored when it is already
a ``Video``).
Args:
video: A path/filename to open, or an existing ``Video`` to crop.
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2`` exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: An object exposing axis-aligned ``.bounds`` (e.g. a shapely
geometry); ``margin`` is applied around it.
center: Window center ``(cx, cy)`` (with ``size``).
size: Fixed output ``(width, height)`` (with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (default), reuse the source decoder.
**kwargs: Forwarded to :meth:`from_filename` for a path input.
Returns:
A new ``Video`` exposing the cropped view.
"""
if isinstance(video, (str, Path)):
video = cls.from_filename(video, **kwargs)
return video.crop(
crop,
bbox=bbox,
roi=roi,
center=center,
size=size,
margin=margin,
fill=fill,
share_decode=share_decode,
)
from_filename(filename, dataset=None, grayscale=None, keep_open=True, source_video=None, **kwargs)
classmethod
¶
Create a Video from a filename.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filename
|
str | list[str]
|
The filename(s) of the video. Supported extensions: "mp4", "avi", "mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif", "tiff", "bmp". If the filename is a list, a list of image filenames are expected. If filename is a folder, it will be searched for images. |
required |
dataset
|
str | None
|
Name of dataset in HDF5 file. |
None
|
grayscale
|
bool | None
|
Whether to force grayscale. If None, autodetect on first frame load. |
None
|
keep_open
|
bool
|
Whether to keep the video reader open between calls to read frames. If False, will close the reader after each call. If True (the default), it will keep the reader open and cache it for subsequent calls which may enhance the performance of reading multiple frames. |
True
|
source_video
|
Video | None
|
The source video object if this is a proxy video. This is present when the video contains an embedded subset of frames from another video. |
None
|
**kwargs
|
Additional backend-specific arguments passed to VideoBackend.from_filename. See VideoBackend.from_filename for supported arguments. |
required |
Returns:
| Type | Description |
|---|---|
VideoBackend
|
Video instance with the appropriate backend instantiated. |
Source code in sleap_io/model/video.py
@classmethod
def from_filename(
cls,
filename: str | list[str],
dataset: str | None = None,
grayscale: bool | None = None,
keep_open: bool = True,
source_video: "Video | None" = None,
**kwargs,
) -> VideoBackend:
"""Create a Video from a filename.
Args:
filename: The filename(s) of the video. Supported extensions: "mp4", "avi",
"mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",
"tiff", "bmp". If the filename is a list, a list of image filenames are
expected. If filename is a folder, it will be searched for images.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
source_video: The source video object if this is a proxy video. This is
present when the video contains an embedded subset of frames from
another video.
**kwargs: Additional backend-specific arguments passed to
VideoBackend.from_filename. See VideoBackend.from_filename for supported
arguments.
Returns:
Video instance with the appropriate backend instantiated.
"""
backend = VideoBackend.from_filename(
filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
**kwargs,
)
# If filename is a directory, VideoBackend.from_filename will expand it
# to a list of paths to images contained within the directory. In this
# case we want to use the expanded list as filename
return cls(
filename=backend.filename,
backend=backend,
source_video=source_video,
)
has_overlapping_images(other)
¶
Check if this video has overlapping images with another video.
This method is specifically for ImageVideo backends (image sequences).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if both are ImageVideo instances with overlapping image files. False if either video is not an ImageVideo or no overlap exists. |
Notes
Only works with ImageVideo backends where filename is a list. Compares individual image filenames (basenames only).
Source code in sleap_io/model/video.py
def has_overlapping_images(self, other: "Video") -> bool:
"""Check if this video has overlapping images with another video.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to compare with.
Returns:
True if both are ImageVideo instances with overlapping image files.
False if either video is not an ImageVideo or no overlap exists.
Notes:
Only works with ImageVideo backends where filename is a list.
Compares individual image filenames (basenames only).
"""
# Both must be image sequences
if not (isinstance(self.filename, list) and isinstance(other.filename, list)):
return False
# Get basenames for comparison
self_basenames = set(Path(f).name for f in self.filename)
other_basenames = set(Path(f).name for f in other.filename)
# Check if there's any overlap
return len(self_basenames & other_basenames) > 0
matches_content(other)
¶
Check if this video has the same content as another video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the videos have the same shape and backend type. |
Notes
This compares metadata like shape and backend type, not actual frame data.
Source code in sleap_io/model/video.py
def matches_content(self, other: "Video") -> bool:
"""Check if this video has the same content as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same shape and backend type.
Notes:
This compares metadata like shape and backend type, not actual frame data.
"""
# Compare shapes
self_shape = self.shape
other_shape = other.shape
if self_shape != other_shape:
return False
# Compare backend types
if self.backend is None and other.backend is None:
return True
elif self.backend is None or other.backend is None:
return False
return type(self.backend).__name__ == type(other.backend).__name__
matches_path(other, strict=False)
¶
Check if this video has the same path as another video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
strict
|
bool
|
If True, require exact path match. If False, consider videos with the same filename (basename) as matching. |
False
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the videos have matching paths, False otherwise. |
Notes
For HDF5 video backends (e.g., embedded videos in .pkg.slp files), matching prioritizes the source_filename attribute since multiple videos can share the same HDF5 file path but reference different source videos. Falls back to dataset name matching if source_filename is not available.
Source code in sleap_io/model/video.py
def matches_path(self, other: "Video", strict: bool = False) -> bool:
"""Check if this video has the same path as another video.
Args:
other: Another video to compare with.
strict: If True, require exact path match. If False, consider videos
with the same filename (basename) as matching.
Returns:
True if the videos have matching paths, False otherwise.
Notes:
For HDF5 video backends (e.g., embedded videos in .pkg.slp files),
matching prioritizes the source_filename attribute since multiple
videos can share the same HDF5 file path but reference different
source videos. Falls back to dataset name matching if source_filename
is not available.
"""
# Handle HDF5 backends specially - prioritize source_filename matching
self_is_hdf5 = isinstance(self.backend, HDF5Video)
other_is_hdf5 = isinstance(other.backend, HDF5Video)
if self_is_hdf5 and other_is_hdf5:
# Both are HDF5 videos - must match by BOTH source_filename AND dataset
# to distinguish different videos embedded in the same pkg.slp file
self_source = self.backend.source_filename
other_source = other.backend.source_filename
self_dataset = self.backend.dataset
other_dataset = other.backend.dataset
# If both have datasets, they must match
if self_dataset is not None and other_dataset is not None:
if self_dataset != other_dataset:
return False # Different datasets = different videos
# If both have source_filenames, compare them
if self_source is not None and other_source is not None:
if strict:
# For HDF5 videos, just compare normalized path strings
# (avoid slow resolve() on network paths)
return Path(self_source).as_posix() == Path(other_source).as_posix()
else:
return Path(self_source).name == Path(other_source).name
# If only datasets available (no source_filename), they must match
if self_dataset is not None and other_dataset is not None:
return self_dataset == other_dataset
# If neither source_filename nor dataset available, cannot match
return False
if isinstance(self.filename, list) and isinstance(other.filename, list):
# Both are image sequences
if strict:
return self.filename == other.filename
else:
# Compare basenames
self_basenames = [Path(f).name for f in self.filename]
other_basenames = [Path(f).name for f in other.filename]
return self_basenames == other_basenames
elif isinstance(self.filename, list) or isinstance(other.filename, list):
# One is image sequence, other is single file
return False
else:
# Both are single files - use resolve() for symlink handling
if strict:
p1, p2 = Path(self.filename), Path(other.filename)
# Fast string comparison first
if p1.as_posix() == p2.as_posix():
return True
# Only resolve if both exist locally (avoid slow network timeouts)
try:
if p1.exists() and p2.exists():
return p1.resolve() == p2.resolve()
except OSError:
pass
return False
else:
return Path(self.filename).name == Path(other.filename).name
matches_shape(other)
¶
Check if this video has the same shape as another video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the videos have the same height, width, and channels. |
Notes
This only compares spatial dimensions, not the number of frames.
Source code in sleap_io/model/video.py
def matches_shape(self, other: "Video") -> bool:
"""Check if this video has the same shape as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same height, width, and channels.
Notes:
This only compares spatial dimensions, not the number of frames.
"""
# Try to get shape from backend metadata first if shape is not available
if self.backend is None and "shape" in self.backend_metadata:
self_shape = self.backend_metadata["shape"]
else:
self_shape = self.shape
if other.backend is None and "shape" in other.backend_metadata:
other_shape = other.backend_metadata["shape"]
else:
other_shape = other.shape
# Handle None shapes
if self_shape is None or other_shape is None:
return False
# Compare only height, width, channels (not frames)
return self_shape[1:] == other_shape[1:]
merge_with(other)
¶
Merge another video's images into this one.
This method is specifically for ImageVideo backends (image sequences).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to merge with. Must also be ImageVideo. |
required |
Returns:
| Type | Description |
|---|---|
Video
|
A new Video object with unique images from both videos. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If either video is not an ImageVideo backend. |
Notes
Only works with ImageVideo backends where filename is a list. The merged video contains all unique images from both videos, with automatic deduplication based on image basename.
Source code in sleap_io/model/video.py
def merge_with(self, other: "Video") -> "Video":
"""Merge another video's images into this one.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to merge with. Must also be ImageVideo.
Returns:
A new Video object with unique images from both videos.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
The merged video contains all unique images from both videos,
with automatic deduplication based on image basename.
"""
if not isinstance(self.filename, list):
raise ValueError("merge_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get all unique images (by basename) preserving order
seen_basenames = set()
merged_paths = []
for path in self.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
for path in other.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
# Create new video with merged images
return Video.from_filename(merged_paths, grayscale=self.grayscale)
open(filename=None, dataset=None, grayscale=None, keep_open=True, plugin=None)
¶
Open the video backend for reading.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filename
|
str | None
|
Filename to open. If not specified, will use the filename set on the video object. |
None
|
dataset
|
str | None
|
Name of dataset in HDF5 file. |
None
|
grayscale
|
str | None
|
Whether to force grayscale. If None, autodetect on first frame load. |
None
|
keep_open
|
bool
|
Whether to keep the video reader open between calls to read frames. If False, will close the reader after each call. If True (the default), it will keep the reader open and cache it for subsequent calls which may enhance the performance of reading multiple frames. |
True
|
plugin
|
str | None
|
Video plugin to use for MediaVideo files. One of "opencv", "FFMPEG", or "pyav". Also accepts aliases (case-insensitive). If not specified, uses the backend metadata, global default, or auto-detection in that order. |
None
|
Notes
This is useful for opening the video backend to read frames and then closing it after reading all the necessary frames.
If the backend was already open, it will be closed before opening a new one. Values for the HDF5 dataset and grayscale will be remembered if not specified.
Source code in sleap_io/model/video.py
def open(
self,
filename: str | None = None,
dataset: str | None = None,
grayscale: str | None = None,
keep_open: bool = True,
plugin: str | None = None,
):
"""Open the video backend for reading.
Args:
filename: Filename to open. If not specified, will use the filename set on
the video object.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
plugin: Video plugin to use for MediaVideo files. One of "opencv",
"FFMPEG", or "pyav". Also accepts aliases (case-insensitive).
If not specified, uses the backend metadata, global default,
or auto-detection in that order.
Notes:
This is useful for opening the video backend to read frames and then closing
it after reading all the necessary frames.
If the backend was already open, it will be closed before opening a new one.
Values for the HDF5 dataset and grayscale will be remembered if not
specified.
"""
if filename is not None:
self.replace_filename(filename, open=False)
# Try to remember values from previous backend if available and not specified.
if self.backend is not None:
if dataset is None:
dataset = getattr(self.backend, "dataset", None)
if grayscale is None:
grayscale = getattr(self.backend, "grayscale", None)
else:
if dataset is None and "dataset" in self.backend_metadata:
dataset = self.backend_metadata["dataset"]
if grayscale is None:
if "grayscale" in self.backend_metadata:
grayscale = self.backend_metadata["grayscale"]
elif "shape" in self.backend_metadata:
grayscale = self.backend_metadata["shape"][-1] == 1
if not self.exists(dataset=dataset):
from sleap_io.io._remote import _is_url, _redact_url
# Redact credential-bearing URLs (e.g. presigned ``?token=`` links)
# so they never surface in tracebacks/logs. Local paths are shown
# verbatim.
name = (
_redact_url(self.filename)
if isinstance(self.filename, str) and _is_url(self.filename)
else self.filename
)
msg = f"Video does not exist or cannot be opened for reading: {name}"
if dataset is not None:
msg += f" (dataset: {dataset})"
raise FileNotFoundError(msg)
# Close previous backend if open.
self.close()
# Handle plugin parameter
backend_kwargs = {}
if plugin is not None:
from sleap_io.io.video_reading import normalize_plugin_name
plugin = normalize_plugin_name(plugin)
self.backend_metadata["plugin"] = plugin
if "plugin" in self.backend_metadata:
backend_kwargs["plugin"] = self.backend_metadata["plugin"]
# Create new backend. Forward the URL auth context so a reopened remote
# HDF5Video stays authenticated (the previous backend, and its headers,
# were dropped by self.close() above).
self.backend = VideoBackend.from_filename(
self.filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
url_headers=self._url_headers,
url_stream_mode=self._url_stream_mode,
**backend_kwargs,
)
# Re-wrap as a crop view if this video records a crop in its metadata.
# The rebuilt backend above is always a plain backend, so this wraps
# exactly once (idempotent across close()->open() and deepcopy).
if "crop" in self.backend_metadata:
from sleap_io.io.video_reading import CropVideoBackend
self.backend = CropVideoBackend.wrap(
inner=self.backend,
crop=tuple(self.backend_metadata["crop"]),
fill=self.backend_metadata.get("crop_fill", 0),
)
replace_filename(new_filename, open=True)
¶
Update the filename of the video, optionally opening the backend.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_filename
|
str | Path | list[str] | list[Path]
|
New filename to set for the video. |
required |
open
|
bool
|
If |
True
|
Source code in sleap_io/model/video.py
def replace_filename(
self, new_filename: str | Path | list[str] | list[Path], open: bool = True
):
"""Update the filename of the video, optionally opening the backend.
Args:
new_filename: New filename to set for the video.
open: If `True` (the default), open the backend with the new filename. If
the new filename does not exist, no error is raised.
"""
if isinstance(new_filename, Path):
new_filename = new_filename.as_posix()
if isinstance(new_filename, list):
new_filename = [
p.as_posix() if isinstance(p, Path) else p for p in new_filename
]
# A relink to a different file makes the recorded shape/grayscale/fps in
# ``backend_metadata`` stale: they describe the OLD file but the new file
# may have a different resolution/channels/frame rate. They must not be
# serialized under the new filename (regression from #483, where
# ``save_slp(prefer_metadata=True)`` prefers these recorded values), so
# invalidate them on a real relink and let them be recomputed from the new
# backend. The no-relink path leaves metadata untouched so golden
# byte-identical saves stay byte-identical.
filename_changed = new_filename != self.filename
self.filename = new_filename
self.backend_metadata["filename"] = new_filename
# Invalidate any cached URL existence results for the previous filename.
self._exists_cache.clear()
if open:
if self.exists():
self.open()
else:
self.close()
# Drop stale metadata AFTER (re)opening: ``open()`` internally calls
# ``close()``, which would otherwise re-stamp the OLD backend's
# shape/grayscale/fps back into ``backend_metadata``.
if filename_changed:
for key in ("shape", "grayscale", "fps"):
self.backend_metadata.pop(key, None)
save(save_path, frame_inds=None, fps=None, video_kwargs=None)
¶
Save video frames to a new video file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
save_path
|
str | Path
|
Path to the new video file. Should end in MP4. |
required |
frame_inds
|
list[int] | ndarray | None
|
Frame indices to save. Can be specified as a list or array of frame integers. If not specified, saves all video frames. |
None
|
fps
|
float | None
|
Frames per second for the output video. If not specified, uses the source video's FPS if available, otherwise defaults to 30. |
None
|
video_kwargs
|
dict[str, Any] | None
|
A dictionary of keyword arguments to provide to
|
None
|
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Source code in sleap_io/model/video.py
def save(
self,
save_path: str | Path,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Save video frames to a new video file.
Args:
save_path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to save. Can be specified as a list or array of
frame integers. If not specified, saves all video frames.
fps: Frames per second for the output video. If not specified, uses the
source video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
`sio.save_video` for video compression.
Returns:
A new `Video` object pointing to the new video file.
"""
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
frame_inds = np.arange(len(self)) if frame_inds is None else frame_inds
# Use source video FPS if not explicitly specified
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(save_path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
new_video = Video.from_filename(save_path, grayscale=self.grayscale)
return new_video
seconds_to_frame(seconds)
¶
Convert a timestamp in seconds to frame index.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seconds
|
float
|
Time in seconds from video start. |
required |
Returns:
| Type | Description |
|---|---|
int | None
|
Zero-indexed frame number (rounded down), or None if FPS unknown. |
Source code in sleap_io/model/video.py
def seconds_to_frame(self, seconds: float) -> int | None:
"""Convert a timestamp in seconds to frame index.
Args:
seconds: Time in seconds from video start.
Returns:
Zero-indexed frame number (rounded down), or None if FPS unknown.
"""
if self.fps is None or self.fps <= 0:
return None
return int(seconds * self.fps)
set_video_plugin(plugin)
¶
Set the video plugin and reopen the video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
plugin
|
str
|
Video plugin to use. One of "opencv", "FFMPEG", or "pyav". Also accepts aliases (case-insensitive). |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the video is not a MediaVideo type. |
Examples:
Source code in sleap_io/model/video.py
def set_video_plugin(self, plugin: str) -> None:
"""Set the video plugin and reopen the video.
Args:
plugin: Video plugin to use. One of "opencv", "FFMPEG", or "pyav".
Also accepts aliases (case-insensitive).
Raises:
ValueError: If the video is not a MediaVideo type.
Examples:
>>> video.set_video_plugin("opencv")
>>> video.set_video_plugin("CV2") # Same as "opencv"
"""
from sleap_io.io.video_reading import MediaVideo, normalize_plugin_name
if not self.filename.endswith(MediaVideo.EXTS):
raise ValueError(f"Cannot set plugin for non-media video: {self.filename}")
plugin = normalize_plugin_name(plugin)
# Close current backend if open
was_open = self.is_open
if was_open:
self.close()
# Update backend metadata
self.backend_metadata["plugin"] = plugin
# Reopen with new plugin if it was open
if was_open:
self.open()
to_crop_coords(points)
¶
Map source-frame (x, y) into this video's cropped frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
points
|
ndarray
|
Coordinate array of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Coordinates translated into the cropped frame. If this video is not
cropped, a copy of |
Source code in sleap_io/model/video.py
def to_crop_coords(self, points: np.ndarray) -> np.ndarray:
"""Map source-frame ``(x, y)`` into this video's cropped frame.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated into the cropped frame. If this video is not
cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else crop_points(points, crop)
to_source_coords(points)
¶
Map cropped-frame (x, y) back to source-frame coordinates.
Inverse of :meth:to_crop_coords.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
points
|
ndarray
|
Coordinate array of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Coordinates translated back to source coordinates. If this video is
not cropped, a copy of |
Source code in sleap_io/model/video.py
def to_source_coords(self, points: np.ndarray) -> np.ndarray:
"""Map cropped-frame ``(x, y)`` back to source-frame coordinates.
Inverse of :meth:`to_crop_coords`.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated back to source coordinates. If this video is
not cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else uncrop_points(points, crop)