camera
sleap_io.model.camera
¶
Data structure for a single camera view in a multi-camera setup.
Classes:
| Name | Description |
|---|---|
Camera |
A camera used to record in a multi-view |
CameraGroup |
A group of cameras used to record a multi-view |
Category |
Ground-truth class membership of a detection (e.g. species, sex, condition). |
FrameGroup |
Defines a group of |
Identity |
Ground-truth animal identity, persistent across sessions and videos. |
Instance |
This class represents a ground truth instance such as an animal. |
Instance3D |
A 3D pose instance with keypoints in world coordinates. |
InstanceGroup |
Defines a group of instances across the same frame index. |
LabeledFrame |
Labeled data for a single frame of a video. |
RecordingSession |
A recording session with multiple cameras. |
Video |
|
Functions:
| Name | Description |
|---|---|
rodrigues_transformation |
Convert between rotation vector and rotation matrix using Rodrigues' formula. |
Attributes:
| Name | Type | Description |
|---|---|---|
__cached__ |
str(object='') -> str |
|
__doc__ |
str(object='') -> str |
|
__file__ |
str(object='') -> str |
|
__name__ |
str(object='') -> str |
|
__package__ |
str(object='') -> str |
__cached__ = '/home/runner/work/sleap-io/sleap-io/sleap_io/model/__pycache__/camera.cpython-313.pyc'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__doc__ = 'Data structure for a single camera view in a multi-camera setup.'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__file__ = '/home/runner/work/sleap-io/sleap-io/sleap_io/model/camera.py'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__name__ = 'sleap_io.model.camera'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__package__ = 'sleap_io.model'
module-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
Camera
¶
A camera used to record in a multi-view RecordingSession.
Attributes:
| Name | Type | Description |
|---|---|---|
matrix |
Intrinsic camera matrix of size (3, 3) and type float64. |
|
dist |
Radial-tangential distortion coefficients [k_1, k_2, p_1, p_2, k_3] of size (5,) and type float64. |
|
size |
Image size (width, height) of camera in pixels of size (2,) and type int. |
|
rvec |
Rotation vector in unnormalized axis-angle representation of size (3,) and type float64. |
|
tvec |
Translation vector of size (3,) and type float64. |
|
extrinsic_matrix |
Extrinsic matrix of camera of size (4, 4) and type float64. |
|
name |
Camera name. |
|
metadata |
Dictionary of metadata. |
Methods:
| Name | Description |
|---|---|
__attrs_post_init__ |
Initialize extrinsic matrix from rotation and translation vectors. |
__init__ |
Method generated by attrs for class Camera. |
__repr__ |
Return a readable representation of the camera. |
__setattr__ |
Method generated by attrs for class Camera. |
get_video |
Get video associated with recording session. |
Source code in sleap_io/model/camera.py
@define(eq=False) # Set eq to false to make class hashable
class Camera:
"""A camera used to record in a multi-view `RecordingSession`.
Attributes:
matrix: Intrinsic camera matrix of size (3, 3) and type float64.
dist: Radial-tangential distortion coefficients [k_1, k_2, p_1, p_2, k_3] of
size (5,) and type float64.
size: Image size (width, height) of camera in pixels of size (2,) and type int.
rvec: Rotation vector in unnormalized axis-angle representation of size (3,) and
type float64.
tvec: Translation vector of size (3,) and type float64.
extrinsic_matrix: Extrinsic matrix of camera of size (4, 4) and type float64.
name: Camera name.
metadata: Dictionary of metadata.
"""
matrix: np.ndarray = field(
default=np.eye(3),
converter=lambda x: np.array(x, dtype="float64"),
)
dist: np.ndarray = field(
default=np.zeros(5), converter=lambda x: np.array(x, dtype="float64").ravel()
)
size: tuple[int, int] = field(
default=None, converter=attrs.converters.optional(tuple)
)
_rvec: np.ndarray = field(
default=np.zeros(3), converter=lambda x: np.array(x, dtype="float64").ravel()
)
_tvec: np.ndarray = field(
default=np.zeros(3), converter=lambda x: np.array(x, dtype="float64").ravel()
)
name: str = field(default=None, converter=attrs.converters.optional(str))
_extrinsic_matrix: np.ndarray = field(init=False)
metadata: dict = field(factory=dict, validator=instance_of(dict))
@matrix.validator
@dist.validator
@size.validator
@_rvec.validator
@_tvec.validator
@_extrinsic_matrix.validator
def _validate_shape(self, attribute: attrs.Attribute, value):
"""Validate shape of attribute based on metadata.
Args:
attribute: Attribute to validate.
value: Value of attribute to validate.
Raises:
ValueError: If attribute shape is not as expected.
"""
# Define metadata for each attribute
attr_metadata = {
"matrix": {"shape": (3, 3), "type": np.ndarray},
"dist": {"shape": (5,), "type": np.ndarray},
"size": {"shape": (2,), "type": tuple},
"_rvec": {"shape": (3,), "type": np.ndarray},
"_tvec": {"shape": (3,), "type": np.ndarray},
"_extrinsic_matrix": {"shape": (4, 4), "type": np.ndarray},
}
optional_attrs = ["size"]
# Skip validation if optional attribute is None
if attribute.name in optional_attrs and value is None:
return
# Validate shape of attribute
expected_shape = attr_metadata[attribute.name]["shape"]
expected_type = attr_metadata[attribute.name]["type"]
if np.shape(value) != expected_shape:
raise ValueError(
f"{attribute.name} must be a {expected_type} of size {expected_shape}, "
f"but received shape: {np.shape(value)} and type: {type(value)} for "
f"value: {value}"
)
def __attrs_post_init__(self):
"""Initialize extrinsic matrix from rotation and translation vectors."""
self._extrinsic_matrix = np.eye(4, dtype="float64")
self._extrinsic_matrix[:3, :3] = rodrigues_transformation(self._rvec)[0]
self._extrinsic_matrix[:3, 3] = self._tvec
@property
def rvec(self) -> np.ndarray:
"""Get rotation vector of camera.
Returns:
Rotation vector of camera of size 3.
"""
return self._rvec
@rvec.setter
def rvec(self, value: np.ndarray):
"""Set rotation vector and update extrinsic matrix.
Args:
value: Rotation vector of size 3.
"""
self._rvec = value
self._extrinsic_matrix[:3, :3] = rodrigues_transformation(self._rvec)[0]
@property
def tvec(self) -> np.ndarray:
"""Get translation vector of camera.
Returns:
Translation vector of camera of size 3.
"""
return self._tvec
@tvec.setter
def tvec(self, value: np.ndarray):
"""Set translation vector and update extrinsic matrix.
Args:
value: Translation vector of size 3.
"""
self._tvec = value
# Update extrinsic matrix
self._extrinsic_matrix[:3, 3] = self._tvec
@property
def extrinsic_matrix(self) -> np.ndarray:
"""Get extrinsic matrix of camera.
Returns:
Extrinsic matrix of camera of size 4 x 4.
"""
return self._extrinsic_matrix
@extrinsic_matrix.setter
def extrinsic_matrix(self, value: np.ndarray):
"""Set extrinsic matrix and update rotation and translation vectors.
Args:
value: Extrinsic matrix of size 4 x 4.
"""
self._extrinsic_matrix = value
# Update rotation and translation vectors
self._rvec = rodrigues_transformation(self._extrinsic_matrix[:3, :3])[0].ravel()
self._tvec = self._extrinsic_matrix[:3, 3]
def get_video(self, session: RecordingSession) -> Video | None:
"""Get video associated with recording session.
Args:
session: Recording session to get video for.
Returns:
Video associated with recording session or None if not found.
"""
return session.get_video(camera=self)
def __repr__(self) -> str:
"""Return a readable representation of the camera."""
matrix_str = (
"identity" if np.array_equal(self.matrix, np.eye(3)) else "non-identity"
)
dist_str = "zero" if np.array_equal(self.dist, np.zeros(5)) else "non-zero"
size_str = "None" if self.size is None else self.size
rvec_str = (
"zero"
if np.array_equal(self.rvec, np.zeros(3))
else np.array2string(self.rvec, precision=2, suppress_small=True)
)
tvec_str = (
"zero"
if np.array_equal(self.tvec, np.zeros(3))
else np.array2string(self.tvec, precision=2, suppress_small=True)
)
name_str = self.name if self.name is not None else "None"
return (
"Camera("
f"matrix={matrix_str}, "
f"dist={dist_str}, "
f"size={size_str}, "
f"rvec={rvec_str}, "
f"tvec={tvec_str}, "
f"name={name_str}"
")"
)
__annotations__ = {'matrix': 'np.ndarray', 'dist': 'np.ndarray', 'size': 'tuple[int, int]', '_rvec': 'np.ndarray', '_tvec': 'np.ndarray', 'name': 'str', '_extrinsic_matrix': 'np.ndarray', 'metadata': 'dict'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'A camera used to record in a multi-view `RecordingSession`.\n\nAttributes:\n matrix: Intrinsic camera matrix of size (3, 3) and type float64.\n dist: Radial-tangential distortion coefficients [k_1, k_2, p_1, p_2, k_3] of\n size (5,) and type float64.\n size: Image size (width, height) of camera in pixels of size (2,) and type int.\n rvec: Rotation vector in unnormalized axis-angle representation of size (3,) and\n type float64.\n tvec: Translation vector of size (3,) and type float64.\n extrinsic_matrix: Extrinsic matrix of camera of size (4, 4) and type float64.\n name: Camera name.\n metadata: Dictionary of metadata.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 257
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('matrix', 'dist', 'size', '_rvec', '_tvec', 'name', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.camera'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('matrix', 'dist', 'size', '_rvec', '_tvec', 'name', '_extrinsic_matrix', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('_extrinsic_matrix', '_rvec', '_tvec')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
extrinsic_matrix
property
¶
Get extrinsic matrix of camera.
Returns:
| Type | Description |
|---|---|
|
Extrinsic matrix of camera of size 4 x 4. |
rvec
property
¶
Get rotation vector of camera.
Returns:
| Type | Description |
|---|---|
|
Rotation vector of camera of size 3. |
tvec
property
¶
Get translation vector of camera.
Returns:
| Type | Description |
|---|---|
|
Translation vector of camera of size 3. |
__attrs_post_init__()
¶
Initialize extrinsic matrix from rotation and translation vectors.
Source code in sleap_io/model/camera.py
__init__(matrix=array([[1., 0., 0.],[0., 1., 0.],[0., 0., 1.]]), dist=array([0., 0., 0., 0., 0.]), size=None, rvec=array([0., 0., 0.]), tvec=array([0., 0., 0.]), name=None, metadata=NOTHING)
¶
Method generated by attrs for class Camera.
Source code in sleap_io/model/camera.py
"""Data structure for a single camera view in a multi-camera setup."""
from __future__ import annotations
import attrs
import numpy as np
from attrs import define, field
from attrs.validators import instance_of
from sleap_io.model.category import Category
from sleap_io.model.identity import Identity
from sleap_io.model.instance import Instance, Instance3D
from sleap_io.model.labeled_frame import LabeledFrame
from sleap_io.model.video import Video
def rodrigues_transformation(input_matrix: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
"""Convert between rotation vector and rotation matrix using Rodrigues' formula.
This function implements the Rodrigues' rotation formula to convert between:
__repr__()
¶
Return a readable representation of the camera.
Source code in sleap_io/model/camera.py
def __repr__(self) -> str:
"""Return a readable representation of the camera."""
matrix_str = (
"identity" if np.array_equal(self.matrix, np.eye(3)) else "non-identity"
)
dist_str = "zero" if np.array_equal(self.dist, np.zeros(5)) else "non-zero"
size_str = "None" if self.size is None else self.size
rvec_str = (
"zero"
if np.array_equal(self.rvec, np.zeros(3))
else np.array2string(self.rvec, precision=2, suppress_small=True)
)
tvec_str = (
"zero"
if np.array_equal(self.tvec, np.zeros(3))
else np.array2string(self.tvec, precision=2, suppress_small=True)
)
name_str = self.name if self.name is not None else "None"
return (
"Camera("
f"matrix={matrix_str}, "
f"dist={dist_str}, "
f"size={size_str}, "
f"rvec={rvec_str}, "
f"tvec={tvec_str}, "
f"name={name_str}"
")"
)
__setattr__(name, val)
¶
Method generated by attrs for class Camera.
get_video(session)
¶
Get video associated with recording session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
session
|
RecordingSession
|
Recording session to get video for. |
required |
Returns:
| Type | Description |
|---|---|
Video | None
|
Video associated with recording session or None if not found. |
Source code in sleap_io/model/camera.py
CameraGroup
¶
A group of cameras used to record a multi-view RecordingSession.
Attributes:
| Name | Type | Description |
|---|---|---|
cameras |
List of |
|
metadata |
Dictionary of metadata. |
Methods:
| Name | Description |
|---|---|
__eq__ |
Method generated by attrs for class CameraGroup. |
__init__ |
Method generated by attrs for class CameraGroup. |
__repr__ |
Return a readable representation of the camera group. |
__setattr__ |
Method generated by attrs for class CameraGroup. |
Source code in sleap_io/model/camera.py
@define
class CameraGroup:
"""A group of cameras used to record a multi-view `RecordingSession`.
Attributes:
cameras: List of `Camera` objects in the group.
metadata: Dictionary of metadata.
"""
cameras: "list[Camera]" = field(factory=list, validator=instance_of(list))
metadata: dict = field(factory=dict, validator=instance_of(dict))
def __repr__(self):
"""Return a readable representation of the camera group."""
camera_names = ", ".join([c.name or "None" for c in self.cameras])
return f"CameraGroup(cameras={len(self.cameras)}:[{camera_names}])"
__annotations__ = {'cameras': "'list[Camera]'", 'metadata': 'dict'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=True, added_ordering=False, hashability=<Hashability.UNHASHABLE: 'unhashable'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'A group of cameras used to record a multi-view `RecordingSession`.\n\nAttributes:\n cameras: List of `Camera` objects in the group.\n metadata: Dictionary of metadata.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 112
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('cameras', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.camera'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('cameras', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ()
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
__eq__(other)
¶
__init__(cameras=NOTHING, metadata=NOTHING)
¶
Method generated by attrs for class CameraGroup.
Source code in sleap_io/model/camera.py
from attrs.validators import instance_of
from sleap_io.model.category import Category
from sleap_io.model.identity import Identity
from sleap_io.model.instance import Instance, Instance3D
from sleap_io.model.labeled_frame import LabeledFrame
from sleap_io.model.video import Video
def rodrigues_transformation(input_matrix: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
"""Convert between rotation vector and rotation matrix using Rodrigues' formula.
This function implements the Rodrigues' rotation formula to convert between:
__repr__()
¶
Return a readable representation of the camera group.
__setattr__(name, val)
¶
Method generated by attrs for class CameraGroup.
Category
¶
Ground-truth class membership of a detection (e.g. species, sex, condition).
Where Track is an ephemeral temporal trajectory within a single video and
Identity names a specific individual across videos, Category names the
class an individual belongs to -- a group of individuals that share some
attribute, typically assigned by classification or retrieved via re-ID (e.g.
"female_fly", "fur_shaved", "mouse"). The per-detection binding is
stored on Instance.category (and the analogous slot on the other detection
modalities), alongside an optional category_score (assignment confidence)
and category_embedding (the appearance vector it was classified from).
Attributes:
| Name | Type | Description |
|---|---|---|
name |
Human-readable name for this category (e.g., |
|
metadata |
Arbitrary string-keyed, string-valued metadata (e.g.
|
Notes
Category objects use object-identity equality (eq=False), matching
Track and Identity. Use matches() (default method="name") to
compare categories across files, where Python object identity is not
meaningful.
Methods:
| Name | Description |
|---|---|
__init__ |
Method generated by attrs for class Category. |
__repr__ |
Return a readable string representation. |
__setattr__ |
Method generated by attrs for class Category. |
matches |
Check if this category matches another category. |
Source code in sleap_io/model/category.py
@define(eq=False)
class Category:
"""Ground-truth class membership of a detection (e.g. species, sex, condition).
Where `Track` is an ephemeral temporal trajectory within a single video and
`Identity` names a specific individual across videos, `Category` names the
*class* an individual belongs to -- a group of individuals that share some
attribute, typically assigned by classification or retrieved via re-ID (e.g.
``"female_fly"``, ``"fur_shaved"``, ``"mouse"``). The per-detection binding is
stored on ``Instance.category`` (and the analogous slot on the other detection
modalities), alongside an optional ``category_score`` (assignment confidence)
and ``category_embedding`` (the appearance vector it was classified from).
Attributes:
name: Human-readable name for this category (e.g., ``"female_fly"``). Not
required to be unique, but ``name`` is how categories are matched
across separately-loaded files and merges.
metadata: Arbitrary string-keyed, string-valued metadata (e.g.
``{"color": "#e6194b", "supercategory": "insect"}``). Empty by default.
Notes:
`Category` objects use object-identity equality (``eq=False``), matching
`Track` and `Identity`. Use `matches()` (default ``method="name"``) to
compare categories across files, where Python object identity is not
meaningful.
"""
name: str = field(default="", validator=instance_of(str))
metadata: dict[str, str] = field(factory=dict, validator=instance_of(dict))
def matches(self, other: "Category", method: str = "name") -> bool:
"""Check if this category matches another category.
Args:
other: Another category to compare with.
method: Matching method:
- ``"name"`` (default): match by the `name` attribute, which
survives serialization and cross-file merges.
- ``"identity"``: match by Python object identity (same object).
Returns:
True if the categories match according to the specified method.
Raises:
ValueError: If `method` is not one of the supported values.
"""
if method == "name":
return self.name == other.name
elif method == "identity":
return self is other
else:
raise ValueError(f"Unknown matching method: {method}")
def __repr__(self) -> str:
"""Return a readable string representation."""
return f'Category(name="{self.name}")'
__annotations__ = {'name': 'str', 'metadata': 'dict[str, str]'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'Ground-truth class membership of a detection (e.g. species, sex, condition).\n\nWhere `Track` is an ephemeral temporal trajectory within a single video and\n`Identity` names a specific individual across videos, `Category` names the\n*class* an individual belongs to -- a group of individuals that share some\nattribute, typically assigned by classification or retrieved via re-ID (e.g.\n``"female_fly"``, ``"fur_shaved"``, ``"mouse"``). The per-detection binding is\nstored on ``Instance.category`` (and the analogous slot on the other detection\nmodalities), alongside an optional ``category_score`` (assignment confidence)\nand ``category_embedding`` (the appearance vector it was classified from).\n\nAttributes:\n name: Human-readable name for this category (e.g., ``"female_fly"``). Not\n required to be unique, but ``name`` is how categories are matched\n across separately-loaded files and merges.\n metadata: Arbitrary string-keyed, string-valued metadata (e.g.\n ``{"color": "#e6194b", "supercategory": "insect"}``). Empty by default.\n\nNotes:\n `Category` objects use object-identity equality (``eq=False``), matching\n `Track` and `Identity`. Use `matches()` (default ``method="name"``) to\n compare categories across files, where Python object identity is not\n meaningful.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 9
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('name', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.category'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('name', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ()
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
__init__(name='', metadata=NOTHING)
¶
Method generated by attrs for class Category.
__repr__()
¶
__setattr__(name, val)
¶
Method generated by attrs for class Category.
matches(other, method='name')
¶
Check if this category matches another category.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Category
|
Another category to compare with. |
required |
method
|
str
|
Matching method:
|
'name'
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the categories match according to the specified method. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in sleap_io/model/category.py
def matches(self, other: "Category", method: str = "name") -> bool:
"""Check if this category matches another category.
Args:
other: Another category to compare with.
method: Matching method:
- ``"name"`` (default): match by the `name` attribute, which
survives serialization and cross-file merges.
- ``"identity"``: match by Python object identity (same object).
Returns:
True if the categories match according to the specified method.
Raises:
ValueError: If `method` is not one of the supported values.
"""
if method == "name":
return self.name == other.name
elif method == "identity":
return self is other
else:
raise ValueError(f"Unknown matching method: {method}")
FrameGroup
¶
Defines a group of InstanceGroups across views at the same frame index.
Attributes:
| Name | Type | Description |
|---|---|---|
frame_idx |
Frame index for the |
|
instance_groups |
List of |
|
cameras |
List of |
|
labeled_frames |
List of |
|
metadata |
Metadata for the |
Methods:
| Name | Description |
|---|---|
__init__ |
Method generated by attrs for class FrameGroup. |
__repr__ |
Return a readable representation of the frame group. |
__setattr__ |
Method generated by attrs for class FrameGroup. |
get_frame |
Get |
Source code in sleap_io/model/camera.py
@define(eq=False) # Set eq to false to make class hashable
class FrameGroup:
"""Defines a group of `InstanceGroups` across views at the same frame index.
Attributes:
frame_idx: Frame index for the `FrameGroup`.
instance_groups: List of `InstanceGroup`s in the `FrameGroup`.
cameras: List of `Camera` objects linked to `LabeledFrame`s in the `FrameGroup`.
labeled_frames: List of `LabeledFrame`s in the `FrameGroup`.
metadata: Metadata for the `FrameGroup` that is provided but not deserialized.
"""
frame_idx: int = field(converter=int)
_instance_groups: list[InstanceGroup] = field(
factory=list, validator=instance_of(list)
)
_labeled_frame_by_camera: dict[Camera, LabeledFrame] = field(
factory=dict, validator=instance_of(dict)
)
metadata: dict = field(factory=dict, validator=instance_of(dict))
@property
def instance_groups(self) -> list[InstanceGroup]:
"""List of `InstanceGroup`s."""
return self._instance_groups
@property
def cameras(self) -> "list[Camera]":
"""List of `Camera` objects."""
return list(self._labeled_frame_by_camera.keys())
@property
def labeled_frames(self) -> list[LabeledFrame]:
"""List of `LabeledFrame`s."""
return list(self._labeled_frame_by_camera.values())
def get_frame(self, camera: Camera) -> LabeledFrame | None:
"""Get `LabeledFrame` associated with `camera`.
Args:
camera: `Camera` to get `LabeledFrame`.
Returns:
`LabeledFrame` associated with `camera` or None if not found.
"""
return self._labeled_frame_by_camera.get(camera, None)
def __repr__(self) -> str:
"""Return a readable representation of the frame group."""
cameras_str = ", ".join([c.name or "None" for c in self.cameras])
return (
f"FrameGroup("
f"frame_idx={self.frame_idx},"
f"instance_groups={len(self.instance_groups)},"
f"cameras={len(self.cameras)}:[{cameras_str}]"
f")"
)
__annotations__ = {'frame_idx': 'int', '_instance_groups': 'list[InstanceGroup]', '_labeled_frame_by_camera': 'dict[Camera, LabeledFrame]', 'metadata': 'dict'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'Defines a group of `InstanceGroups` across views at the same frame index.\n\nAttributes:\n frame_idx: Frame index for the `FrameGroup`.\n instance_groups: List of `InstanceGroup`s in the `FrameGroup`.\n cameras: List of `Camera` objects linked to `LabeledFrame`s in the `FrameGroup`.\n labeled_frames: List of `LabeledFrame`s in the `FrameGroup`.\n metadata: Metadata for the `FrameGroup` that is provided but not deserialized.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 551
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('frame_idx', '_instance_groups', '_labeled_frame_by_camera', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.camera'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('frame_idx', '_instance_groups', '_labeled_frame_by_camera', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ()
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
cameras
property
¶
List of Camera objects.
instance_groups
property
¶
List of InstanceGroups.
labeled_frames
property
¶
List of LabeledFrames.
__init__(frame_idx, instance_groups=NOTHING, labeled_frame_by_camera=NOTHING, metadata=NOTHING)
¶
Method generated by attrs for class FrameGroup.
Source code in sleap_io/model/camera.py
"""Data structure for a single camera view in a multi-camera setup."""
from __future__ import annotations
import attrs
import numpy as np
from attrs import define, field
from attrs.validators import instance_of
from sleap_io.model.category import Category
from sleap_io.model.identity import Identity
from sleap_io.model.instance import Instance, Instance3D
from sleap_io.model.labeled_frame import LabeledFrame
from sleap_io.model.video import Video
def rodrigues_transformation(input_matrix: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
"""Convert between rotation vector and rotation matrix using Rodrigues' formula.
__repr__()
¶
Return a readable representation of the frame group.
Source code in sleap_io/model/camera.py
def __repr__(self) -> str:
"""Return a readable representation of the frame group."""
cameras_str = ", ".join([c.name or "None" for c in self.cameras])
return (
f"FrameGroup("
f"frame_idx={self.frame_idx},"
f"instance_groups={len(self.instance_groups)},"
f"cameras={len(self.cameras)}:[{cameras_str}]"
f")"
)
__setattr__(name, val)
¶
Method generated by attrs for class FrameGroup.
get_frame(camera)
¶
Get LabeledFrame associated with camera.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
camera
|
Camera
|
|
required |
Returns:
| Type | Description |
|---|---|
LabeledFrame | None
|
|
Source code in sleap_io/model/camera.py
Identity
¶
Ground-truth animal identity, persistent across sessions and videos.
Unlike Track (an ephemeral temporal trajectory within a single video),
Identity represents a known animal that can be recognized across videos,
sessions, and experiments. In multi-view setups, multiple per-camera Tracks
may map to a single Identity. The per-detection binding is stored on
Instance.identity (and the analogous slot on the other detection
modalities); the triangulated multi-view binding is stored on
InstanceGroup.identity.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
Human-readable name for this identity (e.g., |
|
metadata |
Arbitrary string-keyed, string-valued metadata (e.g.
|
Notes
Identity objects use object-identity equality (eq=False), matching
Track. Use matches() (default method="name") to compare identities
across files, where Python object identity is not meaningful.
Methods:
| Name | Description |
|---|---|
__init__ |
Method generated by attrs for class Identity. |
__repr__ |
Return a readable string representation. |
__setattr__ |
Method generated by attrs for class Identity. |
matches |
Check if this identity matches another identity. |
Source code in sleap_io/model/identity.py
@define(eq=False)
class Identity:
"""Ground-truth animal identity, persistent across sessions and videos.
Unlike `Track` (an ephemeral temporal trajectory within a single video),
`Identity` represents a known animal that can be recognized across videos,
sessions, and experiments. In multi-view setups, multiple per-camera `Track`s
may map to a single `Identity`. The per-detection binding is stored on
``Instance.identity`` (and the analogous slot on the other detection
modalities); the triangulated multi-view binding is stored on
``InstanceGroup.identity``.
Attributes:
name: Human-readable name for this identity (e.g., ``"mouse_A"``). Not
required to be unique, but ``name`` is how identities are matched
across separately-loaded files and merges.
metadata: Arbitrary string-keyed, string-valued metadata (e.g.
``{"color": "#e6194b", "strain": "C57BL/6"}``). Empty by default.
Notes:
`Identity` objects use object-identity equality (``eq=False``), matching
`Track`. Use `matches()` (default ``method="name"``) to compare identities
across files, where Python object identity is not meaningful.
"""
name: str = field(default="", validator=instance_of(str))
metadata: dict[str, str] = field(factory=dict, validator=instance_of(dict))
def matches(self, other: "Identity", method: str = "name") -> bool:
"""Check if this identity matches another identity.
Args:
other: Another identity to compare with.
method: Matching method:
- ``"name"`` (default): match by the `name` attribute, which
survives serialization and cross-file merges.
- ``"identity"``: match by Python object identity (same object).
Returns:
True if the identities match according to the specified method.
Raises:
ValueError: If `method` is not one of the supported values.
"""
if method == "name":
return self.name == other.name
elif method == "identity":
return self is other
else:
raise ValueError(f"Unknown matching method: {method}")
def __repr__(self) -> str:
"""Return a readable string representation."""
return f'Identity(name="{self.name}")'
__annotations__ = {'name': 'str', 'metadata': 'dict[str, str]'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'Ground-truth animal identity, persistent across sessions and videos.\n\nUnlike `Track` (an ephemeral temporal trajectory within a single video),\n`Identity` represents a known animal that can be recognized across videos,\nsessions, and experiments. In multi-view setups, multiple per-camera `Track`s\nmay map to a single `Identity`. The per-detection binding is stored on\n``Instance.identity`` (and the analogous slot on the other detection\nmodalities); the triangulated multi-view binding is stored on\n``InstanceGroup.identity``.\n\nAttributes:\n name: Human-readable name for this identity (e.g., ``"mouse_A"``). Not\n required to be unique, but ``name`` is how identities are matched\n across separately-loaded files and merges.\n metadata: Arbitrary string-keyed, string-valued metadata (e.g.\n ``{"color": "#e6194b", "strain": "C57BL/6"}``). Empty by default.\n\nNotes:\n `Identity` objects use object-identity equality (``eq=False``), matching\n `Track`. Use `matches()` (default ``method="name"``) to compare identities\n across files, where Python object identity is not meaningful.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 9
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('name', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.identity'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('name', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ()
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
__init__(name='', metadata=NOTHING)
¶
__repr__()
¶
__setattr__(name, val)
¶
Method generated by attrs for class Identity.
matches(other, method='name')
¶
Check if this identity matches another identity.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Identity
|
Another identity to compare with. |
required |
method
|
str
|
Matching method:
|
'name'
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the identities match according to the specified method. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in sleap_io/model/identity.py
def matches(self, other: "Identity", method: str = "name") -> bool:
"""Check if this identity matches another identity.
Args:
other: Another identity to compare with.
method: Matching method:
- ``"name"`` (default): match by the `name` attribute, which
survives serialization and cross-file merges.
- ``"identity"``: match by Python object identity (same object).
Returns:
True if the identities match according to the specified method.
Raises:
ValueError: If `method` is not one of the supported values.
"""
if method == "name":
return self.name == other.name
elif method == "identity":
return self is other
else:
raise ValueError(f"Unknown matching method: {method}")
Instance
¶
This class represents a ground truth instance such as an animal.
An Instance has a set of landmarks (points) that correspond to a Skeleton. Each
point is associated with a Node in the skeleton. The points are stored in a
structured numpy array with columns for x, y, visible, complete and name.
The Instance may also be associated with a Track which links multiple instances
together across frames or videos.
Attributes:
| Name | Type | Description |
|---|---|---|
points |
A numpy structured array with columns for xy, visible and complete. The
array should have shape |
|
skeleton |
The |
|
track |
An optional |
|
tracking_score |
The score associated with the |
|
identity |
An optional |
|
identity_score |
The score associated with the |
|
from_predicted |
The |
|
identity_embedding |
An optional |
|
category |
An optional |
|
category_score |
The score associated with the |
|
category_embedding |
An optional |
Methods:
| Name | Description |
|---|---|
__attrs_post_init__ |
Convert the points array after initialization. |
__getitem__ |
Return the point associated with a node. |
__init__ |
Method generated by attrs for class Instance. |
__len__ |
Return the number of points in the instance. |
__repr__ |
Return a readable representation of the instance. |
__setattr__ |
Method generated by attrs for class Instance. |
__setitem__ |
Set the point associated with a node. |
bounding_box |
Get the bounding box of visible points. |
empty |
Create an empty instance with no points. |
from_numpy |
Create an instance object from a numpy array. |
numpy |
Return the instance points as a |
overlaps_with |
Check if this instance overlaps with another based on bounding box IoU. |
replace_skeleton |
Replace the skeleton associated with the instance. |
same_identity_as |
Check if this instance has the same identity as another instance. |
same_pose_as |
Check if this instance has the same pose as another instance. |
to_bbox |
Create a bounding box from this instance. |
to_centroid |
Create a |
to_mask |
Rasterize this instance's ROI geometry into a segmentation mask. |
to_roi |
Create a region-of-interest geometry from this instance. |
update_skeleton |
Update or replace the skeleton associated with the instance. |
Source code in sleap_io/model/instance.py
@attrs.define(auto_attribs=True, slots=True, eq=False)
class Instance:
"""This class represents a ground truth instance such as an animal.
An `Instance` has a set of landmarks (points) that correspond to a `Skeleton`. Each
point is associated with a `Node` in the skeleton. The points are stored in a
structured numpy array with columns for x, y, visible, complete and name.
The `Instance` may also be associated with a `Track` which links multiple instances
together across frames or videos.
Attributes:
points: A numpy structured array with columns for xy, visible and complete. The
array should have shape `(n_nodes,)`. This representation is useful for
performance efficiency when working with large datasets.
skeleton: The `Skeleton` that describes the `Node`s and `Edge`s associated with
this instance.
track: An optional `Track` associated with a unique animal/object across frames
or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity assignment.
This is `None` if the instance is not associated with a track or if the
track was assigned manually.
identity: An optional `Identity` representing the global, ground-truth animal
this instance belongs to (persistent across videos/sessions). Unlike
`track` (an ephemeral, video-local tracklet), `Identity` is the cross-file
re-identification key. `None` if no global identity is assigned.
identity_score: The score associated with the `identity` assignment (e.g. the
cosine similarity to a re-ID gallery prototype). This is `None` if the
instance has no identity or the identity was assigned manually. Kept
separate from `tracking_score` (short-term tracklet vs long-term identity).
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
identity_embedding: An optional `Embedding` describing this instance's
appearance for re-identification (e.g. a vector produced by a re-ID
model). ``None`` by default.
category: An optional `Category` representing the *class* this instance
belongs to (e.g. ``"female_fly"``, ``"fur_shaved"``), typically
assigned by classification or re-ID. Mirrors `identity` but groups by
class rather than individual. `None` if no category is assigned.
category_score: The score associated with the `category` assignment (e.g.
the classifier confidence). `None` if the instance has no category or
the category was assigned manually.
category_embedding: An optional `Embedding` describing this instance's
appearance for classification (the vector the `category` was
classified from). ``None`` by default.
"""
points: PointsArray = attrs.field(eq=attrs.cmp_using(eq=np.array_equal))
skeleton: Skeleton
track: Track | None = None
tracking_score: float | None = None
identity: Identity | None = None
identity_score: float | None = None
category: Category | None = attrs.field(default=None, converter=to_category)
category_score: float | None = None
from_predicted: "PredictedInstance | None" = None
identity_embedding: Embedding | None = attrs.field(default=None, repr=False)
category_embedding: Embedding | None = attrs.field(default=None, repr=False)
@classmethod
def empty(
cls,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an empty instance with no points.
Args:
skeleton: The `Skeleton` that this `Instance` is associated with.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` with an empty numpy array of shape `(n_nodes,)`.
"""
points = PointsArray.empty(len(skeleton))
points["name"] = skeleton.node_names
return cls(
points=points,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
@classmethod
def _convert_points(
cls, points_data: np.ndarray | dict | list, skeleton: Skeleton
) -> PointsArray:
"""Convert points to a structured numpy array if needed."""
if isinstance(points_data, dict):
return PointsArray.from_dict(points_data, skeleton)
elif isinstance(points_data, (list, np.ndarray)):
if isinstance(points_data, list):
points_data = np.array(points_data)
points = PointsArray.from_array(points_data)
points["name"] = skeleton.node_names
return points
else:
raise ValueError("points must be a numpy array or dictionary.")
@classmethod
def from_numpy(
cls,
points_data: np.ndarray,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an instance object from a numpy array.
Args:
points_data: A numpy array of shape `(n_nodes, D)` corresponding to the
points of the skeleton. Values of `np.nan` indicate "missing" nodes and
will be reflected in the "visible" field.
If `D == 2`, the array should have columns for x and y.
If `D == 3`, the array should have columns for x, y and visible.
If `D == 4`, the array should have columns for x, y, visible and
complete.
If this is provided as a structured array, it will be used without copy
if it has the correct dtype. Otherwise, a new structured array will be
created reusing the provided data.
skeleton: The `Skeleton` that this `Instance` is associated with. It should
have `n_nodes` nodes.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` object with the specified points.
"""
return cls(
points=points_data,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
def __attrs_post_init__(self):
"""Convert the points array after initialization."""
if not isinstance(self.points, PointsArray):
self.points = self._convert_points(self.points, self.skeleton)
# Ensure points have node names
if "name" in self.points.dtype.names and not all(self.points["name"]):
self.points["name"] = self.skeleton.node_names
def numpy(
self,
invisible_as_nan: bool = True,
) -> np.ndarray:
"""Return the instance points as a `(n_nodes, 2)` numpy array.
Args:
invisible_as_nan: If `True` (the default), points that are not visible will
be set to `np.nan`. If `False`, they will be whatever the stored value
of `Instance.points["xy"]` is.
Returns:
A numpy array of shape `(n_nodes, 2)` corresponding to the points of the
skeleton. Values of `np.nan` indicate "missing" nodes.
Notes:
This will always return a copy of the array.
If you need to avoid making a copy, just access the `Instance.points["xy"]`
attribute directly. This will not replace invisible points with `np.nan`.
"""
if invisible_as_nan:
return np.where(
self.points["visible"].reshape(-1, 1), self.points["xy"], np.nan
)
else:
return self.points["xy"].copy()
@property
def centroid_xy(self) -> tuple[float, float] | None:
"""Mean of visible point coordinates as ``(x, y)``, or ``None``.
Returns:
A tuple ``(x, y)`` representing the center of mass of all visible
points, or ``None`` if no points are visible.
"""
pts = self.numpy(invisible_as_nan=True)
visible = ~np.isnan(pts[:, 0])
if not visible.any():
return None
return float(pts[visible, 0].mean()), float(pts[visible, 1].mean())
def to_centroid(
self,
method: str = "center_of_mass",
node: int | str | None = None,
fallback: str | None = None,
error_on_empty: bool = False,
**kwargs,
) -> "Centroid":
"""Create a ``Centroid`` from this instance.
Delegates to ``Centroid.from_pose()``. A ``PredictedInstance`` yields a
``PredictedCentroid`` carrying its ``score``; any other instance yields a
``UserCentroid``. Metadata (``track``, ``tracking_score``, ``identity``,
``identity_score``, ``identity_embedding``, ``category``,
``category_score``, ``category_embedding``, ``instance=self``) is
propagated.
Args:
method: Computation method (``"center_of_mass"``, ``"bbox_center"``,
``"geometric_median"``, or ``"anchor"``).
node: Node specification for the ``"anchor"`` method. Can be a node
name (str) or index (int).
fallback: For the ``"anchor"`` method, a non-anchor method to fall
back to when the anchor node is occluded.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) centroid.
**kwargs: Additional keyword arguments passed to the centroid
constructor.
Returns:
A ``UserCentroid`` or ``PredictedCentroid`` depending on the
instance type.
Raises:
ValueError: For an unknown ``method``, a missing ``node`` for the
``"anchor"`` method, an invalid ``node`` type, or (when
``error_on_empty`` is ``True``) when there are no visible points.
"""
from sleap_io.model.centroid import Centroid
return Centroid.from_pose(
self,
method=method,
node=node,
fallback=fallback,
error_on_empty=error_on_empty,
**kwargs,
)
def to_bbox(
self,
mode: str = "tight",
size: float | tuple[float, float] | None = None,
padding: float | tuple[float, float] = 0.0,
node: int | str | None = None,
center_method: str = "center_of_mass",
rotated: bool = False,
error_on_empty: bool = False,
) -> "BoundingBox":
"""Create a bounding box from this instance.
A ``PredictedInstance`` yields a ``PredictedBoundingBox`` carrying its
``score``; any other instance yields a ``UserBoundingBox``. Metadata
(``track``, ``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
mode: ``"tight"`` to fit the visible points, or ``"centered"`` to
build a fixed-``size`` box centered on a computed centroid.
size: Box size for ``mode="centered"``. A scalar yields a square box;
a ``(w, h)`` tuple sets width and height independently. Required
for ``mode="centered"``.
padding: Amount to inflate the box outward. Scalar applies to both
axes; a ``(px, py)`` tuple applies per-axis. Negative values
shrink the box.
node: Node specification passed to the centroid computation for
``mode="centered"`` with ``center_method="anchor"``.
center_method: Centroid method used to locate the box center for
``mode="centered"`` (see :meth:`to_centroid`).
rotated: For ``mode="tight"``, if ``True`` fit a minimum-area oriented
box from the convex hull of visible points; otherwise fit an
axis-aligned box.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) box.
Returns:
A ``BoundingBox`` enclosing the instance (or NaN corners if empty).
Raises:
ValueError: For an unknown ``mode``, a missing ``size`` for
``mode="centered"``, or (when ``error_on_empty`` is ``True``)
when there are no visible points.
"""
from sleap_io.model.bbox import PredictedBoundingBox, UserBoundingBox
from sleap_io.model.roi import (
_apply_padding,
_geometry_to_bbox_coords,
_pose_to_geometry,
)
nan = float("nan")
angle = 0.0
if mode == "tight":
pts = self.numpy(invisible_as_nan=True)
visible = ~np.isnan(pts[:, 0])
if not visible.any():
if error_on_empty:
raise ValueError("No visible points to compute bounding box.")
x1 = y1 = x2 = y2 = nan
elif rotated:
hull = _pose_to_geometry(
pts, self.skeleton.edge_inds, method="convex_hull"
)
x1, y1, x2, y2, angle = _geometry_to_bbox_coords(hull, rotated=True)
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
vis = pts[visible]
x1 = float(vis[:, 0].min())
y1 = float(vis[:, 1].min())
x2 = float(vis[:, 0].max())
y2 = float(vis[:, 1].max())
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
elif mode == "centered":
if size is None:
raise ValueError("'size' is required for mode='centered'.")
centroid = self.to_centroid(
method=center_method, node=node, error_on_empty=error_on_empty
)
if centroid.is_empty:
x1 = y1 = x2 = y2 = nan
else:
cx, cy = centroid.xy
if isinstance(size, (tuple, list)):
w, h = size
else:
w = h = size
x1 = cx - w / 2
y1 = cy - h / 2
x2 = cx + w / 2
y2 = cy + h / 2
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
raise ValueError(f"Unknown mode {mode!r}. Expected 'tight' or 'centered'.")
kwargs = dict(
x1=x1,
y1=y1,
x2=x2,
y2=y2,
angle=angle,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedBoundingBox(score=self.score, **kwargs)
return UserBoundingBox(**kwargs)
def to_roi(
self,
method: str = "shapes",
node_radius: float = 0.0,
edge_radius: float = 0.0,
radius: float = 0.0,
quad_segs: int = 8,
error_on_empty: bool = False,
) -> "ROI":
"""Create a region-of-interest geometry from this instance.
A ``PredictedInstance`` yields a ``PredictedROI`` carrying its ``score``;
any other instance yields a ``UserROI``. Metadata (``track``,
``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
method: ``"shapes"`` to union buffered node points and/or edge
segments, or ``"convex_hull"`` to take the convex hull of the
visible points.
node_radius: Buffer radius around each visible node (``"shapes"``
only).
edge_radius: Buffer radius around each fully-visible edge segment
(``"shapes"`` only).
radius: Optional buffer applied to the convex hull
(``"convex_hull"`` only).
quad_segs: Number of segments used to approximate a quarter circle
when buffering.
error_on_empty: If ``True``, raise ``ValueError`` when the resulting
geometry is empty instead of returning an empty-geometry ROI.
Returns:
A ``ROI`` whose geometry encloses the instance (an empty ``Polygon``
if there are no visible points).
Raises:
ValueError: If ``method="shapes"`` with both ``node_radius`` and
``edge_radius`` equal to 0 (a misconfiguration, always raised),
for an unknown ``method``, or (when ``error_on_empty`` is
``True``) when the resulting geometry is empty.
"""
from sleap_io.model.roi import PredictedROI, UserROI, _pose_to_geometry
# Misconfiguration: raise before the empty-points check so that an empty
# instance still surfaces the error.
if method == "shapes" and node_radius == 0 and edge_radius == 0:
raise ValueError(
"method='shapes' requires at least one of node_radius or "
"edge_radius to be > 0."
)
geom = _pose_to_geometry(
self.numpy(invisible_as_nan=True),
self.skeleton.edge_inds,
method=method,
node_radius=node_radius,
edge_radius=edge_radius,
radius=radius,
quad_segs=quad_segs,
)
if geom.is_empty and error_on_empty:
raise ValueError("No visible points to compute ROI geometry.")
kwargs = dict(
geometry=geom,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedROI(score=self.score, **kwargs)
return UserROI(**kwargs)
def to_mask(self, height: int, width: int, **roi_kwargs) -> "SegmentationMask":
"""Rasterize this instance's ROI geometry into a segmentation mask.
Equivalent to ``self.to_roi(**roi_kwargs).to_mask(height, width)``,
except that a zero-area hull (``method="convex_hull"`` over fewer than
three visible points yields a ``Point`` or ``LineString``) rasterizes to
an all-background mask here instead of raising. A ``PredictedInstance``
yields a ``PredictedSegmentationMask`` carrying its ``score``; any other
instance yields a ``UserSegmentationMask``. Metadata is propagated.
Args:
height: Height of the output mask in pixels.
width: Width of the output mask in pixels.
**roi_kwargs: Keyword arguments forwarded to :meth:`to_roi` (e.g.
``method``, ``node_radius``, ``edge_radius``, ``radius``,
``quad_segs``, ``error_on_empty``).
Returns:
A ``SegmentationMask`` with the rasterized geometry (all background
if the geometry is empty or has zero area).
Raises:
ValueError: Propagated from :meth:`to_roi` for a ``"shapes"``
misconfiguration, an unknown method, or (when
``error_on_empty`` is ``True``) an empty geometry.
"""
from shapely.geometry import MultiPolygon, Polygon
error_on_empty = roi_kwargs.pop("error_on_empty", False)
roi = self.to_roi(error_on_empty=error_on_empty, **roi_kwargs)
# A non-empty but non-fillable geometry (e.g. convex_hull of <3 visible
# points -> Point/LineString) has zero area; rasterize it as all
# background rather than letting _rasterize_geometry raise a TypeError.
rasterizable = isinstance(roi.geometry, (Polygon, MultiPolygon))
if roi.geometry.is_empty or not rasterizable:
from sleap_io.model.mask import (
PredictedSegmentationMask,
UserSegmentationMask,
)
empty = np.zeros((height, width), dtype=bool)
kwargs = dict(
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
category=self.category,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedSegmentationMask.from_numpy(
empty, score=self.score, **kwargs
)
return UserSegmentationMask.from_numpy(empty, **kwargs)
return roi.to_mask(height, width)
def __getitem__(self, node: int | str | Node) -> np.ndarray:
"""Return the point associated with a node."""
if type(node) is not int:
node = self.skeleton.index(node)
return self.points[node]
def __setitem__(self, node: int | str | Node, value):
"""Set the point associated with a node.
Args:
node: The node to set the point for. Can be an integer index, string name,
or Node object.
value: A tuple or array-like of length 2 containing (x, y) coordinates.
Notes:
This sets the point coordinates and marks the point as visible.
"""
if type(node) is not int:
node = self.skeleton.index(node)
if len(value) < 2:
raise ValueError("Value must have at least 2 elements (x, y)")
self.points[node]["xy"] = value[:2]
self.points[node]["visible"] = True
def __len__(self) -> int:
"""Return the number of points in the instance."""
return len(self.points)
def __repr__(self) -> str:
"""Return a readable representation of the instance."""
pts = self.numpy().tolist()
track = f'"{self.track.name}"' if self.track is not None else self.track
return f"Instance(points={pts}, track={track})"
@property
def n_visible(self) -> int:
"""Return the number of visible points in the instance."""
return sum(self.points["visible"])
@property
def is_empty(self) -> bool:
"""Return `True` if no points are visible on the instance."""
return ~(self.points["visible"].any())
def update_skeleton(self, names_only: bool = False):
"""Update or replace the skeleton associated with the instance.
Args:
names_only: If `True`, only update the node names in the points array. If
`False`, the points array will be updated to match the new skeleton.
"""
if names_only:
# Update the node names.
self.points["name"] = self.skeleton.node_names
return
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(self.points["name"])
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
new_points["name"] = self.skeleton.node_names
self.points = new_points
def replace_skeleton(
self,
new_skeleton: Skeleton,
node_names_map: dict[str, str] | None = None,
):
"""Replace the skeleton associated with the instance.
Args:
new_skeleton: The new `Skeleton` to associate with the instance.
node_names_map: Dictionary mapping nodes in the old skeleton to nodes in the
new skeleton. Keys and values should be specified as lists of strings.
If not provided, only nodes with identical names will be mapped. Points
associated with unmapped nodes will be removed.
Notes:
This method will update the `Instance.skeleton` attribute and the
`Instance.points` attribute in place (a copy is made of the points array).
It is recommended to use `Labels.replace_skeleton` instead of this method if
more flexible node mapping is required.
"""
# Update skeleton object.
# old_skeleton = self.skeleton
self.skeleton = new_skeleton
# Get node names with replacements from node map if possible.
# old_node_names = old_skeleton.node_names
old_node_names = self.points["name"].tolist()
if node_names_map is not None:
old_node_names = [node_names_map.get(node, node) for node in old_node_names]
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(old_node_names)
# old_node_inds = np.array(old_node_inds).reshape(-1, 1)
# new_node_inds = np.array(new_node_inds).reshape(-1, 1)
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
self.points = new_points
self.points["name"] = self.skeleton.node_names
def same_pose_as(self, other: "Instance", tolerance: float = None) -> bool:
"""Check if this instance has the same pose as another instance.
Args:
other: Another instance to compare with.
tolerance: Maximum distance (in pixels) between corresponding points
for them to be considered the same. If None (default), uses exact
comparison including proper NaN handling.
Returns:
True if the instances have the same pose within tolerance, False otherwise.
Notes:
Two instances are considered to have the same pose if:
- They have the same skeleton structure
- When tolerance is None: All coordinates match exactly (including NaN)
- When tolerance is specified: All visible points are within tolerance
distance and NaN patterns match exactly
"""
# Check skeleton compatibility
if not self.skeleton.matches(other.skeleton):
return False
if tolerance is None:
# Exact comparison using numpy arrays with proper NaN handling
return np.array_equal(self.numpy(), other.numpy(), equal_nan=True)
else:
# Tolerance-based comparison with proper NaN handling
self_array = self.numpy()
other_array = other.numpy()
# First, check if NaN patterns match exactly
self_nan_mask = np.isnan(self_array)
other_nan_mask = np.isnan(other_array)
if not np.array_equal(self_nan_mask, other_nan_mask):
return False
# Get mask for non-NaN values
non_nan_mask = ~self_nan_mask
# If all values are NaN, they're considered equal
if not non_nan_mask.any():
return True
# Calculate distances only for non-NaN points
self_pts = self_array[non_nan_mask]
other_pts = other_array[non_nan_mask]
# Reshape to handle the coordinate pairs properly
self_pts = self_pts.reshape(-1, 2)
other_pts = other_pts.reshape(-1, 2)
distances = np.linalg.norm(self_pts - other_pts, axis=1)
return np.all(distances <= tolerance)
def same_identity_as(self, other: "Instance") -> bool:
"""Check if this instance has the same identity as another instance.
Args:
other: Another instance to compare with.
Returns:
True if both instances share the same identity, False otherwise.
Notes:
Global `Identity` takes precedence: if both instances carry an
`Identity`, they match when their `name`s match (which survives
serialization and cross-file merges). Otherwise this falls back to
the ephemeral `Track`, where instances match only when they share the
same `Track` object (by object identity, not just by name).
"""
if self.identity is not None and other.identity is not None:
return self.identity.matches(other.identity, method="name")
if self.track is None or other.track is None:
return False
return self.track is other.track
def overlaps_with(self, other: "Instance", iou_threshold: float = 0.5) -> bool:
"""Check if this instance overlaps with another based on bounding box IoU.
Args:
other: Another instance to compare with.
iou_threshold: Minimum IoU (Intersection over Union) value to consider
the instances as overlapping.
Returns:
True if the instances overlap above the threshold, False otherwise.
Notes:
Overlap is computed using the bounding boxes of visible points.
If either instance has no visible points, they don't overlap.
"""
# Get visible points for both instances
self_visible = self.points["visible"]
other_visible = other.points["visible"]
if not self_visible.any() or not other_visible.any():
return False
# Calculate bounding boxes
self_pts = self.points["xy"][self_visible]
other_pts = other.points["xy"][other_visible]
self_bbox = np.array(
[
[np.min(self_pts[:, 0]), np.min(self_pts[:, 1])], # min x, y
[np.max(self_pts[:, 0]), np.max(self_pts[:, 1])], # max x, y
]
)
other_bbox = np.array(
[
[np.min(other_pts[:, 0]), np.min(other_pts[:, 1])],
[np.max(other_pts[:, 0]), np.max(other_pts[:, 1])],
]
)
# Calculate intersection
intersection_min = np.maximum(self_bbox[0], other_bbox[0])
intersection_max = np.minimum(self_bbox[1], other_bbox[1])
if np.any(intersection_min >= intersection_max):
# No intersection
return False
intersection_area = np.prod(intersection_max - intersection_min)
# Calculate union
self_area = np.prod(self_bbox[1] - self_bbox[0])
other_area = np.prod(other_bbox[1] - other_bbox[0])
union_area = self_area + other_area - intersection_area
# Calculate IoU
iou = intersection_area / union_area if union_area > 0 else 0
return iou >= iou_threshold
def bounding_box(self) -> np.ndarray | None:
"""Get the bounding box of visible points.
Returns:
A numpy array of shape (2, 2) with [[min_x, min_y], [max_x, max_y]],
or None if there are no visible points.
"""
visible = self.points["visible"]
if not visible.any():
return None
pts = self.points["xy"][visible]
return np.array(
[
[np.min(pts[:, 0]), np.min(pts[:, 1])],
[np.max(pts[:, 0]), np.max(pts[:, 1])],
]
)
__annotations__ = {'points': 'PointsArray', 'skeleton': 'Skeleton', 'track': 'Track | None', 'tracking_score': 'float | None', 'identity': 'Identity | None', 'identity_score': 'float | None', 'category': 'Category | None', 'category_score': 'float | None', 'from_predicted': "'PredictedInstance | None'", 'identity_embedding': 'Embedding | None', 'category_embedding': 'Embedding | None'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'This class represents a ground truth instance such as an animal.\n\nAn `Instance` has a set of landmarks (points) that correspond to a `Skeleton`. Each\npoint is associated with a `Node` in the skeleton. The points are stored in a\nstructured numpy array with columns for x, y, visible, complete and name.\n\nThe `Instance` may also be associated with a `Track` which links multiple instances\ntogether across frames or videos.\n\nAttributes:\n points: A numpy structured array with columns for xy, visible and complete. The\n array should have shape `(n_nodes,)`. This representation is useful for\n performance efficiency when working with large datasets.\n skeleton: The `Skeleton` that describes the `Node`s and `Edge`s associated with\n this instance.\n track: An optional `Track` associated with a unique animal/object across frames\n or videos.\n tracking_score: The score associated with the `Track` assignment. This is\n typically the value from the score matrix used in an identity assignment.\n This is `None` if the instance is not associated with a track or if the\n track was assigned manually.\n identity: An optional `Identity` representing the global, ground-truth animal\n this instance belongs to (persistent across videos/sessions). Unlike\n `track` (an ephemeral, video-local tracklet), `Identity` is the cross-file\n re-identification key. `None` if no global identity is assigned.\n identity_score: The score associated with the `identity` assignment (e.g. the\n cosine similarity to a re-ID gallery prototype). This is `None` if the\n instance has no identity or the identity was assigned manually. Kept\n separate from `tracking_score` (short-term tracklet vs long-term identity).\n from_predicted: The `PredictedInstance` (if any) that this instance was\n initialized from. This is used with human-in-the-loop workflows.\n identity_embedding: An optional `Embedding` describing this instance\'s\n appearance for re-identification (e.g. a vector produced by a re-ID\n model). ``None`` by default.\n category: An optional `Category` representing the *class* this instance\n belongs to (e.g. ``"female_fly"``, ``"fur_shaved"``), typically\n assigned by classification or re-ID. Mirrors `identity` but groups by\n class rather than individual. `None` if no category is assigned.\n category_score: The score associated with the `category` assignment (e.g.\n the classifier confidence). `None` if the instance has no category or\n the category was assigned manually.\n category_embedding: An optional `Embedding` describing this instance\'s\n appearance for classification (the vector the `category` was\n classified from). ``None`` by default.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 397
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('points', 'skeleton', 'track', 'tracking_score', 'identity', 'identity_score', 'category', 'category_score', 'from_predicted', 'identity_embedding', 'category_embedding')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.instance'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('points', 'skeleton', 'track', 'tracking_score', 'identity', 'identity_score', 'category', 'category_score', 'from_predicted', 'identity_embedding', 'category_embedding', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('points', 'skeleton')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
centroid_xy
property
¶
Mean of visible point coordinates as (x, y), or None.
Returns:
| Type | Description |
|---|---|
|
A tuple |
is_empty
property
¶
Return True if no points are visible on the instance.
n_visible
property
¶
Return the number of visible points in the instance.
__attrs_post_init__()
¶
Convert the points array after initialization.
Source code in sleap_io/model/instance.py
def __attrs_post_init__(self):
"""Convert the points array after initialization."""
if not isinstance(self.points, PointsArray):
self.points = self._convert_points(self.points, self.skeleton)
# Ensure points have node names
if "name" in self.points.dtype.names and not all(self.points["name"]):
self.points["name"] = self.skeleton.node_names
__getitem__(node)
¶
__init__(points, skeleton, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, from_predicted=None, identity_embedding=None, category_embedding=None)
¶
Method generated by attrs for class Instance.
Source code in sleap_io/model/instance.py
"""Data structures for data associated with a single instance such as an animal.
The `Instance` class is a SLEAP data structure that contains a collection of points that
correspond to landmarks within a `Skeleton`.
`PredictedInstance` additionally contains metadata associated with how the instance was
estimated, such as confidence scores.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import attrs
__len__()
¶
__repr__()
¶
Return a readable representation of the instance.
__setattr__(name, val)
¶
Method generated by attrs for class Instance.
Source code in sleap_io/model/instance.py
__setitem__(node, value)
¶
Set the point associated with a node.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
int | str | Node
|
The node to set the point for. Can be an integer index, string name, or Node object. |
required |
value
|
A tuple or array-like of length 2 containing (x, y) coordinates. |
required |
Notes
This sets the point coordinates and marks the point as visible.
Source code in sleap_io/model/instance.py
def __setitem__(self, node: int | str | Node, value):
"""Set the point associated with a node.
Args:
node: The node to set the point for. Can be an integer index, string name,
or Node object.
value: A tuple or array-like of length 2 containing (x, y) coordinates.
Notes:
This sets the point coordinates and marks the point as visible.
"""
if type(node) is not int:
node = self.skeleton.index(node)
if len(value) < 2:
raise ValueError("Value must have at least 2 elements (x, y)")
self.points[node]["xy"] = value[:2]
self.points[node]["visible"] = True
bounding_box()
¶
Get the bounding box of visible points.
Returns:
| Type | Description |
|---|---|
ndarray | None
|
A numpy array of shape (2, 2) with [[min_x, min_y], [max_x, max_y]], or None if there are no visible points. |
Source code in sleap_io/model/instance.py
def bounding_box(self) -> np.ndarray | None:
"""Get the bounding box of visible points.
Returns:
A numpy array of shape (2, 2) with [[min_x, min_y], [max_x, max_y]],
or None if there are no visible points.
"""
visible = self.points["visible"]
if not visible.any():
return None
pts = self.points["xy"][visible]
return np.array(
[
[np.min(pts[:, 0]), np.min(pts[:, 1])],
[np.max(pts[:, 0]), np.max(pts[:, 1])],
]
)
empty(skeleton, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, identity_embedding=None, category_embedding=None, from_predicted=None)
classmethod
¶
Create an empty instance with no points.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
skeleton
|
Skeleton
|
The |
required |
track
|
Track | None
|
An optional |
None
|
tracking_score
|
float | None
|
The score associated with the |
None
|
identity
|
Identity | None
|
An optional global |
None
|
identity_score
|
float | None
|
The score associated with the |
None
|
category
|
Category | None
|
An optional |
None
|
category_score
|
float | None
|
The score associated with the |
None
|
identity_embedding
|
Embedding | None
|
An optional re-ID |
None
|
category_embedding
|
Embedding | None
|
An optional classification |
None
|
from_predicted
|
PredictedInstance | None
|
The |
None
|
Returns:
| Type | Description |
|---|---|
Instance
|
An |
Source code in sleap_io/model/instance.py
@classmethod
def empty(
cls,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an empty instance with no points.
Args:
skeleton: The `Skeleton` that this `Instance` is associated with.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` with an empty numpy array of shape `(n_nodes,)`.
"""
points = PointsArray.empty(len(skeleton))
points["name"] = skeleton.node_names
return cls(
points=points,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
from_numpy(points_data, skeleton, track=None, tracking_score=None, identity=None, identity_score=None, category=None, category_score=None, identity_embedding=None, category_embedding=None, from_predicted=None)
classmethod
¶
Create an instance object from a numpy array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
points_data
|
ndarray
|
A numpy array of shape If If this is provided as a structured array, it will be used without copy if it has the correct dtype. Otherwise, a new structured array will be created reusing the provided data. |
required |
skeleton
|
Skeleton
|
The |
required |
track
|
Track | None
|
An optional |
None
|
tracking_score
|
float | None
|
The score associated with the |
None
|
identity
|
Identity | None
|
An optional global |
None
|
identity_score
|
float | None
|
The score associated with the |
None
|
category
|
Category | None
|
An optional |
None
|
category_score
|
float | None
|
The score associated with the |
None
|
identity_embedding
|
Embedding | None
|
An optional re-ID |
None
|
category_embedding
|
Embedding | None
|
An optional classification |
None
|
from_predicted
|
PredictedInstance | None
|
The |
None
|
Returns:
| Type | Description |
|---|---|
Instance
|
An |
Source code in sleap_io/model/instance.py
@classmethod
def from_numpy(
cls,
points_data: np.ndarray,
skeleton: Skeleton,
track: Track | None = None,
tracking_score: float | None = None,
identity: Identity | None = None,
identity_score: float | None = None,
category: Category | None = None,
category_score: float | None = None,
identity_embedding: Embedding | None = None,
category_embedding: Embedding | None = None,
from_predicted: "PredictedInstance | None" = None,
) -> "Instance":
"""Create an instance object from a numpy array.
Args:
points_data: A numpy array of shape `(n_nodes, D)` corresponding to the
points of the skeleton. Values of `np.nan` indicate "missing" nodes and
will be reflected in the "visible" field.
If `D == 2`, the array should have columns for x and y.
If `D == 3`, the array should have columns for x, y and visible.
If `D == 4`, the array should have columns for x, y, visible and
complete.
If this is provided as a structured array, it will be used without copy
if it has the correct dtype. Otherwise, a new structured array will be
created reusing the provided data.
skeleton: The `Skeleton` that this `Instance` is associated with. It should
have `n_nodes` nodes.
track: An optional `Track` associated with a unique animal/object across
frames or videos.
tracking_score: The score associated with the `Track` assignment. This is
typically the value from the score matrix used in an identity
assignment. This is `None` if the instance is not associated with a
track or if the track was assigned manually.
identity: An optional global `Identity` for this instance.
identity_score: The score associated with the `identity` assignment.
category: An optional `Category` (class) for this instance.
category_score: The score associated with the `category` assignment.
identity_embedding: An optional re-ID `Embedding` for this instance.
category_embedding: An optional classification `Embedding` for this
instance.
from_predicted: The `PredictedInstance` (if any) that this instance was
initialized from. This is used with human-in-the-loop workflows.
Returns:
An `Instance` object with the specified points.
"""
return cls(
points=points_data,
skeleton=skeleton,
track=track,
tracking_score=tracking_score,
identity=identity,
identity_score=identity_score,
category=category,
category_score=category_score,
identity_embedding=identity_embedding,
category_embedding=category_embedding,
from_predicted=from_predicted,
)
numpy(invisible_as_nan=True)
¶
Return the instance points as a (n_nodes, 2) numpy array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
invisible_as_nan
|
bool
|
If |
True
|
Returns:
| Type | Description |
|---|---|
ndarray
|
A numpy array of shape |
Notes
This will always return a copy of the array.
If you need to avoid making a copy, just access the Instance.points["xy"]
attribute directly. This will not replace invisible points with np.nan.
Source code in sleap_io/model/instance.py
def numpy(
self,
invisible_as_nan: bool = True,
) -> np.ndarray:
"""Return the instance points as a `(n_nodes, 2)` numpy array.
Args:
invisible_as_nan: If `True` (the default), points that are not visible will
be set to `np.nan`. If `False`, they will be whatever the stored value
of `Instance.points["xy"]` is.
Returns:
A numpy array of shape `(n_nodes, 2)` corresponding to the points of the
skeleton. Values of `np.nan` indicate "missing" nodes.
Notes:
This will always return a copy of the array.
If you need to avoid making a copy, just access the `Instance.points["xy"]`
attribute directly. This will not replace invisible points with `np.nan`.
"""
if invisible_as_nan:
return np.where(
self.points["visible"].reshape(-1, 1), self.points["xy"], np.nan
)
else:
return self.points["xy"].copy()
overlaps_with(other, iou_threshold=0.5)
¶
Check if this instance overlaps with another based on bounding box IoU.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Instance
|
Another instance to compare with. |
required |
iou_threshold
|
float
|
Minimum IoU (Intersection over Union) value to consider the instances as overlapping. |
0.5
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the instances overlap above the threshold, False otherwise. |
Notes
Overlap is computed using the bounding boxes of visible points. If either instance has no visible points, they don't overlap.
Source code in sleap_io/model/instance.py
def overlaps_with(self, other: "Instance", iou_threshold: float = 0.5) -> bool:
"""Check if this instance overlaps with another based on bounding box IoU.
Args:
other: Another instance to compare with.
iou_threshold: Minimum IoU (Intersection over Union) value to consider
the instances as overlapping.
Returns:
True if the instances overlap above the threshold, False otherwise.
Notes:
Overlap is computed using the bounding boxes of visible points.
If either instance has no visible points, they don't overlap.
"""
# Get visible points for both instances
self_visible = self.points["visible"]
other_visible = other.points["visible"]
if not self_visible.any() or not other_visible.any():
return False
# Calculate bounding boxes
self_pts = self.points["xy"][self_visible]
other_pts = other.points["xy"][other_visible]
self_bbox = np.array(
[
[np.min(self_pts[:, 0]), np.min(self_pts[:, 1])], # min x, y
[np.max(self_pts[:, 0]), np.max(self_pts[:, 1])], # max x, y
]
)
other_bbox = np.array(
[
[np.min(other_pts[:, 0]), np.min(other_pts[:, 1])],
[np.max(other_pts[:, 0]), np.max(other_pts[:, 1])],
]
)
# Calculate intersection
intersection_min = np.maximum(self_bbox[0], other_bbox[0])
intersection_max = np.minimum(self_bbox[1], other_bbox[1])
if np.any(intersection_min >= intersection_max):
# No intersection
return False
intersection_area = np.prod(intersection_max - intersection_min)
# Calculate union
self_area = np.prod(self_bbox[1] - self_bbox[0])
other_area = np.prod(other_bbox[1] - other_bbox[0])
union_area = self_area + other_area - intersection_area
# Calculate IoU
iou = intersection_area / union_area if union_area > 0 else 0
return iou >= iou_threshold
replace_skeleton(new_skeleton, node_names_map=None)
¶
Replace the skeleton associated with the instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_skeleton
|
Skeleton
|
The new |
required |
node_names_map
|
dict[str, str] | None
|
Dictionary mapping nodes in the old skeleton to nodes in the new skeleton. Keys and values should be specified as lists of strings. If not provided, only nodes with identical names will be mapped. Points associated with unmapped nodes will be removed. |
None
|
Notes
This method will update the Instance.skeleton attribute and the
Instance.points attribute in place (a copy is made of the points array).
It is recommended to use Labels.replace_skeleton instead of this method if
more flexible node mapping is required.
Source code in sleap_io/model/instance.py
def replace_skeleton(
self,
new_skeleton: Skeleton,
node_names_map: dict[str, str] | None = None,
):
"""Replace the skeleton associated with the instance.
Args:
new_skeleton: The new `Skeleton` to associate with the instance.
node_names_map: Dictionary mapping nodes in the old skeleton to nodes in the
new skeleton. Keys and values should be specified as lists of strings.
If not provided, only nodes with identical names will be mapped. Points
associated with unmapped nodes will be removed.
Notes:
This method will update the `Instance.skeleton` attribute and the
`Instance.points` attribute in place (a copy is made of the points array).
It is recommended to use `Labels.replace_skeleton` instead of this method if
more flexible node mapping is required.
"""
# Update skeleton object.
# old_skeleton = self.skeleton
self.skeleton = new_skeleton
# Get node names with replacements from node map if possible.
# old_node_names = old_skeleton.node_names
old_node_names = self.points["name"].tolist()
if node_names_map is not None:
old_node_names = [node_names_map.get(node, node) for node in old_node_names]
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(old_node_names)
# old_node_inds = np.array(old_node_inds).reshape(-1, 1)
# new_node_inds = np.array(new_node_inds).reshape(-1, 1)
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
self.points = new_points
self.points["name"] = self.skeleton.node_names
same_identity_as(other)
¶
Check if this instance has the same identity as another instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Instance
|
Another instance to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if both instances share the same identity, False otherwise. |
Notes
Global Identity takes precedence: if both instances carry an
Identity, they match when their names match (which survives
serialization and cross-file merges). Otherwise this falls back to
the ephemeral Track, where instances match only when they share the
same Track object (by object identity, not just by name).
Source code in sleap_io/model/instance.py
def same_identity_as(self, other: "Instance") -> bool:
"""Check if this instance has the same identity as another instance.
Args:
other: Another instance to compare with.
Returns:
True if both instances share the same identity, False otherwise.
Notes:
Global `Identity` takes precedence: if both instances carry an
`Identity`, they match when their `name`s match (which survives
serialization and cross-file merges). Otherwise this falls back to
the ephemeral `Track`, where instances match only when they share the
same `Track` object (by object identity, not just by name).
"""
if self.identity is not None and other.identity is not None:
return self.identity.matches(other.identity, method="name")
if self.track is None or other.track is None:
return False
return self.track is other.track
same_pose_as(other, tolerance=None)
¶
Check if this instance has the same pose as another instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Instance
|
Another instance to compare with. |
required |
tolerance
|
float
|
Maximum distance (in pixels) between corresponding points for them to be considered the same. If None (default), uses exact comparison including proper NaN handling. |
None
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the instances have the same pose within tolerance, False otherwise. |
Notes
Two instances are considered to have the same pose if: - They have the same skeleton structure - When tolerance is None: All coordinates match exactly (including NaN) - When tolerance is specified: All visible points are within tolerance distance and NaN patterns match exactly
Source code in sleap_io/model/instance.py
def same_pose_as(self, other: "Instance", tolerance: float = None) -> bool:
"""Check if this instance has the same pose as another instance.
Args:
other: Another instance to compare with.
tolerance: Maximum distance (in pixels) between corresponding points
for them to be considered the same. If None (default), uses exact
comparison including proper NaN handling.
Returns:
True if the instances have the same pose within tolerance, False otherwise.
Notes:
Two instances are considered to have the same pose if:
- They have the same skeleton structure
- When tolerance is None: All coordinates match exactly (including NaN)
- When tolerance is specified: All visible points are within tolerance
distance and NaN patterns match exactly
"""
# Check skeleton compatibility
if not self.skeleton.matches(other.skeleton):
return False
if tolerance is None:
# Exact comparison using numpy arrays with proper NaN handling
return np.array_equal(self.numpy(), other.numpy(), equal_nan=True)
else:
# Tolerance-based comparison with proper NaN handling
self_array = self.numpy()
other_array = other.numpy()
# First, check if NaN patterns match exactly
self_nan_mask = np.isnan(self_array)
other_nan_mask = np.isnan(other_array)
if not np.array_equal(self_nan_mask, other_nan_mask):
return False
# Get mask for non-NaN values
non_nan_mask = ~self_nan_mask
# If all values are NaN, they're considered equal
if not non_nan_mask.any():
return True
# Calculate distances only for non-NaN points
self_pts = self_array[non_nan_mask]
other_pts = other_array[non_nan_mask]
# Reshape to handle the coordinate pairs properly
self_pts = self_pts.reshape(-1, 2)
other_pts = other_pts.reshape(-1, 2)
distances = np.linalg.norm(self_pts - other_pts, axis=1)
return np.all(distances <= tolerance)
to_bbox(mode='tight', size=None, padding=0.0, node=None, center_method='center_of_mass', rotated=False, error_on_empty=False)
¶
Create a bounding box from this instance.
A PredictedInstance yields a PredictedBoundingBox carrying its
score; any other instance yields a UserBoundingBox. Metadata
(track, tracking_score, identity, identity_score,
identity_embedding, category, category_score,
category_embedding, instance=self) is propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mode
|
str
|
|
'tight'
|
size
|
float | tuple[float, float] | None
|
Box size for |
None
|
padding
|
float | tuple[float, float]
|
Amount to inflate the box outward. Scalar applies to both
axes; a |
0.0
|
node
|
int | str | None
|
Node specification passed to the centroid computation for
|
None
|
center_method
|
str
|
Centroid method used to locate the box center for
|
'center_of_mass'
|
rotated
|
bool
|
For |
False
|
error_on_empty
|
bool
|
If |
False
|
Returns:
| Type | Description |
|---|---|
BoundingBox
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
For an unknown |
Source code in sleap_io/model/instance.py
def to_bbox(
self,
mode: str = "tight",
size: float | tuple[float, float] | None = None,
padding: float | tuple[float, float] = 0.0,
node: int | str | None = None,
center_method: str = "center_of_mass",
rotated: bool = False,
error_on_empty: bool = False,
) -> "BoundingBox":
"""Create a bounding box from this instance.
A ``PredictedInstance`` yields a ``PredictedBoundingBox`` carrying its
``score``; any other instance yields a ``UserBoundingBox``. Metadata
(``track``, ``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
mode: ``"tight"`` to fit the visible points, or ``"centered"`` to
build a fixed-``size`` box centered on a computed centroid.
size: Box size for ``mode="centered"``. A scalar yields a square box;
a ``(w, h)`` tuple sets width and height independently. Required
for ``mode="centered"``.
padding: Amount to inflate the box outward. Scalar applies to both
axes; a ``(px, py)`` tuple applies per-axis. Negative values
shrink the box.
node: Node specification passed to the centroid computation for
``mode="centered"`` with ``center_method="anchor"``.
center_method: Centroid method used to locate the box center for
``mode="centered"`` (see :meth:`to_centroid`).
rotated: For ``mode="tight"``, if ``True`` fit a minimum-area oriented
box from the convex hull of visible points; otherwise fit an
axis-aligned box.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) box.
Returns:
A ``BoundingBox`` enclosing the instance (or NaN corners if empty).
Raises:
ValueError: For an unknown ``mode``, a missing ``size`` for
``mode="centered"``, or (when ``error_on_empty`` is ``True``)
when there are no visible points.
"""
from sleap_io.model.bbox import PredictedBoundingBox, UserBoundingBox
from sleap_io.model.roi import (
_apply_padding,
_geometry_to_bbox_coords,
_pose_to_geometry,
)
nan = float("nan")
angle = 0.0
if mode == "tight":
pts = self.numpy(invisible_as_nan=True)
visible = ~np.isnan(pts[:, 0])
if not visible.any():
if error_on_empty:
raise ValueError("No visible points to compute bounding box.")
x1 = y1 = x2 = y2 = nan
elif rotated:
hull = _pose_to_geometry(
pts, self.skeleton.edge_inds, method="convex_hull"
)
x1, y1, x2, y2, angle = _geometry_to_bbox_coords(hull, rotated=True)
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
vis = pts[visible]
x1 = float(vis[:, 0].min())
y1 = float(vis[:, 1].min())
x2 = float(vis[:, 0].max())
y2 = float(vis[:, 1].max())
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
elif mode == "centered":
if size is None:
raise ValueError("'size' is required for mode='centered'.")
centroid = self.to_centroid(
method=center_method, node=node, error_on_empty=error_on_empty
)
if centroid.is_empty:
x1 = y1 = x2 = y2 = nan
else:
cx, cy = centroid.xy
if isinstance(size, (tuple, list)):
w, h = size
else:
w = h = size
x1 = cx - w / 2
y1 = cy - h / 2
x2 = cx + w / 2
y2 = cy + h / 2
x1, y1, x2, y2 = _apply_padding(x1, y1, x2, y2, padding)
else:
raise ValueError(f"Unknown mode {mode!r}. Expected 'tight' or 'centered'.")
kwargs = dict(
x1=x1,
y1=y1,
x2=x2,
y2=y2,
angle=angle,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedBoundingBox(score=self.score, **kwargs)
return UserBoundingBox(**kwargs)
to_centroid(method='center_of_mass', node=None, fallback=None, error_on_empty=False, **kwargs)
¶
Create a Centroid from this instance.
Delegates to Centroid.from_pose(). A PredictedInstance yields a
PredictedCentroid carrying its score; any other instance yields a
UserCentroid. Metadata (track, tracking_score, identity,
identity_score, identity_embedding, category,
category_score, category_embedding, instance=self) is
propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
str
|
Computation method ( |
'center_of_mass'
|
node
|
int | str | None
|
Node specification for the |
None
|
fallback
|
str | None
|
For the |
None
|
error_on_empty
|
bool
|
If |
False
|
**kwargs
|
Additional keyword arguments passed to the centroid constructor. |
required |
Returns:
| Type | Description |
|---|---|
Centroid
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
For an unknown |
Source code in sleap_io/model/instance.py
def to_centroid(
self,
method: str = "center_of_mass",
node: int | str | None = None,
fallback: str | None = None,
error_on_empty: bool = False,
**kwargs,
) -> "Centroid":
"""Create a ``Centroid`` from this instance.
Delegates to ``Centroid.from_pose()``. A ``PredictedInstance`` yields a
``PredictedCentroid`` carrying its ``score``; any other instance yields a
``UserCentroid``. Metadata (``track``, ``tracking_score``, ``identity``,
``identity_score``, ``identity_embedding``, ``category``,
``category_score``, ``category_embedding``, ``instance=self``) is
propagated.
Args:
method: Computation method (``"center_of_mass"``, ``"bbox_center"``,
``"geometric_median"``, or ``"anchor"``).
node: Node specification for the ``"anchor"`` method. Can be a node
name (str) or index (int).
fallback: For the ``"anchor"`` method, a non-anchor method to fall
back to when the anchor node is occluded.
error_on_empty: If ``True``, raise ``ValueError`` when there are no
visible points instead of returning a degenerate (NaN) centroid.
**kwargs: Additional keyword arguments passed to the centroid
constructor.
Returns:
A ``UserCentroid`` or ``PredictedCentroid`` depending on the
instance type.
Raises:
ValueError: For an unknown ``method``, a missing ``node`` for the
``"anchor"`` method, an invalid ``node`` type, or (when
``error_on_empty`` is ``True``) when there are no visible points.
"""
from sleap_io.model.centroid import Centroid
return Centroid.from_pose(
self,
method=method,
node=node,
fallback=fallback,
error_on_empty=error_on_empty,
**kwargs,
)
to_mask(height, width, **roi_kwargs)
¶
Rasterize this instance's ROI geometry into a segmentation mask.
Equivalent to self.to_roi(**roi_kwargs).to_mask(height, width),
except that a zero-area hull (method="convex_hull" over fewer than
three visible points yields a Point or LineString) rasterizes to
an all-background mask here instead of raising. A PredictedInstance
yields a PredictedSegmentationMask carrying its score; any other
instance yields a UserSegmentationMask. Metadata is propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
height
|
int
|
Height of the output mask in pixels. |
required |
width
|
int
|
Width of the output mask in pixels. |
required |
**roi_kwargs
|
Keyword arguments forwarded to :meth: |
required |
Returns:
| Type | Description |
|---|---|
SegmentationMask
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
Propagated from :meth: |
Source code in sleap_io/model/instance.py
def to_mask(self, height: int, width: int, **roi_kwargs) -> "SegmentationMask":
"""Rasterize this instance's ROI geometry into a segmentation mask.
Equivalent to ``self.to_roi(**roi_kwargs).to_mask(height, width)``,
except that a zero-area hull (``method="convex_hull"`` over fewer than
three visible points yields a ``Point`` or ``LineString``) rasterizes to
an all-background mask here instead of raising. A ``PredictedInstance``
yields a ``PredictedSegmentationMask`` carrying its ``score``; any other
instance yields a ``UserSegmentationMask``. Metadata is propagated.
Args:
height: Height of the output mask in pixels.
width: Width of the output mask in pixels.
**roi_kwargs: Keyword arguments forwarded to :meth:`to_roi` (e.g.
``method``, ``node_radius``, ``edge_radius``, ``radius``,
``quad_segs``, ``error_on_empty``).
Returns:
A ``SegmentationMask`` with the rasterized geometry (all background
if the geometry is empty or has zero area).
Raises:
ValueError: Propagated from :meth:`to_roi` for a ``"shapes"``
misconfiguration, an unknown method, or (when
``error_on_empty`` is ``True``) an empty geometry.
"""
from shapely.geometry import MultiPolygon, Polygon
error_on_empty = roi_kwargs.pop("error_on_empty", False)
roi = self.to_roi(error_on_empty=error_on_empty, **roi_kwargs)
# A non-empty but non-fillable geometry (e.g. convex_hull of <3 visible
# points -> Point/LineString) has zero area; rasterize it as all
# background rather than letting _rasterize_geometry raise a TypeError.
rasterizable = isinstance(roi.geometry, (Polygon, MultiPolygon))
if roi.geometry.is_empty or not rasterizable:
from sleap_io.model.mask import (
PredictedSegmentationMask,
UserSegmentationMask,
)
empty = np.zeros((height, width), dtype=bool)
kwargs = dict(
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
category=self.category,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedSegmentationMask.from_numpy(
empty, score=self.score, **kwargs
)
return UserSegmentationMask.from_numpy(empty, **kwargs)
return roi.to_mask(height, width)
to_roi(method='shapes', node_radius=0.0, edge_radius=0.0, radius=0.0, quad_segs=8, error_on_empty=False)
¶
Create a region-of-interest geometry from this instance.
A PredictedInstance yields a PredictedROI carrying its score;
any other instance yields a UserROI. Metadata (track,
tracking_score, identity, identity_score,
identity_embedding, category, category_score,
category_embedding, instance=self) is propagated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
str
|
|
'shapes'
|
node_radius
|
float
|
Buffer radius around each visible node ( |
0.0
|
edge_radius
|
float
|
Buffer radius around each fully-visible edge segment
( |
0.0
|
radius
|
float
|
Optional buffer applied to the convex hull
( |
0.0
|
quad_segs
|
int
|
Number of segments used to approximate a quarter circle when buffering. |
8
|
error_on_empty
|
bool
|
If |
False
|
Returns:
| Type | Description |
|---|---|
ROI
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in sleap_io/model/instance.py
def to_roi(
self,
method: str = "shapes",
node_radius: float = 0.0,
edge_radius: float = 0.0,
radius: float = 0.0,
quad_segs: int = 8,
error_on_empty: bool = False,
) -> "ROI":
"""Create a region-of-interest geometry from this instance.
A ``PredictedInstance`` yields a ``PredictedROI`` carrying its ``score``;
any other instance yields a ``UserROI``. Metadata (``track``,
``tracking_score``, ``identity``, ``identity_score``,
``identity_embedding``, ``category``, ``category_score``,
``category_embedding``, ``instance=self``) is propagated.
Args:
method: ``"shapes"`` to union buffered node points and/or edge
segments, or ``"convex_hull"`` to take the convex hull of the
visible points.
node_radius: Buffer radius around each visible node (``"shapes"``
only).
edge_radius: Buffer radius around each fully-visible edge segment
(``"shapes"`` only).
radius: Optional buffer applied to the convex hull
(``"convex_hull"`` only).
quad_segs: Number of segments used to approximate a quarter circle
when buffering.
error_on_empty: If ``True``, raise ``ValueError`` when the resulting
geometry is empty instead of returning an empty-geometry ROI.
Returns:
A ``ROI`` whose geometry encloses the instance (an empty ``Polygon``
if there are no visible points).
Raises:
ValueError: If ``method="shapes"`` with both ``node_radius`` and
``edge_radius`` equal to 0 (a misconfiguration, always raised),
for an unknown ``method``, or (when ``error_on_empty`` is
``True``) when the resulting geometry is empty.
"""
from sleap_io.model.roi import PredictedROI, UserROI, _pose_to_geometry
# Misconfiguration: raise before the empty-points check so that an empty
# instance still surfaces the error.
if method == "shapes" and node_radius == 0 and edge_radius == 0:
raise ValueError(
"method='shapes' requires at least one of node_radius or "
"edge_radius to be > 0."
)
geom = _pose_to_geometry(
self.numpy(invisible_as_nan=True),
self.skeleton.edge_inds,
method=method,
node_radius=node_radius,
edge_radius=edge_radius,
radius=radius,
quad_segs=quad_segs,
)
if geom.is_empty and error_on_empty:
raise ValueError("No visible points to compute ROI geometry.")
kwargs = dict(
geometry=geom,
track=self.track,
tracking_score=self.tracking_score,
identity=self.identity,
identity_score=self.identity_score,
identity_embedding=self.identity_embedding,
category=self.category,
category_score=self.category_score,
category_embedding=self.category_embedding,
instance=self,
)
if isinstance(self, PredictedInstance):
return PredictedROI(score=self.score, **kwargs)
return UserROI(**kwargs)
update_skeleton(names_only=False)
¶
Update or replace the skeleton associated with the instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
names_only
|
bool
|
If |
False
|
Source code in sleap_io/model/instance.py
def update_skeleton(self, names_only: bool = False):
"""Update or replace the skeleton associated with the instance.
Args:
names_only: If `True`, only update the node names in the points array. If
`False`, the points array will be updated to match the new skeleton.
"""
if names_only:
# Update the node names.
self.points["name"] = self.skeleton.node_names
return
# Find correspondences.
new_node_inds, old_node_inds = self.skeleton.match_nodes(self.points["name"])
# Update the points.
new_points = PointsArray.empty(len(self.skeleton))
new_points[new_node_inds] = self.points[old_node_inds]
new_points["name"] = self.skeleton.node_names
self.points = new_points
Instance3D
¶
A 3D pose instance with keypoints in world coordinates.
Stores triangulated (or otherwise derived) 3D keypoints. Always associated with an InstanceGroup that contains the source 2D instances.
Attributes:
| Name | Type | Description |
|---|---|---|
points |
3D keypoint coordinates as (N, 3) float64 array. NaN values indicate missing/unresolved keypoints. |
|
skeleton |
The skeleton defining keypoint semantics. |
|
score |
Optional instance-level confidence score. |
|
metadata |
Arbitrary metadata dictionary. |
Methods:
| Name | Description |
|---|---|
__init__ |
Method generated by attrs for class Instance3D. |
__repr__ |
Return a readable representation of the 3D instance. |
__setattr__ |
Method generated by attrs for class Instance3D. |
numpy |
Return 3D points as (N, 3) float64 array. |
Source code in sleap_io/model/instance.py
@attrs.define(eq=False)
class Instance3D:
"""A 3D pose instance with keypoints in world coordinates.
Stores triangulated (or otherwise derived) 3D keypoints. Always associated
with an InstanceGroup that contains the source 2D instances.
Attributes:
points: 3D keypoint coordinates as (N, 3) float64 array.
NaN values indicate missing/unresolved keypoints.
skeleton: The skeleton defining keypoint semantics.
score: Optional instance-level confidence score.
metadata: Arbitrary metadata dictionary.
"""
points: np.ndarray = attrs.field(
converter=lambda x: np.array(x, dtype="float64") if x is not None else None
)
skeleton: Skeleton = attrs.field()
score: float | None = attrs.field(
default=None, converter=attrs.converters.optional(float)
)
metadata: dict = attrs.field(
factory=dict, validator=attrs.validators.instance_of(dict)
)
def __repr__(self) -> str:
"""Return a readable representation of the 3D instance."""
n_valid = 0
if self.points is not None:
n_valid = int(np.sum(~np.isnan(self.points).any(axis=1)))
n_total = len(self.skeleton.nodes)
return f"Instance3D(n_points={n_valid}/{n_total})"
@property
def n_visible(self) -> int:
"""Number of non-NaN 3D keypoints."""
if self.points is None:
return 0
return int(np.sum(~np.isnan(self.points).any(axis=1)))
@property
def is_empty(self) -> bool:
"""Whether all keypoints are NaN or points is None."""
return self.n_visible == 0
def numpy(self) -> np.ndarray:
"""Return 3D points as (N, 3) float64 array."""
if self.points is None:
return np.full((len(self.skeleton.nodes), 3), np.nan, dtype="float64")
return self.points.copy()
__annotations__ = {'points': 'np.ndarray', 'skeleton': 'Skeleton', 'score': 'float | None', 'metadata': 'dict'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'A 3D pose instance with keypoints in world coordinates.\n\nStores triangulated (or otherwise derived) 3D keypoints. Always associated\nwith an InstanceGroup that contains the source 2D instances.\n\nAttributes:\n points: 3D keypoint coordinates as (N, 3) float64 array.\n NaN values indicate missing/unresolved keypoints.\n skeleton: The skeleton defining keypoint semantics.\n score: Optional instance-level confidence score.\n metadata: Arbitrary metadata dictionary.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 1497
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('points', 'skeleton', 'score', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.instance'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('points', 'skeleton', 'score', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ()
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
is_empty
property
¶
Whether all keypoints are NaN or points is None.
n_visible
property
¶
Number of non-NaN 3D keypoints.
__init__(points, skeleton, score=None, metadata=NOTHING)
¶
Method generated by attrs for class Instance3D.
Source code in sleap_io/model/instance.py
"""Data structures for data associated with a single instance such as an animal.
The `Instance` class is a SLEAP data structure that contains a collection of points that
correspond to landmarks within a `Skeleton`.
`PredictedInstance` additionally contains metadata associated with how the instance was
estimated, such as confidence scores.
"""
from __future__ import annotations
__repr__()
¶
Return a readable representation of the 3D instance.
Source code in sleap_io/model/instance.py
__setattr__(name, val)
¶
Method generated by attrs for class Instance3D.
Source code in sleap_io/model/instance.py
numpy()
¶
InstanceGroup
¶
Defines a group of instances across the same frame index.
Attributes:
| Name | Type | Description |
|---|---|---|
instances_by_camera |
Dictionary of |
|
instances |
List of |
|
cameras |
List of |
|
score |
Optional score for the |
|
instance_3d |
Optional |
|
points |
Optional 3D points for the |
|
identity |
Optional |
|
category |
Optional |
|
metadata |
Dictionary of metadata. |
Methods:
| Name | Description |
|---|---|
__init__ |
Method generated by attrs for class InstanceGroup. |
__repr__ |
Return a readable representation of the instance group. |
__setattr__ |
Method generated by attrs for class InstanceGroup. |
get_instance |
Get |
Source code in sleap_io/model/camera.py
@define(eq=False) # Set eq to false to make class hashable
class InstanceGroup:
"""Defines a group of instances across the same frame index.
Attributes:
instances_by_camera: Dictionary of `Instance` objects by `Camera`.
instances: List of `Instance` objects in the group.
cameras: List of `Camera` objects that have an `Instance` associated.
score: Optional score for the `InstanceGroup`. Setting the score will also
update the score for all `instances` already in the `InstanceGroup`. The
score for `instances` will not be updated upon initialization.
instance_3d: Optional `Instance3D` with triangulated 3D keypoints.
points: Optional 3D points for the `InstanceGroup`. Delegates to
`instance_3d.points` if present.
identity: Optional `Identity` for this group (which animal).
category: Optional `Category` for this group (which class). Mirrors
`identity` but groups by class rather than individual.
metadata: Dictionary of metadata.
"""
_instance_by_camera: dict[Camera, Instance] = field(
factory=dict, validator=instance_of(dict)
)
_score: float | None = field(
default=None, converter=attrs.converters.optional(float)
)
_instance_3d: "Instance3D | None" = field(default=None)
identity: "Identity | None" = field(default=None)
category: "Category | None" = field(default=None)
metadata: dict = field(factory=dict, validator=instance_of(dict))
@property
def instance_by_camera(self) -> dict[Camera, Instance]:
"""Get dictionary of `Instance` objects by `Camera`."""
return self._instance_by_camera
@property
def instances(self) -> list[Instance]:
"""List of `Instance` objects."""
return list(self._instance_by_camera.values())
@property
def cameras(self) -> "list[Camera]":
"""List of `Camera` objects."""
return list(self._instance_by_camera.keys())
@property
def score(self) -> float | None:
"""Get score for `InstanceGroup`."""
return self._score
@property
def instance_3d(self) -> "Instance3D | None":
"""The 3D instance for this group."""
return self._instance_3d
@property
def points(self) -> np.ndarray | None:
"""3D keypoint coordinates. Delegates to instance_3d.points."""
if self._instance_3d is not None:
return self._instance_3d.points
return None
@points.setter
def points(self, value: np.ndarray | None):
"""Set 3D points. Creates/updates Instance3D as needed."""
if value is None:
self._instance_3d = None
elif self._instance_3d is not None:
self._instance_3d.points = np.array(value, dtype="float64")
else:
# Need a skeleton — get from first instance if available
skeleton = None
for inst in self._instance_by_camera.values():
skeleton = inst.skeleton
break
if skeleton is None:
raise ValueError(
"Cannot set 3D points: no skeleton available from "
"instance_by_camera."
)
self._instance_3d = Instance3D(points=value, skeleton=skeleton)
def get_instance(self, camera: Camera) -> Instance | None:
"""Get `Instance` associated with `camera`.
Args:
camera: `Camera` to get `Instance`.
Returns:
`Instance` associated with `camera` or None if not found.
"""
return self._instance_by_camera.get(camera, None)
def __repr__(self) -> str:
"""Return a readable representation of the instance group."""
n_cams = len(self._instance_by_camera)
has_3d = self._instance_3d is not None
parts = [f"InstanceGroup(n_cameras={n_cams}, has_3d={has_3d}"]
if self.identity is not None:
parts.append(f', identity="{self.identity.name}"')
if self.category is not None:
parts.append(f', category="{self.category.name}"')
parts.append(")")
return "".join(parts)
__annotations__ = {'_instance_by_camera': 'dict[Camera, Instance]', '_score': 'float | None', '_instance_3d': "'Instance3D | None'", 'identity': "'Identity | None'", 'category': "'Category | None'", 'metadata': 'dict'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'Defines a group of instances across the same frame index.\n\nAttributes:\n instances_by_camera: Dictionary of `Instance` objects by `Camera`.\n instances: List of `Instance` objects in the group.\n cameras: List of `Camera` objects that have an `Instance` associated.\n score: Optional score for the `InstanceGroup`. Setting the score will also\n update the score for all `instances` already in the `InstanceGroup`. The\n score for `instances` will not be updated upon initialization.\n instance_3d: Optional `Instance3D` with triangulated 3D keypoints.\n points: Optional 3D points for the `InstanceGroup`. Delegates to\n `instance_3d.points` if present.\n identity: Optional `Identity` for this group (which animal).\n category: Optional `Category` for this group (which class). Mirrors\n `identity` but groups by class rather than individual.\n metadata: Dictionary of metadata.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 444
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('_instance_by_camera', '_score', '_instance_3d', 'identity', 'category', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.camera'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('_instance_by_camera', '_score', '_instance_3d', 'identity', 'category', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('_instance_3d',)
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
cameras
property
¶
List of Camera objects.
instance_3d
property
¶
The 3D instance for this group.
instance_by_camera
property
¶
Get dictionary of Instance objects by Camera.
instances
property
¶
List of Instance objects.
points
property
¶
3D keypoint coordinates. Delegates to instance_3d.points.
score
property
¶
Get score for InstanceGroup.
__init__(instance_by_camera=NOTHING, score=None, instance_3d=None, identity=None, category=None, metadata=NOTHING)
¶
Method generated by attrs for class InstanceGroup.
Source code in sleap_io/model/camera.py
"""Data structure for a single camera view in a multi-camera setup."""
from __future__ import annotations
import attrs
import numpy as np
from attrs import define, field
from attrs.validators import instance_of
from sleap_io.model.category import Category
from sleap_io.model.identity import Identity
from sleap_io.model.instance import Instance, Instance3D
from sleap_io.model.labeled_frame import LabeledFrame
from sleap_io.model.video import Video
def rodrigues_transformation(input_matrix: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
__repr__()
¶
Return a readable representation of the instance group.
Source code in sleap_io/model/camera.py
def __repr__(self) -> str:
"""Return a readable representation of the instance group."""
n_cams = len(self._instance_by_camera)
has_3d = self._instance_3d is not None
parts = [f"InstanceGroup(n_cameras={n_cams}, has_3d={has_3d}"]
if self.identity is not None:
parts.append(f', identity="{self.identity.name}"')
if self.category is not None:
parts.append(f', category="{self.category.name}"')
parts.append(")")
return "".join(parts)
__setattr__(name, val)
¶
Method generated by attrs for class InstanceGroup.
LabeledFrame
¶
Labeled data for a single frame of a video.
Attributes:
| Name | Type | Description |
|---|---|---|
video |
The |
|
frame_idx |
The index of the |
|
instances |
List of |
|
is_negative |
If True, this frame is explicitly marked as containing no instances (a "negative" or background frame for training). This is distinct from frames that are simply empty (e.g., instances were deleted). |
|
centroids |
List of |
|
bboxes |
List of |
|
masks |
List of |
|
label_images |
List of |
|
rois |
List of |
Notes
Instances of this class are hashed by identity, not by value. This means that
two LabeledFrame instances with the same attributes will NOT be considered
equal in a set or dict.
Methods:
| Name | Description |
|---|---|
__getitem__ |
Return the |
__init__ |
Method generated by attrs for class LabeledFrame. |
__iter__ |
Iterate over |
__len__ |
Return the number of instances in the frame. |
__repr__ |
Method generated by attrs for class LabeledFrame. |
__setattr__ |
Method generated by attrs for class LabeledFrame. |
append |
Append an annotation to the appropriate frame-level container. |
convert |
Convert annotations between detection modalities. |
matches |
Check if this frame matches another frame's identity. |
merge |
Merge instances from another frame into this frame. |
numpy |
Return all instances in the frame as a numpy array. |
remove_empty_instances |
Remove all instances with no visible points. |
remove_predictions |
Remove all predicted instances and annotations from the frame. |
similarity_to |
Calculate instance overlap metrics with another frame. |
Source code in sleap_io/model/labeled_frame.py
@define(eq=False)
class LabeledFrame:
"""Labeled data for a single frame of a video.
Attributes:
video: The `Video` associated with this `LabeledFrame`.
frame_idx: The index of the `LabeledFrame` in the `Video`.
instances: List of `Instance` objects associated with this `LabeledFrame`.
is_negative: If True, this frame is explicitly marked as containing no
instances (a "negative" or background frame for training). This is
distinct from frames that are simply empty (e.g., instances were deleted).
centroids: List of `Centroid` annotations for this frame.
bboxes: List of `BoundingBox` annotations for this frame.
masks: List of `SegmentationMask` annotations for this frame.
label_images: List of `LabelImage` annotations for this frame.
rois: List of `ROI` annotations for this frame.
Notes:
Instances of this class are hashed by identity, not by value. This means that
two `LabeledFrame` instances with the same attributes will NOT be considered
equal in a set or dict.
"""
video: Video
frame_idx: int = field(converter=int)
instances: list[Instance | PredictedInstance] = field(factory=list)
is_negative: bool = field(default=False)
centroids: "list[Centroid]" = field(factory=list)
bboxes: "list[BoundingBox]" = field(factory=list)
masks: "list[SegmentationMask]" = field(factory=list)
label_images: "list[LabelImage]" = field(factory=list)
rois: "list[ROI]" = field(factory=list)
def append(
self,
annotation: (
"Instance | PredictedInstance | Centroid"
" | BoundingBox | SegmentationMask | LabelImage | ROI"
),
) -> None:
"""Append an annotation to the appropriate frame-level container.
Routes the annotation to the correct list based on its type:
``Instance``/``PredictedInstance`` → ``instances``,
``Centroid`` → ``centroids``, ``BoundingBox`` → ``bboxes``,
``SegmentationMask`` → ``masks``, ``LabelImage`` → ``label_images``,
``ROI`` → ``rois``.
Args:
annotation: The annotation object to add.
Raises:
TypeError: If the annotation type is not recognized.
"""
from sleap_io.model.bbox import BoundingBox
from sleap_io.model.centroid import Centroid
from sleap_io.model.label_image import LabelImage
from sleap_io.model.mask import SegmentationMask
from sleap_io.model.roi import ROI
if isinstance(annotation, (Instance, PredictedInstance)):
self.instances.append(annotation)
elif isinstance(annotation, Centroid):
self.centroids.append(annotation)
elif isinstance(annotation, BoundingBox):
self.bboxes.append(annotation)
elif isinstance(annotation, SegmentationMask):
self.masks.append(annotation)
elif isinstance(annotation, LabelImage):
self.label_images.append(annotation)
elif isinstance(annotation, ROI):
self.rois.append(annotation)
else:
raise TypeError(
f"Cannot append {type(annotation).__name__} to LabeledFrame. "
f"Expected one of: Instance, PredictedInstance, Centroid, "
f"BoundingBox, SegmentationMask, LabelImage, ROI."
)
def __len__(self) -> int:
"""Return the number of instances in the frame."""
return len(self.instances)
def __getitem__(self, key: int) -> Instance | PredictedInstance:
"""Return the `Instance` at `key` index in the `instances` list."""
return self.instances[key]
def __iter__(self):
"""Iterate over `Instance`s in `instances` list."""
return iter(self.instances)
@property
def user_instances(self) -> list[Instance]:
"""Frame instances that are user-labeled (`Instance` objects)."""
return [inst for inst in self.instances if type(inst) is Instance]
@property
def has_user_instances(self) -> bool:
"""Return True if the frame has any user-labeled instances."""
for inst in self.instances:
if type(inst) is Instance:
return True
return False
@property
def is_user_labeled(self) -> bool:
"""Return True if frame has user instances/annotations OR is negative.
This property indicates whether the frame represents intentional user
annotation, either through labeled instances, user annotations
(centroids, bboxes, ROIs, masks, label images), or explicit marking as a
negative/background frame.
"""
from sleap_io.model.label_image import PredictedLabelImage
from sleap_io.model.mask import PredictedSegmentationMask
return (
self.has_user_instances
or self.is_negative
or any(not c.is_predicted for c in self.centroids)
or any(not b.is_predicted for b in self.bboxes)
or any(not r.is_predicted for r in self.rois)
or any(not isinstance(m, PredictedSegmentationMask) for m in self.masks)
or any(not isinstance(li, PredictedLabelImage) for li in self.label_images)
)
@property
def predicted_instances(self) -> list[Instance]:
"""Frame instances that are predicted by a model (`PredictedInstance`)."""
return [inst for inst in self.instances if type(inst) is PredictedInstance]
@property
def has_predicted_instances(self) -> bool:
"""Return True if the frame has any predicted instances."""
for inst in self.instances:
if type(inst) is PredictedInstance:
return True
return False
def numpy(self) -> np.ndarray:
"""Return all instances in the frame as a numpy array.
Returns:
Points as a numpy array of shape `(n_instances, n_nodes, 2)`.
Note that the order of the instances is arbitrary.
"""
n_instances = len(self.instances)
n_nodes = len(self.instances[0]) if n_instances > 0 else 0
pts = np.full((n_instances, n_nodes, 2), np.nan)
for i, inst in enumerate(self.instances):
pts[i] = inst.numpy()[:, 0:2]
return pts
@property
def image(self) -> np.ndarray:
"""Return the image of the frame as a numpy array."""
return self.video[self.frame_idx]
@property
def unused_predictions(self) -> list[Instance]:
"""Return a list of "unused" `PredictedInstance` objects in frame.
This is all of the `PredictedInstance` objects which do not have a corresponding
`Instance` in the same track in the same frame.
"""
unused_predictions = []
any_tracks = [inst.track for inst in self.instances if inst.track is not None]
if len(any_tracks):
# Use tracks to determine which predicted instances have been used
used_tracks = [
inst.track
for inst in self.instances
if type(inst) is Instance and inst.track is not None
]
unused_predictions = [
inst
for inst in self.instances
if inst.track not in used_tracks and type(inst) is PredictedInstance
]
else:
# Use from_predicted to determine which predicted instances have been used
# TODO: should we always do this instead of using tracks?
used_instances = [
inst.from_predicted
for inst in self.instances
if inst.from_predicted is not None
]
unused_predictions = [
inst
for inst in self.instances
if type(inst) is PredictedInstance and inst not in used_instances
]
return unused_predictions
@property
def unused_predicted_masks(self) -> list["SegmentationMask"]:
"""Return predicted masks in this frame not yet adopted by a user mask.
A `PredictedSegmentationMask` is considered *adopted* (and so excluded
from the result) when some `UserSegmentationMask` in the same frame
either links to it via `from_predicted` (checked first) or, lacking an
explicit link, spatially overlaps it (bbox-centroid distance within 5 px,
the auto-merge default). This mirrors the link-first, spatial-fallback
precedence used by the auto-merge cascade and supports the
"retrain only what a human corrected" workflow.
This is the segmentation-mask analogue of `unused_predictions` (which
covers `PredictedInstance` objects).
Returns:
The `PredictedSegmentationMask` objects with no adopting user mask.
"""
from sleap_io.model.mask import PredictedSegmentationMask
predicted = [m for m in self.masks if isinstance(m, PredictedSegmentationMask)]
if not predicted:
return []
user_masks = [m for m in self.masks if not m.is_predicted]
adopted: set[int] = set()
# Link-first: predicted masks explicitly adopted via from_predicted.
for u in user_masks:
src = getattr(u, "from_predicted", None)
if src is not None:
adopted.add(id(src))
# Spatial fallback: a user mask overlaps a still-unadopted prediction.
remaining = [m for m in predicted if id(m) not in adopted]
if remaining and user_masks:
for self_idx, _other_idx, _score in _find_annotation_matches(
remaining, user_masks, "masks", 5.0
):
adopted.add(id(remaining[self_idx]))
return [m for m in predicted if id(m) not in adopted]
def remove_predictions(self):
"""Remove all predicted instances and annotations from the frame."""
from sleap_io.model.bbox import PredictedBoundingBox
from sleap_io.model.centroid import PredictedCentroid
from sleap_io.model.label_image import PredictedLabelImage
from sleap_io.model.mask import PredictedSegmentationMask
from sleap_io.model.roi import PredictedROI
self.instances = [inst for inst in self.instances if type(inst) is Instance]
self.centroids = [
c for c in self.centroids if not isinstance(c, PredictedCentroid)
]
self.bboxes = [
b for b in self.bboxes if not isinstance(b, PredictedBoundingBox)
]
self.masks = [
m for m in self.masks if not isinstance(m, PredictedSegmentationMask)
]
self.label_images = [
li for li in self.label_images if not isinstance(li, PredictedLabelImage)
]
self.rois = [r for r in self.rois if not isinstance(r, PredictedROI)]
def remove_empty_instances(self):
"""Remove all instances with no visible points."""
self.instances = [inst for inst in self.instances if not inst.is_empty]
def convert(
self,
to: str,
source: str = "pose",
inplace: bool = False,
**kwargs,
) -> list:
"""Convert annotations between detection modalities.
Reads every annotation of the ``source`` modality from this frame and
converts each one to the ``to`` modality by dispatching to the matching
per-object verb (``to_centroid``, ``to_bbox``, ``to_mask``, ``to_roi`` or
``to_pose``). Keyword arguments are forwarded unchanged to the per-object
verb (e.g. ``height``/``width`` for ``to="mask"``).
Args:
to: Target modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``.
source: Source modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``. Reads from the matching frame list
(``instances``, ``centroids``, ``bboxes``, ``masks`` or ``rois``).
inplace: If ``True``, append each produced annotation to this frame
(via `append`) in addition to returning them. If ``False``
(default), the frame is left unmodified.
**kwargs: Forwarded to the per-object conversion verb.
Returns:
A list of the produced annotations (one per source annotation), of the
``to`` modality.
Raises:
ValueError: If ``to`` or ``source`` is not a recognized modality, if
``to="pose"`` is requested from a non-centroid source (only
``centroid`` → ``pose`` is defined), or if a source annotation
lacks the target conversion verb.
"""
modalities = {
"pose": "instances",
"centroid": "centroids",
"bbox": "bboxes",
"mask": "masks",
"roi": "rois",
}
if to not in modalities:
raise ValueError(
f"Unknown target modality {to!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if source not in modalities:
raise ValueError(
f"Unknown source modality {source!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if to == "pose" and source != "centroid":
raise ValueError(
f"Conversion from {source!r} to 'pose' is not supported; only "
"'centroid' -> 'pose' is defined."
)
verb = "to_pose" if to == "pose" else f"to_{to}"
sources = getattr(self, modalities[source])
results = []
for obj in sources:
method = getattr(obj, verb, None)
if method is None:
raise ValueError(
f"Cannot convert {source!r} to {to!r}: "
f"{type(obj).__name__} has no {verb}() method."
)
result = method(**kwargs)
results.append(result)
if inplace:
self.append(result)
return results
def matches(self, other: "LabeledFrame", video_must_match: bool = True) -> bool:
"""Check if this frame matches another frame's identity.
Args:
other: Another LabeledFrame to compare with.
video_must_match: If True, frames must be from the same video.
If False, only frame index needs to match.
Returns:
True if the frames have the same identity, False otherwise.
Notes:
Frame identity is determined by video and frame index.
This does not compare the instances within the frame.
"""
if self.frame_idx != other.frame_idx:
return False
if video_must_match:
# Check if videos are the same object
if self.video is other.video:
return True
# Check if videos have matching paths
return self.video.matches_path(other.video, strict=False)
return True
def similarity_to(self, other: "LabeledFrame") -> dict[str, any]:
"""Calculate instance overlap metrics with another frame.
Args:
other: Another LabeledFrame to compare with.
Returns:
A dictionary with similarity metrics:
- 'n_user_self': Number of user instances in this frame
- 'n_user_other': Number of user instances in the other frame
- 'n_pred_self': Number of predicted instances in this frame
- 'n_pred_other': Number of predicted instances in the other frame
- 'n_overlapping': Number of instances that overlap (by IoU)
- 'mean_pose_distance': Mean distance between matching poses
"""
metrics = {
"n_user_self": len(self.user_instances),
"n_user_other": len(other.user_instances),
"n_pred_self": len(self.predicted_instances),
"n_pred_other": len(other.predicted_instances),
"n_overlapping": 0,
"mean_pose_distance": None,
}
# Count overlapping instances and compute pose distances
pose_distances = []
for inst1 in self.instances:
for inst2 in other.instances:
# Check if instances overlap
if inst1.overlaps_with(inst2, iou_threshold=0.1):
metrics["n_overlapping"] += 1
# If they have the same skeleton, compute pose distance
if inst1.skeleton.matches(inst2.skeleton):
# Get visible points for both
pts1 = inst1.numpy()
pts2 = inst2.numpy()
# Compute distances for visible points in both
valid = ~(np.isnan(pts1[:, 0]) | np.isnan(pts2[:, 0]))
if valid.any():
distances = np.linalg.norm(
pts1[valid] - pts2[valid], axis=1
)
pose_distances.extend(distances.tolist())
if pose_distances:
metrics["mean_pose_distance"] = np.mean(pose_distances)
return metrics
def merge(
self,
other: "LabeledFrame",
instance: "InstanceMatcher | None" = None,
frame: str = "auto",
) -> tuple[list[Instance], list[tuple[Instance, Instance, str]]]:
"""Merge instances from another frame into this frame.
Args:
other: Another LabeledFrame to merge instances from.
instance: Matcher to use for finding duplicate instances.
If None, uses default spatial matching with 5px tolerance.
frame: Merge strategy:
- "auto": Keep user labels, update predictions only if no user label
- "keep_original": Keep all original instances, ignore new ones
- "keep_new": Replace with new instances
- "keep_both": Keep all instances from both frames
- "update_tracks": Update track and score of the original instances
from the new instances.
- "replace_predictions": Keep all user instances from original frame,
remove all predictions from original frame, add only predictions
from the incoming frame. No spatial matching is performed.
Returns:
A tuple of (merged_instances, conflicts) where:
- merged_instances: List of instances after merging
- conflicts: List of (original, new, resolution) tuples for conflicts
Notes:
The merged instance list is returned (not assigned back) so the
caller can decide what to do with it. Frame-level annotations
(centroids, bboxes, masks, label images, rois) and the
``is_negative`` flag are updated on this frame in place.
"""
from sleap_io.model.matching import InstanceMatcher, InstanceMatchMethod
if instance is None:
instance_matcher = InstanceMatcher(
method=InstanceMatchMethod.SPATIAL, threshold=5.0
)
else:
instance_matcher = instance
conflicts = []
if frame == "keep_original":
self._merge_annotations(other, strategy="keep_original")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances.copy(), conflicts
elif frame == "keep_new":
self._merge_annotations(other, strategy="keep_new")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, other.instances
)
return other.instances.copy(), conflicts
elif frame == "keep_both":
self._merge_annotations(other, strategy="keep_both")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances + other.instances
)
return self.instances + other.instances, conflicts
elif frame == "update_tracks":
# match instances and update .track and tracking score of the old instances
matches = instance_matcher.find_matches(self.instances, other.instances)
for self_idx, other_idx, score in matches:
self.instances[self_idx].track = other.instances[other_idx].track
self.instances[self_idx].tracking_score = other.instances[
other_idx
].tracking_score
self._merge_annotations(
other,
strategy="update_tracks",
threshold=instance_matcher.threshold,
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances, conflicts
elif frame == "replace_predictions":
# Keep all user instances from original frame
merged = [inst for inst in self.instances if type(inst) is Instance]
# Add only predictions from incoming frame (not user instances)
merged.extend(
inst for inst in other.instances if type(inst) is PredictedInstance
)
self._merge_annotations(other, strategy="replace_predictions")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged
)
# No instance conflicts to report - this is a clean replacement
return merged, []
# Auto merging strategy
merged_instances = []
used_indices = set()
# First, keep all user instances from self
for inst in self.instances:
if type(inst) is Instance:
merged_instances.append(inst)
# Find matches between instances
matches = instance_matcher.find_matches(self.instances, other.instances)
# Group matches by instance in other frame
other_to_self = {}
for self_idx, other_idx, score in matches:
if other_idx not in other_to_self or score > other_to_self[other_idx][1]:
other_to_self[other_idx] = (self_idx, score)
# Process instances from other frame
for other_idx, other_inst in enumerate(other.instances):
if other_idx in other_to_self:
self_idx, score = other_to_self[other_idx]
self_inst = self.instances[self_idx]
# Check for conflicts
if type(self_inst) is Instance and type(other_inst) is Instance:
# Both are user instances - conflict
conflicts.append((self_inst, other_inst, "kept_original"))
used_indices.add(self_idx)
elif (
type(self_inst) is PredictedInstance
and type(other_inst) is Instance
):
# Replace prediction with user instance
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
elif (
type(self_inst) is Instance
and type(other_inst) is PredictedInstance
):
# Keep user instance, ignore prediction
conflicts.append((self_inst, other_inst, "kept_user"))
used_indices.add(self_idx)
else:
# Both are predictions - keep the new one
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
else:
# No match found, add new instance
merged_instances.append(other_inst)
# Add remaining instances from self that weren't matched
for self_idx, self_inst in enumerate(self.instances):
if type(self_inst) is PredictedInstance and self_idx not in used_indices:
# Check if this prediction should be kept
# NOTE: This defensive logic should be unreachable under normal
# circumstances since all matched instances should have been added to
# used_indices above. However, we keep this as a safety net for edge
# cases or future changes.
keep = True
for other_idx, (matched_self_idx, _) in other_to_self.items():
if matched_self_idx == self_idx:
keep = False
break
if keep:
merged_instances.append(self_inst)
# Merge annotations from the other frame (spatial matching + resolution)
self._merge_annotations(
other, strategy="auto", threshold=instance_matcher.threshold
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged_instances
)
return merged_instances, conflicts
def _merge_annotations(
self,
other: "LabeledFrame",
strategy: str = "keep_both",
threshold: float = 5.0,
):
"""Merge annotation lists from another frame into this frame.
Shallow-copies annotations from the other frame to avoid mutating the
source when references are later remapped. Video and track references
are preserved so that ``_remap_frame_annotations`` can find them in
the mapping dicts.
Args:
other: The frame to merge annotations from.
strategy: The merge strategy, matching the ``frame`` parameter of
``merge()``. Controls which annotations are kept:
- ``"keep_original"``: Keep self only.
- ``"keep_new"``: Replace with other's annotations.
- ``"keep_both"``: Keep self + add other's (default).
- ``"replace_predictions"``: Keep user from self, replace
predicted with other's predicted.
- ``"auto"``: Spatial matching + user-vs-predicted resolution
cascade (mirrors instance auto-merge logic).
- ``"update_tracks"``: Spatial matching, then update track
assignments on matched self annotations.
threshold: Maximum centroid distance (pixels) for spatial matching
in ``"auto"`` and ``"update_tracks"`` strategies.
"""
attrs = ("centroids", "bboxes", "masks", "label_images", "rois")
if strategy == "keep_original":
return
if strategy == "keep_new":
for attr in attrs:
memo: dict[int, Any] = {}
new_list = [
_copy_with_memo(item, memo) for item in getattr(other, attr)
]
_relink_from_predicted(new_list, memo)
setattr(self, attr, new_list)
return
if strategy == "replace_predictions":
for attr in attrs:
memo = {}
kept = [a for a in getattr(self, attr) if not a.is_predicted]
for item in getattr(other, attr):
if item.is_predicted:
kept.append(_copy_with_memo(item, memo))
_relink_from_predicted(kept, memo)
setattr(self, attr, kept)
return
if strategy == "auto":
for attr in attrs:
setattr(
self,
attr,
_resolve_annotation_auto(
getattr(self, attr), getattr(other, attr), attr, threshold
),
)
return
if strategy == "update_tracks":
for attr in attrs:
_resolve_annotation_update_tracks(
getattr(self, attr), getattr(other, attr), attr, threshold
)
return
# "keep_both" (default)
for attr in attrs:
memo = {}
target = getattr(self, attr)
existing_ids = set(id(x) for x in target)
for item in getattr(other, attr):
if id(item) not in existing_ids:
target.append(_copy_with_memo(item, memo))
_relink_from_predicted(target, memo)
__annotations__ = {'video': 'Video', 'frame_idx': 'int', 'instances': 'list[Instance | PredictedInstance]', 'is_negative': 'bool', 'centroids': "'list[Centroid]'", 'bboxes': "'list[BoundingBox]'", 'masks': "'list[SegmentationMask]'", 'label_images': "'list[LabelImage]'", 'rois': "'list[ROI]'"}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=True, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'Labeled data for a single frame of a video.\n\nAttributes:\n video: The `Video` associated with this `LabeledFrame`.\n frame_idx: The index of the `LabeledFrame` in the `Video`.\n instances: List of `Instance` objects associated with this `LabeledFrame`.\n is_negative: If True, this frame is explicitly marked as containing no\n instances (a "negative" or background frame for training). This is\n distinct from frames that are simply empty (e.g., instances were deleted).\n centroids: List of `Centroid` annotations for this frame.\n bboxes: List of `BoundingBox` annotations for this frame.\n masks: List of `SegmentationMask` annotations for this frame.\n label_images: List of `LabelImage` annotations for this frame.\n rois: List of `ROI` annotations for this frame.\n\nNotes:\n Instances of this class are hashed by identity, not by value. This means that\n two `LabeledFrame` instances with the same attributes will NOT be considered\n equal in a set or dict.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 329
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('video', 'frame_idx', 'instances', 'is_negative', 'centroids', 'bboxes', 'masks', 'label_images', 'rois')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.labeled_frame'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('video', 'frame_idx', 'instances', 'is_negative', 'centroids', 'bboxes', 'masks', 'label_images', 'rois', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('bboxes', 'centroids', 'instances', 'is_negative', 'label_images', 'masks', 'rois')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
has_predicted_instances
property
¶
Return True if the frame has any predicted instances.
has_user_instances
property
¶
Return True if the frame has any user-labeled instances.
image
property
¶
Return the image of the frame as a numpy array.
is_user_labeled
property
¶
Return True if frame has user instances/annotations OR is negative.
This property indicates whether the frame represents intentional user annotation, either through labeled instances, user annotations (centroids, bboxes, ROIs, masks, label images), or explicit marking as a negative/background frame.
predicted_instances
property
¶
Frame instances that are predicted by a model (PredictedInstance).
unused_predicted_masks
property
¶
Return predicted masks in this frame not yet adopted by a user mask.
A PredictedSegmentationMask is considered adopted (and so excluded
from the result) when some UserSegmentationMask in the same frame
either links to it via from_predicted (checked first) or, lacking an
explicit link, spatially overlaps it (bbox-centroid distance within 5 px,
the auto-merge default). This mirrors the link-first, spatial-fallback
precedence used by the auto-merge cascade and supports the
"retrain only what a human corrected" workflow.
This is the segmentation-mask analogue of unused_predictions (which
covers PredictedInstance objects).
Returns:
| Type | Description |
|---|---|
|
The |
unused_predictions
property
¶
Return a list of "unused" PredictedInstance objects in frame.
This is all of the PredictedInstance objects which do not have a corresponding
Instance in the same track in the same frame.
user_instances
property
¶
Frame instances that are user-labeled (Instance objects).
__getitem__(key)
¶
__init__(video, frame_idx, instances=NOTHING, is_negative=False, centroids=NOTHING, bboxes=NOTHING, masks=NOTHING, label_images=NOTHING, rois=NOTHING)
¶
Method generated by attrs for class LabeledFrame.
Source code in sleap_io/model/labeled_frame.py
from sleap_io.model.instance import Instance, PredictedInstance
from sleap_io.model.video import Video
if TYPE_CHECKING:
from sleap_io.model.bbox import BoundingBox
from sleap_io.model.centroid import Centroid
from sleap_io.model.label_image import LabelImage
from sleap_io.model.mask import SegmentationMask
from sleap_io.model.matching import InstanceMatcher
from sleap_io.model.roi import ROI
def _annotation_centroid_xy(annotation: Any, attr: str) -> tuple[float, float] | None:
"""Extract centroid (x, y) from an annotation based on its modality.
Args:
annotation: An annotation object (Centroid, BoundingBox, etc.).
attr: The attribute name indicating the modality.
Returns:
A tuple of (x, y) coordinates, or ``None`` if the centroid cannot be
computed (e.g., empty mask or empty ROI geometry).
"""
if attr == "centroids":
return (annotation.x, annotation.y)
elif attr == "bboxes":
return annotation.centroid_xy
elif attr == "rois":
if annotation.geometry.is_empty:
__iter__()
¶
__len__()
¶
__repr__()
¶
Method generated by attrs for class LabeledFrame.
Source code in sleap_io/model/labeled_frame.py
"""Data structures for data contained within a single video frame.
The `LabeledFrame` class is a data structure that contains `Instance`s and
`PredictedInstance`s that are associated with a single frame within a video.
"""
from __future__ import annotations
import math
from copy import copy
from typing import TYPE_CHECKING, Any
import numpy as np
from attrs import define, field
__setattr__(name, val)
¶
Method generated by attrs for class LabeledFrame.
append(annotation)
¶
Append an annotation to the appropriate frame-level container.
Routes the annotation to the correct list based on its type:
Instance/PredictedInstance → instances,
Centroid → centroids, BoundingBox → bboxes,
SegmentationMask → masks, LabelImage → label_images,
ROI → rois.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
annotation
|
Instance | PredictedInstance | Centroid | BoundingBox | SegmentationMask | LabelImage | ROI
|
The annotation object to add. |
required |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the annotation type is not recognized. |
Source code in sleap_io/model/labeled_frame.py
def append(
self,
annotation: (
"Instance | PredictedInstance | Centroid"
" | BoundingBox | SegmentationMask | LabelImage | ROI"
),
) -> None:
"""Append an annotation to the appropriate frame-level container.
Routes the annotation to the correct list based on its type:
``Instance``/``PredictedInstance`` → ``instances``,
``Centroid`` → ``centroids``, ``BoundingBox`` → ``bboxes``,
``SegmentationMask`` → ``masks``, ``LabelImage`` → ``label_images``,
``ROI`` → ``rois``.
Args:
annotation: The annotation object to add.
Raises:
TypeError: If the annotation type is not recognized.
"""
from sleap_io.model.bbox import BoundingBox
from sleap_io.model.centroid import Centroid
from sleap_io.model.label_image import LabelImage
from sleap_io.model.mask import SegmentationMask
from sleap_io.model.roi import ROI
if isinstance(annotation, (Instance, PredictedInstance)):
self.instances.append(annotation)
elif isinstance(annotation, Centroid):
self.centroids.append(annotation)
elif isinstance(annotation, BoundingBox):
self.bboxes.append(annotation)
elif isinstance(annotation, SegmentationMask):
self.masks.append(annotation)
elif isinstance(annotation, LabelImage):
self.label_images.append(annotation)
elif isinstance(annotation, ROI):
self.rois.append(annotation)
else:
raise TypeError(
f"Cannot append {type(annotation).__name__} to LabeledFrame. "
f"Expected one of: Instance, PredictedInstance, Centroid, "
f"BoundingBox, SegmentationMask, LabelImage, ROI."
)
convert(to, source='pose', inplace=False, **kwargs)
¶
Convert annotations between detection modalities.
Reads every annotation of the source modality from this frame and
converts each one to the to modality by dispatching to the matching
per-object verb (to_centroid, to_bbox, to_mask, to_roi or
to_pose). Keyword arguments are forwarded unchanged to the per-object
verb (e.g. height/width for to="mask").
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
to
|
str
|
Target modality, one of |
required |
source
|
str
|
Source modality, one of |
'pose'
|
inplace
|
bool
|
If |
False
|
**kwargs
|
Forwarded to the per-object conversion verb. |
required |
Returns:
| Type | Description |
|---|---|
list
|
A list of the produced annotations (one per source annotation), of the
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in sleap_io/model/labeled_frame.py
def convert(
self,
to: str,
source: str = "pose",
inplace: bool = False,
**kwargs,
) -> list:
"""Convert annotations between detection modalities.
Reads every annotation of the ``source`` modality from this frame and
converts each one to the ``to`` modality by dispatching to the matching
per-object verb (``to_centroid``, ``to_bbox``, ``to_mask``, ``to_roi`` or
``to_pose``). Keyword arguments are forwarded unchanged to the per-object
verb (e.g. ``height``/``width`` for ``to="mask"``).
Args:
to: Target modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``.
source: Source modality, one of ``"pose"``, ``"centroid"``, ``"bbox"``,
``"mask"`` or ``"roi"``. Reads from the matching frame list
(``instances``, ``centroids``, ``bboxes``, ``masks`` or ``rois``).
inplace: If ``True``, append each produced annotation to this frame
(via `append`) in addition to returning them. If ``False``
(default), the frame is left unmodified.
**kwargs: Forwarded to the per-object conversion verb.
Returns:
A list of the produced annotations (one per source annotation), of the
``to`` modality.
Raises:
ValueError: If ``to`` or ``source`` is not a recognized modality, if
``to="pose"`` is requested from a non-centroid source (only
``centroid`` → ``pose`` is defined), or if a source annotation
lacks the target conversion verb.
"""
modalities = {
"pose": "instances",
"centroid": "centroids",
"bbox": "bboxes",
"mask": "masks",
"roi": "rois",
}
if to not in modalities:
raise ValueError(
f"Unknown target modality {to!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if source not in modalities:
raise ValueError(
f"Unknown source modality {source!r}. Expected one of: "
f"{', '.join(modalities)}."
)
if to == "pose" and source != "centroid":
raise ValueError(
f"Conversion from {source!r} to 'pose' is not supported; only "
"'centroid' -> 'pose' is defined."
)
verb = "to_pose" if to == "pose" else f"to_{to}"
sources = getattr(self, modalities[source])
results = []
for obj in sources:
method = getattr(obj, verb, None)
if method is None:
raise ValueError(
f"Cannot convert {source!r} to {to!r}: "
f"{type(obj).__name__} has no {verb}() method."
)
result = method(**kwargs)
results.append(result)
if inplace:
self.append(result)
return results
matches(other, video_must_match=True)
¶
Check if this frame matches another frame's identity.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
LabeledFrame
|
Another LabeledFrame to compare with. |
required |
video_must_match
|
bool
|
If True, frames must be from the same video. If False, only frame index needs to match. |
True
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the frames have the same identity, False otherwise. |
Notes
Frame identity is determined by video and frame index. This does not compare the instances within the frame.
Source code in sleap_io/model/labeled_frame.py
def matches(self, other: "LabeledFrame", video_must_match: bool = True) -> bool:
"""Check if this frame matches another frame's identity.
Args:
other: Another LabeledFrame to compare with.
video_must_match: If True, frames must be from the same video.
If False, only frame index needs to match.
Returns:
True if the frames have the same identity, False otherwise.
Notes:
Frame identity is determined by video and frame index.
This does not compare the instances within the frame.
"""
if self.frame_idx != other.frame_idx:
return False
if video_must_match:
# Check if videos are the same object
if self.video is other.video:
return True
# Check if videos have matching paths
return self.video.matches_path(other.video, strict=False)
return True
merge(other, instance=None, frame='auto')
¶
Merge instances from another frame into this frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
LabeledFrame
|
Another LabeledFrame to merge instances from. |
required |
instance
|
InstanceMatcher | None
|
Matcher to use for finding duplicate instances. If None, uses default spatial matching with 5px tolerance. |
None
|
frame
|
str
|
Merge strategy: - "auto": Keep user labels, update predictions only if no user label - "keep_original": Keep all original instances, ignore new ones - "keep_new": Replace with new instances - "keep_both": Keep all instances from both frames - "update_tracks": Update track and score of the original instances from the new instances. - "replace_predictions": Keep all user instances from original frame, remove all predictions from original frame, add only predictions from the incoming frame. No spatial matching is performed. |
'auto'
|
Returns:
| Type | Description |
|---|---|
tuple[list[Instance], list[tuple[Instance, Instance, str]]]
|
A tuple of (merged_instances, conflicts) where: - merged_instances: List of instances after merging - conflicts: List of (original, new, resolution) tuples for conflicts |
Notes
The merged instance list is returned (not assigned back) so the
caller can decide what to do with it. Frame-level annotations
(centroids, bboxes, masks, label images, rois) and the
is_negative flag are updated on this frame in place.
Source code in sleap_io/model/labeled_frame.py
def merge(
self,
other: "LabeledFrame",
instance: "InstanceMatcher | None" = None,
frame: str = "auto",
) -> tuple[list[Instance], list[tuple[Instance, Instance, str]]]:
"""Merge instances from another frame into this frame.
Args:
other: Another LabeledFrame to merge instances from.
instance: Matcher to use for finding duplicate instances.
If None, uses default spatial matching with 5px tolerance.
frame: Merge strategy:
- "auto": Keep user labels, update predictions only if no user label
- "keep_original": Keep all original instances, ignore new ones
- "keep_new": Replace with new instances
- "keep_both": Keep all instances from both frames
- "update_tracks": Update track and score of the original instances
from the new instances.
- "replace_predictions": Keep all user instances from original frame,
remove all predictions from original frame, add only predictions
from the incoming frame. No spatial matching is performed.
Returns:
A tuple of (merged_instances, conflicts) where:
- merged_instances: List of instances after merging
- conflicts: List of (original, new, resolution) tuples for conflicts
Notes:
The merged instance list is returned (not assigned back) so the
caller can decide what to do with it. Frame-level annotations
(centroids, bboxes, masks, label images, rois) and the
``is_negative`` flag are updated on this frame in place.
"""
from sleap_io.model.matching import InstanceMatcher, InstanceMatchMethod
if instance is None:
instance_matcher = InstanceMatcher(
method=InstanceMatchMethod.SPATIAL, threshold=5.0
)
else:
instance_matcher = instance
conflicts = []
if frame == "keep_original":
self._merge_annotations(other, strategy="keep_original")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances.copy(), conflicts
elif frame == "keep_new":
self._merge_annotations(other, strategy="keep_new")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, other.instances
)
return other.instances.copy(), conflicts
elif frame == "keep_both":
self._merge_annotations(other, strategy="keep_both")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances + other.instances
)
return self.instances + other.instances, conflicts
elif frame == "update_tracks":
# match instances and update .track and tracking score of the old instances
matches = instance_matcher.find_matches(self.instances, other.instances)
for self_idx, other_idx, score in matches:
self.instances[self_idx].track = other.instances[other_idx].track
self.instances[self_idx].tracking_score = other.instances[
other_idx
].tracking_score
self._merge_annotations(
other,
strategy="update_tracks",
threshold=instance_matcher.threshold,
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, self.instances
)
return self.instances, conflicts
elif frame == "replace_predictions":
# Keep all user instances from original frame
merged = [inst for inst in self.instances if type(inst) is Instance]
# Add only predictions from incoming frame (not user instances)
merged.extend(
inst for inst in other.instances if type(inst) is PredictedInstance
)
self._merge_annotations(other, strategy="replace_predictions")
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged
)
# No instance conflicts to report - this is a clean replacement
return merged, []
# Auto merging strategy
merged_instances = []
used_indices = set()
# First, keep all user instances from self
for inst in self.instances:
if type(inst) is Instance:
merged_instances.append(inst)
# Find matches between instances
matches = instance_matcher.find_matches(self.instances, other.instances)
# Group matches by instance in other frame
other_to_self = {}
for self_idx, other_idx, score in matches:
if other_idx not in other_to_self or score > other_to_self[other_idx][1]:
other_to_self[other_idx] = (self_idx, score)
# Process instances from other frame
for other_idx, other_inst in enumerate(other.instances):
if other_idx in other_to_self:
self_idx, score = other_to_self[other_idx]
self_inst = self.instances[self_idx]
# Check for conflicts
if type(self_inst) is Instance and type(other_inst) is Instance:
# Both are user instances - conflict
conflicts.append((self_inst, other_inst, "kept_original"))
used_indices.add(self_idx)
elif (
type(self_inst) is PredictedInstance
and type(other_inst) is Instance
):
# Replace prediction with user instance
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
elif (
type(self_inst) is Instance
and type(other_inst) is PredictedInstance
):
# Keep user instance, ignore prediction
conflicts.append((self_inst, other_inst, "kept_user"))
used_indices.add(self_idx)
else:
# Both are predictions - keep the new one
if self_idx not in used_indices:
merged_instances.append(other_inst)
used_indices.add(self_idx)
else:
# No match found, add new instance
merged_instances.append(other_inst)
# Add remaining instances from self that weren't matched
for self_idx, self_inst in enumerate(self.instances):
if type(self_inst) is PredictedInstance and self_idx not in used_indices:
# Check if this prediction should be kept
# NOTE: This defensive logic should be unreachable under normal
# circumstances since all matched instances should have been added to
# used_indices above. However, we keep this as a safety net for edge
# cases or future changes.
keep = True
for other_idx, (matched_self_idx, _) in other_to_self.items():
if matched_self_idx == self_idx:
keep = False
break
if keep:
merged_instances.append(self_inst)
# Merge annotations from the other frame (spatial matching + resolution)
self._merge_annotations(
other, strategy="auto", threshold=instance_matcher.threshold
)
self.is_negative, _ = _resolve_merged_is_negative(
self.is_negative, other.is_negative, merged_instances
)
return merged_instances, conflicts
numpy()
¶
Return all instances in the frame as a numpy array.
Returns:
| Type | Description |
|---|---|
ndarray
|
Points as a numpy array of shape Note that the order of the instances is arbitrary. |
Source code in sleap_io/model/labeled_frame.py
def numpy(self) -> np.ndarray:
"""Return all instances in the frame as a numpy array.
Returns:
Points as a numpy array of shape `(n_instances, n_nodes, 2)`.
Note that the order of the instances is arbitrary.
"""
n_instances = len(self.instances)
n_nodes = len(self.instances[0]) if n_instances > 0 else 0
pts = np.full((n_instances, n_nodes, 2), np.nan)
for i, inst in enumerate(self.instances):
pts[i] = inst.numpy()[:, 0:2]
return pts
remove_empty_instances()
¶
remove_predictions()
¶
Remove all predicted instances and annotations from the frame.
Source code in sleap_io/model/labeled_frame.py
def remove_predictions(self):
"""Remove all predicted instances and annotations from the frame."""
from sleap_io.model.bbox import PredictedBoundingBox
from sleap_io.model.centroid import PredictedCentroid
from sleap_io.model.label_image import PredictedLabelImage
from sleap_io.model.mask import PredictedSegmentationMask
from sleap_io.model.roi import PredictedROI
self.instances = [inst for inst in self.instances if type(inst) is Instance]
self.centroids = [
c for c in self.centroids if not isinstance(c, PredictedCentroid)
]
self.bboxes = [
b for b in self.bboxes if not isinstance(b, PredictedBoundingBox)
]
self.masks = [
m for m in self.masks if not isinstance(m, PredictedSegmentationMask)
]
self.label_images = [
li for li in self.label_images if not isinstance(li, PredictedLabelImage)
]
self.rois = [r for r in self.rois if not isinstance(r, PredictedROI)]
similarity_to(other)
¶
Calculate instance overlap metrics with another frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
LabeledFrame
|
Another LabeledFrame to compare with. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, any]
|
A dictionary with similarity metrics: - 'n_user_self': Number of user instances in this frame - 'n_user_other': Number of user instances in the other frame - 'n_pred_self': Number of predicted instances in this frame - 'n_pred_other': Number of predicted instances in the other frame - 'n_overlapping': Number of instances that overlap (by IoU) - 'mean_pose_distance': Mean distance between matching poses |
Source code in sleap_io/model/labeled_frame.py
def similarity_to(self, other: "LabeledFrame") -> dict[str, any]:
"""Calculate instance overlap metrics with another frame.
Args:
other: Another LabeledFrame to compare with.
Returns:
A dictionary with similarity metrics:
- 'n_user_self': Number of user instances in this frame
- 'n_user_other': Number of user instances in the other frame
- 'n_pred_self': Number of predicted instances in this frame
- 'n_pred_other': Number of predicted instances in the other frame
- 'n_overlapping': Number of instances that overlap (by IoU)
- 'mean_pose_distance': Mean distance between matching poses
"""
metrics = {
"n_user_self": len(self.user_instances),
"n_user_other": len(other.user_instances),
"n_pred_self": len(self.predicted_instances),
"n_pred_other": len(other.predicted_instances),
"n_overlapping": 0,
"mean_pose_distance": None,
}
# Count overlapping instances and compute pose distances
pose_distances = []
for inst1 in self.instances:
for inst2 in other.instances:
# Check if instances overlap
if inst1.overlaps_with(inst2, iou_threshold=0.1):
metrics["n_overlapping"] += 1
# If they have the same skeleton, compute pose distance
if inst1.skeleton.matches(inst2.skeleton):
# Get visible points for both
pts1 = inst1.numpy()
pts2 = inst2.numpy()
# Compute distances for visible points in both
valid = ~(np.isnan(pts1[:, 0]) | np.isnan(pts2[:, 0]))
if valid.any():
distances = np.linalg.norm(
pts1[valid] - pts2[valid], axis=1
)
pose_distances.extend(distances.tolist())
if pose_distances:
metrics["mean_pose_distance"] = np.mean(pose_distances)
return metrics
RecordingSession
¶
A recording session with multiple cameras.
Attributes:
| Name | Type | Description |
|---|---|---|
camera_group |
|
|
frame_groups |
Dictionary mapping frame index to |
|
videos |
List of |
|
cameras |
List of |
|
metadata |
Dictionary of metadata. |
Methods:
| Name | Description |
|---|---|
__init__ |
Method generated by attrs for class RecordingSession. |
__repr__ |
Return a readable representation of the session. |
__setattr__ |
Method generated by attrs for class RecordingSession. |
add_video |
Add |
get_camera |
Get |
get_video |
Get |
remove_video |
Remove |
Source code in sleap_io/model/camera.py
@define(eq=False) # Set eq to false to make class hashable
class RecordingSession:
"""A recording session with multiple cameras.
Attributes:
camera_group: `CameraGroup` object containing cameras in the session.
frame_groups: Dictionary mapping frame index to `FrameGroup`.
videos: List of `Video` objects linked to `Camera`s in the session.
cameras: List of `Camera` objects linked to `Video`s in the session.
metadata: Dictionary of metadata.
"""
camera_group: CameraGroup = field(
factory=CameraGroup, validator=instance_of(CameraGroup)
)
_video_by_camera: "dict[Camera, Video]" = field(
factory=dict, validator=instance_of(dict)
)
_camera_by_video: "dict[Video, Camera]" = field(
factory=dict, validator=instance_of(dict)
)
_frame_group_by_frame_idx: "dict[int, FrameGroup]" = field(
factory=dict, validator=instance_of(dict)
)
metadata: dict = field(factory=dict, validator=instance_of(dict))
@property
def frame_groups(self) -> "dict[int, FrameGroup]":
"""Get dictionary of `FrameGroup` objects by frame index.
Returns:
Dictionary of `FrameGroup` objects by frame index.
"""
return self._frame_group_by_frame_idx
@property
def videos(self) -> list[Video]:
"""Get list of `Video` objects in the `RecordingSession`.
Returns:
List of `Video` objects in `RecordingSession`.
"""
return list(self._video_by_camera.values())
@property
def cameras(self) -> "list[Camera]":
"""Get list of `Camera` objects linked to `Video`s in the `RecordingSession`.
Returns:
List of `Camera` objects in `RecordingSession`.
"""
return list(self._video_by_camera.keys())
def get_camera(self, video: Video) -> "Camera | None":
"""Get `Camera` associated with `video`.
Args:
video: `Video` to get `Camera`
Returns:
`Camera` associated with `video` or None if not found
"""
return self._camera_by_video.get(video, None)
def get_video(self, camera: "Camera") -> Video | None:
"""Get `Video` associated with `camera`.
Args:
camera: `Camera` to get `Video`
Returns:
`Video` associated with `camera` or None if not found
"""
return self._video_by_camera.get(camera, None)
def add_video(self, video: Video, camera: "Camera"):
"""Add `video` to `RecordingSession` and mapping to `camera`.
Args:
video: `Video` object to add to `RecordingSession`.
camera: `Camera` object to associate with `video`.
Raises:
ValueError: If `camera` is not in associated `CameraGroup`.
ValueError: If `video` is not a `Video` object.
"""
# Raise ValueError if camera is not in associated camera group
self.camera_group.cameras.index(camera)
# Raise ValueError if `Video` is not a `Video` object
if not isinstance(video, Video):
raise ValueError(
f"Expected `Video` object, but received {type(video)} object."
)
# Add camera to video mapping
self._video_by_camera[camera] = video
# Add video to camera mapping
self._camera_by_video[video] = camera
def remove_video(self, video: Video):
"""Remove `video` from `RecordingSession` and mapping to `Camera`.
Args:
video: `Video` object to remove from `RecordingSession`.
Raises:
ValueError: If `video` is not in associated `RecordingSession`.
"""
# Remove video from camera mapping
camera = self._camera_by_video.pop(video)
# Remove camera from video mapping
self._video_by_camera.pop(camera)
def __repr__(self) -> str:
"""Return a readable representation of the session."""
return (
"RecordingSession("
f"camera_group={len(self.camera_group.cameras)}cameras, "
f"videos={len(self.videos)}, "
f"frame_groups={len(self.frame_groups)}"
")"
)
__annotations__ = {'camera_group': 'CameraGroup', '_video_by_camera': "'dict[Camera, Video]'", '_camera_by_video': "'dict[Video, Camera]'", '_frame_group_by_frame_idx': "'dict[int, FrameGroup]'", 'metadata': 'dict'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = True
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = 'A recording session with multiple cameras.\n\nAttributes:\n camera_group: `CameraGroup` object containing cameras in the session.\n frame_groups: Dictionary mapping frame index to `FrameGroup`.\n videos: List of `Video` objects linked to `Camera`s in the session.\n cameras: List of `Camera` objects linked to `Video`s in the session.\n metadata: Dictionary of metadata.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 130
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('camera_group', '_video_by_camera', '_camera_by_video', '_frame_group_by_frame_idx', 'metadata')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.camera'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('camera_group', '_video_by_camera', '_camera_by_video', '_frame_group_by_frame_idx', 'metadata', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ()
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
cameras
property
¶
Get list of Camera objects linked to Videos in the RecordingSession.
Returns:
| Type | Description |
|---|---|
|
List of |
frame_groups
property
¶
Get dictionary of FrameGroup objects by frame index.
Returns:
| Type | Description |
|---|---|
|
Dictionary of |
videos
property
¶
Get list of Video objects in the RecordingSession.
Returns:
| Type | Description |
|---|---|
|
List of |
__init__(camera_group=NOTHING, video_by_camera=NOTHING, camera_by_video=NOTHING, frame_group_by_frame_idx=NOTHING, metadata=NOTHING)
¶
Method generated by attrs for class RecordingSession.
Source code in sleap_io/model/camera.py
"""Data structure for a single camera view in a multi-camera setup."""
from __future__ import annotations
import attrs
import numpy as np
from attrs import define, field
from attrs.validators import instance_of
from sleap_io.model.category import Category
from sleap_io.model.identity import Identity
from sleap_io.model.instance import Instance, Instance3D
from sleap_io.model.labeled_frame import LabeledFrame
from sleap_io.model.video import Video
def rodrigues_transformation(input_matrix: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
"""Convert between rotation vector and rotation matrix using Rodrigues' formula.
This function implements the Rodrigues' rotation formula to convert between:
1. A 3D rotation vector (axis-angle representation) to a 3x3 rotation matrix
2. A 3x3 rotation matrix to a 3D rotation vector
Args:
input_matrix: A 3x3 rotation matrix or a 3x1 rotation vector.
Returns:
A tuple containing the converted matrix/vector and the Jacobian (None for now).
__repr__()
¶
Return a readable representation of the session.
__setattr__(name, val)
¶
Method generated by attrs for class RecordingSession.
add_video(video, camera)
¶
Add video to RecordingSession and mapping to camera.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
video
|
Video
|
|
required |
camera
|
Camera
|
|
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
Source code in sleap_io/model/camera.py
def add_video(self, video: Video, camera: "Camera"):
"""Add `video` to `RecordingSession` and mapping to `camera`.
Args:
video: `Video` object to add to `RecordingSession`.
camera: `Camera` object to associate with `video`.
Raises:
ValueError: If `camera` is not in associated `CameraGroup`.
ValueError: If `video` is not a `Video` object.
"""
# Raise ValueError if camera is not in associated camera group
self.camera_group.cameras.index(camera)
# Raise ValueError if `Video` is not a `Video` object
if not isinstance(video, Video):
raise ValueError(
f"Expected `Video` object, but received {type(video)} object."
)
# Add camera to video mapping
self._video_by_camera[camera] = video
# Add video to camera mapping
self._camera_by_video[video] = camera
get_camera(video)
¶
get_video(camera)
¶
remove_video(video)
¶
Remove video from RecordingSession and mapping to Camera.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
video
|
Video
|
|
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in sleap_io/model/camera.py
def remove_video(self, video: Video):
"""Remove `video` from `RecordingSession` and mapping to `Camera`.
Args:
video: `Video` object to remove from `RecordingSession`.
Raises:
ValueError: If `video` is not in associated `RecordingSession`.
"""
# Remove video from camera mapping
camera = self._camera_by_video.pop(video)
# Remove camera from video mapping
self._video_by_camera.pop(camera)
Video
¶
Video class used by sleap to represent videos and data associated with them.
This class is used to store information regarding a video and its components.
It is used to store the video's filename, shape, and the video's backend.
To create a Video object, use the from_filename method which will select the
backend appropriately.
Attributes:
| Name | Type | Description |
|---|---|---|
filename |
The filename(s) of the video. Supported extensions: "mp4", "avi", "mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif", "tiff", "bmp", "seq". If the filename is a list, a list of image filenames are expected. If filename is a folder, it will be searched for images. |
|
backend |
An object that implements the basic methods for reading and manipulating frames of a specific video type. |
|
backend_metadata |
A dictionary of metadata specific to the backend. This is useful for storing metadata that requires an open backend (e.g., shape information) without having access to the video file itself. |
|
source_video |
The source video object if this is a proxy video. This is present when the video contains an embedded subset of frames from another video. |
|
open_backend |
Whether to open the backend when the video is available. If |
|
_exists_cache |
Per-instance TTL cache for the result of |
Notes
Instances of this class are hashed by identity, not by value. This means that
two Video instances with the same attributes will NOT be considered equal in a
set or dict.
Media Video Plugin Support
For media files (mp4, avi, etc.), the following plugins are supported: - "opencv": Uses OpenCV (cv2) for video reading - "FFMPEG": Uses imageio-ffmpeg for video reading - "pyav": Uses PyAV for video reading
Plugin aliases (case-insensitive): - opencv: "opencv", "cv", "cv2", "ocv" - FFMPEG: "FFMPEG", "ffmpeg", "imageio-ffmpeg", "imageio_ffmpeg" - pyav: "pyav", "av"
Plugin selection priority: 1. Explicitly specified plugin parameter 2. Backend metadata plugin value 3. Global default (set via sio.set_default_video_plugin) 4. Auto-detection based on available packages
See Also
VideoBackend: The backend interface for reading video data. sleap_io.set_default_video_plugin: Set global default plugin. sleap_io.get_default_video_plugin: Get current default plugin.
Methods:
| Name | Description |
|---|---|
__attrs_post_init__ |
Post init syntactic sugar. |
__deepcopy__ |
Deep copy the video object. |
__getitem__ |
Return the frames of the video at the given indices. |
__init__ |
Method generated by attrs for class Video. |
__len__ |
Return the length of the video as the number of frames. |
__repr__ |
Informal string representation (for print or format). |
__str__ |
Informal string representation (for print or format). |
apply_crop |
Bake this video's virtual crop into a new physical video file. |
close |
Close the video backend. |
crop |
Return a virtual, on-read cropped view of this video. |
deduplicate_with |
Create a new video with duplicate images removed. |
exists |
Check if the video file exists and is accessible. |
frame_to_seconds |
Convert a frame index to timestamp in seconds. |
from_crop |
Open |
from_filename |
Create a Video from a filename. |
has_overlapping_images |
Check if this video has overlapping images with another video. |
matches_content |
Check if this video has the same content as another video. |
matches_path |
Check if this video has the same path as another video. |
matches_shape |
Check if this video has the same shape as another video. |
merge_with |
Merge another video's images into this one. |
open |
Open the video backend for reading. |
replace_filename |
Update the filename of the video, optionally opening the backend. |
save |
Save video frames to a new video file. |
seconds_to_frame |
Convert a timestamp in seconds to frame index. |
set_video_plugin |
Set the video plugin and reopen the video. |
to_crop_coords |
Map source-frame |
to_source_coords |
Map cropped-frame |
Source code in sleap_io/model/video.py
@attrs.define(eq=False)
class Video:
"""`Video` class used by sleap to represent videos and data associated with them.
This class is used to store information regarding a video and its components.
It is used to store the video's `filename`, `shape`, and the video's `backend`.
To create a `Video` object, use the `from_filename` method which will select the
backend appropriately.
Attributes:
filename: The filename(s) of the video. Supported extensions: "mp4", "avi",
"mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",
"tiff", "bmp", "seq". If the filename is a list, a list of image filenames
are expected. If filename is a folder, it will be searched for images.
backend: An object that implements the basic methods for reading and
manipulating frames of a specific video type.
backend_metadata: A dictionary of metadata specific to the backend. This is
useful for storing metadata that requires an open backend (e.g., shape
information) without having access to the video file itself.
source_video: The source video object if this is a proxy video. This is present
when the video contains an embedded subset of frames from another video.
open_backend: Whether to open the backend when the video is available. If `True`
(the default), the backend will be automatically opened if the video exists.
Set this to `False` when you want to manually open the backend, or when the
you know the video file does not exist and you want to avoid trying to open
the file.
_exists_cache: Per-instance TTL cache for the result of `exists()` when the
`filename` is a remote URL. Keyed by `(filename, dataset)` and storing
`(exists_bool, monotonic_timestamp)`. This avoids issuing a network probe
on every call (e.g. from the `is_open` property, which GUIs poll on each
render). The TTL defaults to 60 seconds and can be overridden via the
`SLEAP_IO_EXISTS_TTL` environment variable. The cache is cleared on
`replace_filename`.
Notes:
Instances of this class are hashed by identity, not by value. This means that
two `Video` instances with the same attributes will NOT be considered equal in a
set or dict.
Media Video Plugin Support:
For media files (mp4, avi, etc.), the following plugins are supported:
- "opencv": Uses OpenCV (cv2) for video reading
- "FFMPEG": Uses imageio-ffmpeg for video reading
- "pyav": Uses PyAV for video reading
Plugin aliases (case-insensitive):
- opencv: "opencv", "cv", "cv2", "ocv"
- FFMPEG: "FFMPEG", "ffmpeg", "imageio-ffmpeg", "imageio_ffmpeg"
- pyav: "pyav", "av"
Plugin selection priority:
1. Explicitly specified plugin parameter
2. Backend metadata plugin value
3. Global default (set via sio.set_default_video_plugin)
4. Auto-detection based on available packages
See Also:
VideoBackend: The backend interface for reading video data.
sleap_io.set_default_video_plugin: Set global default plugin.
sleap_io.get_default_video_plugin: Get current default plugin.
"""
filename: str | list[str]
backend: VideoBackend | None = None
backend_metadata: dict[str, any] = attrs.field(factory=dict)
source_video: "Video | None" = None
open_backend: bool = True
_exists_cache: dict[tuple[str, str | None], tuple[bool, float]] = attrs.field(
init=False, factory=dict, repr=False, eq=False
)
# URL auth context, threaded in by `make_video` for remote loads. Persisted
# on the Video (not just the backend) so existence probes and a later
# `open()` reconstruction stay authenticated after the backend is closed.
_url_headers: dict[str, str] | None = attrs.field(
init=False, default=None, repr=False, eq=False
)
_url_stream_mode: str = attrs.field(
init=False, default="blockcache", repr=False, eq=False
)
EXTS = MediaVideo.EXTS + HDF5Video.EXTS + ImageVideo.EXTS + ("seq",)
def _backend_url_headers(self) -> dict[str, str] | None:
"""Return the HTTP headers to authenticate remote existence probes.
Prefers the URL auth context stored on this `Video` (set by `make_video`
at load time); falls back to the live backend's headers when present.
Returns `None` for local files and unauthenticated URLs.
"""
if self._url_headers is not None:
return self._url_headers
if isinstance(self.backend, HDF5Video):
return getattr(self.backend, "_url_headers", None)
return None
@property
def original_video(self) -> "Video | None":
"""The root video in the provenance chain.
For embedded videos, this returns the ultimate source video by
traversing the source_video chain. Returns None if this video
has no source_video (i.e., it IS an original).
This property is computed by following the source_video chain to find
the root. For a single-level embedding (A embeds from B), original_video
returns B. For multi-level embedding (A <- B <- C), it returns C.
"""
if self.source_video is None:
return None # This IS the original
# Traverse to root
v = self.source_video
while v.source_video is not None:
v = v.source_video
return v
def __attrs_post_init__(self):
"""Post init syntactic sugar."""
if self.open_backend and self.backend is None and self.exists():
try:
self.open()
except Exception:
# If we can't open the backend, just ignore it for now so we don't
# prevent the user from building the Video object entirely.
pass
def __deepcopy__(self, memo):
"""Deep copy the video object."""
if id(self) in memo:
return memo[id(self)]
reopen = False
if self.is_open:
reopen = True
self.close()
new_video = Video(
filename=self.filename,
backend=None,
backend_metadata=self.backend_metadata.copy(),
source_video=self.source_video,
open_backend=self.open_backend,
)
memo[id(self)] = new_video
if reopen:
self.open()
return new_video
@classmethod
def from_filename(
cls,
filename: str | list[str],
dataset: str | None = None,
grayscale: bool | None = None,
keep_open: bool = True,
source_video: "Video | None" = None,
**kwargs,
) -> VideoBackend:
"""Create a Video from a filename.
Args:
filename: The filename(s) of the video. Supported extensions: "mp4", "avi",
"mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",
"tiff", "bmp". If the filename is a list, a list of image filenames are
expected. If filename is a folder, it will be searched for images.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
source_video: The source video object if this is a proxy video. This is
present when the video contains an embedded subset of frames from
another video.
**kwargs: Additional backend-specific arguments passed to
VideoBackend.from_filename. See VideoBackend.from_filename for supported
arguments.
Returns:
Video instance with the appropriate backend instantiated.
"""
backend = VideoBackend.from_filename(
filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
**kwargs,
)
# If filename is a directory, VideoBackend.from_filename will expand it
# to a list of paths to images contained within the directory. In this
# case we want to use the expanded list as filename
return cls(
filename=backend.filename,
backend=backend,
source_video=source_video,
)
def crop(
self,
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
) -> "Video":
"""Return a virtual, on-read cropped view of this video.
Exactly one region spec must be given: ``crop`` (explicit
``(x1, y1, x2, y2)`` rect), ``bbox``, ``roi`` (its axis-aligned bounds +
``margin``), or (``center``, ``size``) for a fixed-size centered/
centroid-following window. The returned ``Video`` shares no pixels with
this one; frames are decoded on read and cropped (byte-identical to
:func:`sleap_io.transform.frame.crop_frame`). Out-of-bounds regions are
pad-filled with ``fill`` (never clamped), so the output shape is always
exactly ``(y2 - y1, x2 - x1)``.
The crop composes (FLATTENS when fills agree and the region is in-bounds)
with any existing crop on this video via
:meth:`CropVideoBackend.wrap`. ``source_video`` is set to this video for
provenance. When ``share_decode`` (the default), the new crop reuses this
video's backend instance as the shared inner so a mosaic of tiles over
one file decodes each source frame once; in that case the new tile does
NOT own the shared decoder (this video does).
Args:
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2``
exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: Any object exposing axis-aligned ``.bounds`` as
``(minx, miny, maxx, maxy)`` (e.g. a shapely geometry).
center: Window center ``(cx, cy)`` (used with ``size``).
size: Fixed output ``(width, height)`` (used with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (the default), reuse this video's backend
as the shared inner so tiles decode each frame once; the new tile
does not own the shared decoder.
Returns:
A new ``Video`` exposing the cropped view.
"""
from sleap_io.io.video_reading import CropVideoBackend
rect = _resolve_crop_rect(crop, bbox, roi, center, size, margin)
if self.backend is None and self.open_backend:
self.open()
if self.backend is None:
raise ValueError(
"Cannot crop a video with no open backend. Open it first (set "
"open_backend=True or call .open()) before cropping."
)
inner = self.backend
cropped_backend = CropVideoBackend.wrap(
inner=inner, crop=rect, fill=fill, owns_inner=not share_decode
)
cropped = Video(
filename=self.filename,
backend=cropped_backend,
source_video=self,
open_backend=self.open_backend,
)
x1, y1, x2, y2 = cropped_backend.crop
src_shape = self.shape
cropped.backend_metadata = {
**self.backend_metadata,
"shape": (src_shape[0], y2 - y1, x2 - x1, src_shape[3])
if src_shape is not None
else None,
# The uncropped source shape, so a closed re-serialize keeps videos_json
# describing the full frame even without a live source_video (D-120/DI-2).
"source_shape": list(src_shape) if src_shape is not None else None,
# COMPOSED source rect from wrap (D-120): keeps open/closed crop keys
# identical and root-canonical, and survives close()->open().
"crop": list(cropped_backend.crop),
"crop_fill": cropped_backend.fill,
}
return cropped
@classmethod
def from_crop(
cls,
video: "str | Path | Video",
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
**kwargs,
) -> "Video":
"""Open ``video`` (path or ``Video``) and return a virtual crop.
Accepts the same region specs as :meth:`crop` (``crop``/``bbox``/``roi``/
``center``+``size``); extra keyword arguments are forwarded to
:meth:`from_filename` when ``video`` is a path (ignored when it is already
a ``Video``).
Args:
video: A path/filename to open, or an existing ``Video`` to crop.
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2`` exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: An object exposing axis-aligned ``.bounds`` (e.g. a shapely
geometry); ``margin`` is applied around it.
center: Window center ``(cx, cy)`` (with ``size``).
size: Fixed output ``(width, height)`` (with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (default), reuse the source decoder.
**kwargs: Forwarded to :meth:`from_filename` for a path input.
Returns:
A new ``Video`` exposing the cropped view.
"""
if isinstance(video, (str, Path)):
video = cls.from_filename(video, **kwargs)
return video.crop(
crop,
bbox=bbox,
roi=roi,
center=center,
size=size,
margin=margin,
fill=fill,
share_decode=share_decode,
)
def _crop_tuple(self) -> tuple[int, int, int, int] | None:
"""Return this video's crop rect ``(x1, y1, x2, y2)`` or ``None``.
Reads ``backend.crop`` when the backend is a ``CropVideoBackend`` (open
path), else ``backend_metadata["crop"]`` (closed path), else ``None``
(uncropped).
"""
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
return tuple(self.backend.crop)
crop = self.backend_metadata.get("crop")
return tuple(crop) if crop is not None else None
def _crop_fill(self) -> int | tuple[int, ...]:
"""Return this video's crop fill value (open: backend; closed: metadata).
Returns ``0`` for an uncropped video. Mirrors :meth:`_crop_tuple`.
"""
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
return self.backend.fill
return self.backend_metadata.get("crop_fill", 0)
@property
def is_cropped(self) -> bool:
"""Whether this video is a virtual crop of another video."""
return self._crop_tuple() is not None
@property
def crop_rect(self) -> tuple[int, int, int, int] | None:
"""Crop rect ``(x1, y1, x2, y2)`` in source coords, or ``None`` if uncropped."""
return self._crop_tuple()
@property
def crop_fill(self) -> int | tuple[int, ...]:
"""The out-of-bounds fill value for this video's crop (``0`` if uncropped)."""
return self._crop_fill()
def to_crop_coords(self, points: np.ndarray) -> np.ndarray:
"""Map source-frame ``(x, y)`` into this video's cropped frame.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated into the cropped frame. If this video is not
cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else crop_points(points, crop)
def to_source_coords(self, points: np.ndarray) -> np.ndarray:
"""Map cropped-frame ``(x, y)`` back to source-frame coordinates.
Inverse of :meth:`to_crop_coords`.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated back to source coordinates. If this video is
not cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else uncrop_points(points, crop)
@property
def shape(self) -> tuple[int, int, int, int] | None:
"""Return the shape of the video as (num_frames, height, width, channels).
If the video backend is not set or it cannot determine the shape of the video,
this will return None.
"""
return self._get_shape()
def _get_shape(self) -> tuple[int, int, int, int] | None:
"""Return the shape of the video as (num_frames, height, width, channels).
This suppresses errors related to querying the backend for the video shape, such
as when it has not been set or when the video file is not found.
"""
try:
return self.backend.shape
except Exception:
if "shape" in self.backend_metadata:
return self.backend_metadata["shape"]
return None
@property
def grayscale(self) -> bool | None:
"""Return whether the video is grayscale.
If the video backend is not set or it cannot determine whether the video is
grayscale, this will return None.
"""
shape = self.shape
if shape is not None:
return shape[-1] == 1
else:
grayscale = None
if "grayscale" in self.backend_metadata:
grayscale = self.backend_metadata["grayscale"]
return grayscale
@grayscale.setter
def grayscale(self, value: bool):
"""Set the grayscale value and adjust the backend."""
if self.backend is not None:
self.backend.grayscale = value
self.backend._cached_shape = None
self.backend_metadata["grayscale"] = value
@property
def fps(self) -> float | None:
"""Return the frames per second of the video.
For MediaVideo backends, this reads FPS from the video container metadata.
For other backends (ImageVideo, HDF5Video, TiffVideo), this returns the
explicitly set value or None if not set.
Returns:
The FPS if known, or None if unavailable/unknown.
"""
if self.backend is not None:
return self.backend.fps
return self.backend_metadata.get("fps")
@fps.setter
def fps(self, value: float | None):
"""Set the frames per second.
Args:
value: Frames per second. Must be positive if not None.
Raises:
ValueError: If value is not positive.
Notes:
For MediaVideo backends, setting FPS overrides the value from container
metadata. For other backends, this sets the FPS directly.
"""
if value is not None and value <= 0:
raise ValueError(f"FPS must be positive, got {value}")
if self.backend is not None:
self.backend.fps = value
self.backend_metadata["fps"] = value
def frame_to_seconds(self, frame_idx: int) -> float | None:
"""Convert a frame index to timestamp in seconds.
Args:
frame_idx: Zero-indexed frame number.
Returns:
Time in seconds, or None if FPS is unknown.
Notes:
This assumes constant frame rate. For variable frame rate videos,
the returned timestamp may be approximate.
"""
if self.fps is None or self.fps <= 0:
return None
return frame_idx / self.fps
def seconds_to_frame(self, seconds: float) -> int | None:
"""Convert a timestamp in seconds to frame index.
Args:
seconds: Time in seconds from video start.
Returns:
Zero-indexed frame number (rounded down), or None if FPS unknown.
"""
if self.fps is None or self.fps <= 0:
return None
return int(seconds * self.fps)
def __len__(self) -> int:
"""Return the length of the video as the number of frames."""
shape = self.shape
return 0 if shape is None else shape[0]
def __repr__(self) -> str:
"""Informal string representation (for print or format)."""
dataset = (
f"dataset={self.backend.dataset}, "
if getattr(self.backend, "dataset", "")
else ""
)
return (
"Video("
f'filename="{self.filename}", '
f"shape={self.shape}, "
f"{dataset}"
f"backend={type(self.backend).__name__}"
")"
)
def __str__(self) -> str:
"""Informal string representation (for print or format)."""
return self.__repr__()
def __getitem__(self, inds: int | list[int] | slice) -> np.ndarray:
"""Return the frames of the video at the given indices.
Args:
inds: Index or list of indices of frames to read.
Returns:
Frame or frames as a numpy array of shape `(height, width, channels)` if a
scalar index is provided, or `(frames, height, width, channels)` if a list
of indices is provided.
See also: VideoBackend.get_frame, VideoBackend.get_frames
"""
if not self.is_open:
if self.open_backend:
self.open()
else:
raise ValueError(
"Video backend is not open. Call video.open() or set "
"video.open_backend to True to do automatically on frame read."
)
return self.backend[inds]
def exists(self, check_all: bool = False, dataset: str | None = None) -> bool:
"""Check if the video file exists and is accessible.
Args:
check_all: If `True`, check that all filenames in a list exist. If `False`
(the default), check that the first filename exists.
dataset: Name of dataset in HDF5 file. If specified, this will function will
return `False` if the dataset does not exist.
Returns:
`True` if the file exists and is accessible, `False` otherwise.
"""
if isinstance(self.filename, list):
if check_all:
for f in self.filename:
if not is_file_accessible(f):
return False
return True
else:
return is_file_accessible(self.filename[0])
# URL fast path: must run BEFORE `is_file_accessible`, which treats the
# filename as a local path and would spuriously return False for a URL.
from sleap_io.io._remote import _is_url
if _is_url(self.filename):
return self._url_exists(dataset)
file_is_accessible = is_file_accessible(self.filename)
if not file_is_accessible:
# Check if it's a directory (ImageVideo source)
if Path(self.filename).is_dir():
return True
return False
if dataset is None or dataset == "":
dataset = self.backend_metadata.get("dataset", None)
if dataset is not None and dataset != "":
has_dataset = False
if (
self.backend is not None
and type(self.backend) is HDF5Video
and self.backend._open_reader is not None
):
has_dataset = dataset in self.backend._open_reader
else:
with h5py.File(self.filename, "r") as f:
has_dataset = dataset in f
return has_dataset
return True
def _url_exists(self, dataset: str | None) -> bool:
"""Check whether a remote URL `filename` exists, with a TTL cache.
Args:
dataset: Name of dataset in the (remote) HDF5 file. If specified (or
derivable from `backend_metadata`), existence additionally requires
that the dataset be present in the file.
Returns:
`True` if the URL is reachable (and, if a dataset was requested, the
dataset exists), `False` otherwise.
Notes:
Results are cached per instance keyed by `(filename, dataset)` for a
TTL (default 60s, overridable via the `SLEAP_IO_EXISTS_TTL` env var) so
repeated calls (e.g. from the `is_open` property in a GUI render loop)
do not issue a network probe each time.
"""
from sleap_io.io._remote import _head_or_range_probe
key = (self.filename, dataset)
try:
ttl = float(os.environ.get("SLEAP_IO_EXISTS_TTL", "60"))
except ValueError:
# A malformed env value must not break the never-raise bool
# contract of exists()/is_open; fall back to the 60s default.
ttl = 60.0
cached = self._exists_cache.get(key)
if cached is not None and (time.monotonic() - cached[1]) < ttl:
return cached[0]
try:
if not _head_or_range_probe(
self.filename, headers=self._backend_url_headers()
):
result = False
else:
if dataset is None or dataset == "":
dataset = self.backend_metadata.get("dataset", None)
if dataset is None or dataset == "":
result = True
else:
result = self._url_dataset_exists(dataset)
except Exception:
result = False
self._exists_cache[key] = (result, time.monotonic())
return result
def _url_dataset_exists(self, dataset: str) -> bool:
"""Check whether `dataset` is present in the remote HDF5 file.
Reuses the backend's already-open HDF5 reader when available; otherwise
opens the remote file via fsspec for a single membership check.
Args:
dataset: Name of dataset in the remote HDF5 file.
Returns:
`True` if the dataset is present, `False` otherwise.
"""
if (
self.backend is not None
and type(self.backend) is HDF5Video
and self.backend._open_reader is not None
):
return dataset in self.backend._open_reader
from sleap_io.io._remote import open_remote_h5
url_file = open_remote_h5(self.filename, headers=self._backend_url_headers())
try:
with h5py.File(url_file, "r") as f:
return dataset in f
finally:
url_file.close()
@property
def is_open(self) -> bool:
"""Check if the video backend is open."""
return self.exists() and self.backend is not None
def open(
self,
filename: str | None = None,
dataset: str | None = None,
grayscale: str | None = None,
keep_open: bool = True,
plugin: str | None = None,
):
"""Open the video backend for reading.
Args:
filename: Filename to open. If not specified, will use the filename set on
the video object.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
plugin: Video plugin to use for MediaVideo files. One of "opencv",
"FFMPEG", or "pyav". Also accepts aliases (case-insensitive).
If not specified, uses the backend metadata, global default,
or auto-detection in that order.
Notes:
This is useful for opening the video backend to read frames and then closing
it after reading all the necessary frames.
If the backend was already open, it will be closed before opening a new one.
Values for the HDF5 dataset and grayscale will be remembered if not
specified.
"""
if filename is not None:
self.replace_filename(filename, open=False)
# Try to remember values from previous backend if available and not specified.
if self.backend is not None:
if dataset is None:
dataset = getattr(self.backend, "dataset", None)
if grayscale is None:
grayscale = getattr(self.backend, "grayscale", None)
else:
if dataset is None and "dataset" in self.backend_metadata:
dataset = self.backend_metadata["dataset"]
if grayscale is None:
if "grayscale" in self.backend_metadata:
grayscale = self.backend_metadata["grayscale"]
elif "shape" in self.backend_metadata:
grayscale = self.backend_metadata["shape"][-1] == 1
if not self.exists(dataset=dataset):
from sleap_io.io._remote import _is_url, _redact_url
# Redact credential-bearing URLs (e.g. presigned ``?token=`` links)
# so they never surface in tracebacks/logs. Local paths are shown
# verbatim.
name = (
_redact_url(self.filename)
if isinstance(self.filename, str) and _is_url(self.filename)
else self.filename
)
msg = f"Video does not exist or cannot be opened for reading: {name}"
if dataset is not None:
msg += f" (dataset: {dataset})"
raise FileNotFoundError(msg)
# Close previous backend if open.
self.close()
# Handle plugin parameter
backend_kwargs = {}
if plugin is not None:
from sleap_io.io.video_reading import normalize_plugin_name
plugin = normalize_plugin_name(plugin)
self.backend_metadata["plugin"] = plugin
if "plugin" in self.backend_metadata:
backend_kwargs["plugin"] = self.backend_metadata["plugin"]
# Create new backend. Forward the URL auth context so a reopened remote
# HDF5Video stays authenticated (the previous backend, and its headers,
# were dropped by self.close() above).
self.backend = VideoBackend.from_filename(
self.filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
url_headers=self._url_headers,
url_stream_mode=self._url_stream_mode,
**backend_kwargs,
)
# Re-wrap as a crop view if this video records a crop in its metadata.
# The rebuilt backend above is always a plain backend, so this wraps
# exactly once (idempotent across close()->open() and deepcopy).
if "crop" in self.backend_metadata:
from sleap_io.io.video_reading import CropVideoBackend
self.backend = CropVideoBackend.wrap(
inner=self.backend,
crop=tuple(self.backend_metadata["crop"]),
fill=self.backend_metadata.get("crop_fill", 0),
)
def close(self):
"""Close the video backend."""
if self.backend is not None:
# Try to remember values from previous backend if available and not
# specified.
try:
self.backend_metadata["dataset"] = getattr(
self.backend, "dataset", None
)
self.backend_metadata["grayscale"] = getattr(
self.backend, "grayscale", None
)
self.backend_metadata["shape"] = getattr(self.backend, "shape", None)
self.backend_metadata["fps"] = getattr(self.backend, "fps", None)
# Persist the crop so a Video cropped in-memory (never loaded
# from disk) survives a close()->open() and deepcopy: open()
# re-wraps from these keys (the closed-path shape above is
# already the cropped shape).
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
self.backend_metadata["crop"] = list(self.backend.crop)
self.backend_metadata["crop_fill"] = self.backend.fill
except Exception:
pass
# Deterministically release the backend's open handles (the cached
# reader and, for a remote HDF5Video, the fsspec URL file-like)
# rather than relying on garbage collection.
try:
self.backend.close()
except Exception:
pass
del self.backend
self.backend = None
def replace_filename(
self, new_filename: str | Path | list[str] | list[Path], open: bool = True
):
"""Update the filename of the video, optionally opening the backend.
Args:
new_filename: New filename to set for the video.
open: If `True` (the default), open the backend with the new filename. If
the new filename does not exist, no error is raised.
"""
if isinstance(new_filename, Path):
new_filename = new_filename.as_posix()
if isinstance(new_filename, list):
new_filename = [
p.as_posix() if isinstance(p, Path) else p for p in new_filename
]
# A relink to a different file makes the recorded shape/grayscale/fps in
# ``backend_metadata`` stale: they describe the OLD file but the new file
# may have a different resolution/channels/frame rate. They must not be
# serialized under the new filename (regression from #483, where
# ``save_slp(prefer_metadata=True)`` prefers these recorded values), so
# invalidate them on a real relink and let them be recomputed from the new
# backend. The no-relink path leaves metadata untouched so golden
# byte-identical saves stay byte-identical.
filename_changed = new_filename != self.filename
self.filename = new_filename
self.backend_metadata["filename"] = new_filename
# Invalidate any cached URL existence results for the previous filename.
self._exists_cache.clear()
if open:
if self.exists():
self.open()
else:
self.close()
# Drop stale metadata AFTER (re)opening: ``open()`` internally calls
# ``close()``, which would otherwise re-stamp the OLD backend's
# shape/grayscale/fps back into ``backend_metadata``.
if filename_changed:
for key in ("shape", "grayscale", "fps"):
self.backend_metadata.pop(key, None)
def matches_path(self, other: "Video", strict: bool = False) -> bool:
"""Check if this video has the same path as another video.
Args:
other: Another video to compare with.
strict: If True, require exact path match. If False, consider videos
with the same filename (basename) as matching.
Returns:
True if the videos have matching paths, False otherwise.
Notes:
For HDF5 video backends (e.g., embedded videos in .pkg.slp files),
matching prioritizes the source_filename attribute since multiple
videos can share the same HDF5 file path but reference different
source videos. Falls back to dataset name matching if source_filename
is not available.
"""
# Handle HDF5 backends specially - prioritize source_filename matching
self_is_hdf5 = isinstance(self.backend, HDF5Video)
other_is_hdf5 = isinstance(other.backend, HDF5Video)
if self_is_hdf5 and other_is_hdf5:
# Both are HDF5 videos - must match by BOTH source_filename AND dataset
# to distinguish different videos embedded in the same pkg.slp file
self_source = self.backend.source_filename
other_source = other.backend.source_filename
self_dataset = self.backend.dataset
other_dataset = other.backend.dataset
# If both have datasets, they must match
if self_dataset is not None and other_dataset is not None:
if self_dataset != other_dataset:
return False # Different datasets = different videos
# If both have source_filenames, compare them
if self_source is not None and other_source is not None:
if strict:
# For HDF5 videos, just compare normalized path strings
# (avoid slow resolve() on network paths)
return Path(self_source).as_posix() == Path(other_source).as_posix()
else:
return Path(self_source).name == Path(other_source).name
# If only datasets available (no source_filename), they must match
if self_dataset is not None and other_dataset is not None:
return self_dataset == other_dataset
# If neither source_filename nor dataset available, cannot match
return False
if isinstance(self.filename, list) and isinstance(other.filename, list):
# Both are image sequences
if strict:
return self.filename == other.filename
else:
# Compare basenames
self_basenames = [Path(f).name for f in self.filename]
other_basenames = [Path(f).name for f in other.filename]
return self_basenames == other_basenames
elif isinstance(self.filename, list) or isinstance(other.filename, list):
# One is image sequence, other is single file
return False
else:
# Both are single files - use resolve() for symlink handling
if strict:
p1, p2 = Path(self.filename), Path(other.filename)
# Fast string comparison first
if p1.as_posix() == p2.as_posix():
return True
# Only resolve if both exist locally (avoid slow network timeouts)
try:
if p1.exists() and p2.exists():
return p1.resolve() == p2.resolve()
except OSError:
pass
return False
else:
return Path(self.filename).name == Path(other.filename).name
def matches_content(self, other: "Video") -> bool:
"""Check if this video has the same content as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same shape and backend type.
Notes:
This compares metadata like shape and backend type, not actual frame data.
"""
# Compare shapes
self_shape = self.shape
other_shape = other.shape
if self_shape != other_shape:
return False
# Compare backend types
if self.backend is None and other.backend is None:
return True
elif self.backend is None or other.backend is None:
return False
return type(self.backend).__name__ == type(other.backend).__name__
def matches_shape(self, other: "Video") -> bool:
"""Check if this video has the same shape as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same height, width, and channels.
Notes:
This only compares spatial dimensions, not the number of frames.
"""
# Try to get shape from backend metadata first if shape is not available
if self.backend is None and "shape" in self.backend_metadata:
self_shape = self.backend_metadata["shape"]
else:
self_shape = self.shape
if other.backend is None and "shape" in other.backend_metadata:
other_shape = other.backend_metadata["shape"]
else:
other_shape = other.shape
# Handle None shapes
if self_shape is None or other_shape is None:
return False
# Compare only height, width, channels (not frames)
return self_shape[1:] == other_shape[1:]
def has_overlapping_images(self, other: "Video") -> bool:
"""Check if this video has overlapping images with another video.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to compare with.
Returns:
True if both are ImageVideo instances with overlapping image files.
False if either video is not an ImageVideo or no overlap exists.
Notes:
Only works with ImageVideo backends where filename is a list.
Compares individual image filenames (basenames only).
"""
# Both must be image sequences
if not (isinstance(self.filename, list) and isinstance(other.filename, list)):
return False
# Get basenames for comparison
self_basenames = set(Path(f).name for f in self.filename)
other_basenames = set(Path(f).name for f in other.filename)
# Check if there's any overlap
return len(self_basenames & other_basenames) > 0
def deduplicate_with(self, other: "Video") -> "Video":
"""Create a new video with duplicate images removed.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to deduplicate against. Must also be ImageVideo.
Returns:
A new Video object with duplicate images removed from this video,
or None if all images were duplicates.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
Images are considered duplicates if they have the same basename.
The returned video contains only images from this video that are
not present in the other video.
"""
if not isinstance(self.filename, list):
raise ValueError("deduplicate_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get basenames from other video
other_basenames = set(Path(f).name for f in other.filename)
# Keep only non-duplicate images
deduplicated_paths = [
f for f in self.filename if Path(f).name not in other_basenames
]
if not deduplicated_paths:
# All images were duplicates
return None
# Create new video with deduplicated images
return Video.from_filename(deduplicated_paths, grayscale=self.grayscale)
def merge_with(self, other: "Video") -> "Video":
"""Merge another video's images into this one.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to merge with. Must also be ImageVideo.
Returns:
A new Video object with unique images from both videos.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
The merged video contains all unique images from both videos,
with automatic deduplication based on image basename.
"""
if not isinstance(self.filename, list):
raise ValueError("merge_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get all unique images (by basename) preserving order
seen_basenames = set()
merged_paths = []
for path in self.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
for path in other.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
# Create new video with merged images
return Video.from_filename(merged_paths, grayscale=self.grayscale)
def save(
self,
save_path: str | Path,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Save video frames to a new video file.
Args:
save_path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to save. Can be specified as a list or array of
frame integers. If not specified, saves all video frames.
fps: Frames per second for the output video. If not specified, uses the
source video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
`sio.save_video` for video compression.
Returns:
A new `Video` object pointing to the new video file.
"""
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
frame_inds = np.arange(len(self)) if frame_inds is None else frame_inds
# Use source video FPS if not explicitly specified
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(save_path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
new_video = Video.from_filename(save_path, grayscale=self.grayscale)
return new_video
def apply_crop(
self,
path: str | Path,
*,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Bake this video's virtual crop into a new physical video file.
Materializes the cropped frames (``self[i]``, already cropped by the
virtual :class:`~sleap_io.io.video_reading.CropVideoBackend`) to ``path``
via :class:`~sleap_io.io.video_writing.VideoWriter`. The crop becomes
physical: the returned video has no ``CropVideoBackend`` / ``/video_crops``
entry. ``baked.shape`` equals this video's cropped shape when the cropped
width and height are multiples of 16; otherwise the H.264 encoder pads the
bottom/right edges up to the next multiple of 16 (the macro-block size),
so ``baked.shape`` may exceed the cropped shape on those edges. The
top-left content is preserved, so coordinates stay aligned regardless.
This operation is coordinate-neutral. A virtual crop already presents
cropped-frame coordinates, so baking the cropped pixels does not change
any point coordinates (unlike ``sio transform --crop``, which applies a
new crop and adjusts coordinates).
Provenance is preserved: the returned video's ``source_video`` is the
uncropped original — ``self.source_video`` (the parent a virtual crop is
created against), or, for a manually-built crop with no parent, an
uncropped view reconstructed from the crop backend's inner. So
``baked.source_video.shape`` is the uncropped shape while ``baked.shape``
is the cropped shape, and ``baked.grayscale`` is carried from this video.
Args:
path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to bake. Can be specified as a list or array
of frame integers. If not specified, bakes all video frames.
fps: Frames per second for the output video. If not specified, uses
this video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
``sio.save_video`` for video compression.
Returns:
A new ``Video`` pointing to the baked file, with ``source_video`` set
to the uncropped original (or this video) and ``grayscale`` carried
from this video.
Raises:
ValueError: If this video has no virtual crop to apply (i.e.,
:meth:`_crop_tuple` returns ``None``). Use :meth:`save` to
re-encode an uncropped video.
"""
if self._crop_tuple() is None:
raise ValueError(
"apply_crop requires a cropped video (a virtual crop created via "
"Video.crop / Video.from_crop), but this video has no crop to "
"apply. Use Video.save to re-encode an uncropped video."
)
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
if frame_inds is None:
# A crop over a SPARSELY embedded video (frame_map keys are not the dense
# range 0..N-1, e.g. {5, 9}) cannot be baked by default: writing the frames
# compacts them to 0..k-1, so any labeled frame referencing a source index
# (5, 9) would dangle. Refuse with a clear error rather than crash or
# silently misalign. An explicit frame_inds bypasses this for advanced use.
inner = getattr(self.backend, "inner", None)
frame_map = getattr(inner, "frame_map", None)
if frame_map:
keys = sorted(frame_map.keys())
if keys != list(range(len(keys))):
raise ValueError(
"Cannot bake a virtual crop over a video with sparsely "
f"embedded frames (frame_map keys {keys}): baking would "
"compact frames to a contiguous range and break frame_idx "
"references. Pass explicit frame_inds to override, or "
"materialize from the original source video."
)
frame_inds = np.arange(len(self))
# Use this video's FPS if not explicitly specified.
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
baked = Video.from_filename(path, grayscale=self.grayscale)
# Provenance: the uncropped original. Walk past any still-virtual crop
# ancestors (a flattened crop-of-crop's source_video may itself be a crop)
# to the first uncropped ancestor. For a manually-built crop with no parent,
# reconstruct an uncropped view from the crop backend's inner, so
# source_video is never a cropped video.
source = self.source_video
while source is not None and source._crop_tuple() is not None:
source = source.source_video
if source is None:
inner = getattr(self.backend, "inner", None)
source = (
Video(filename=inner.filename, backend=inner)
if inner is not None
else self
)
baked.source_video = source
return baked
def set_video_plugin(self, plugin: str) -> None:
"""Set the video plugin and reopen the video.
Args:
plugin: Video plugin to use. One of "opencv", "FFMPEG", or "pyav".
Also accepts aliases (case-insensitive).
Raises:
ValueError: If the video is not a MediaVideo type.
Examples:
>>> video.set_video_plugin("opencv")
>>> video.set_video_plugin("CV2") # Same as "opencv"
"""
from sleap_io.io.video_reading import MediaVideo, normalize_plugin_name
if not self.filename.endswith(MediaVideo.EXTS):
raise ValueError(f"Cannot set plugin for non-media video: {self.filename}")
plugin = normalize_plugin_name(plugin)
# Close current backend if open
was_open = self.is_open
if was_open:
self.close()
# Update backend metadata
self.backend_metadata["plugin"] = plugin
# Reopen with new plugin if it was open
if was_open:
self.open()
EXTS = ('mp4', 'avi', 'mov', 'mj2', 'mkv', 'h5', 'hdf5', 'slp', 'png', 'jpg', 'jpeg', 'tif', 'tiff', 'bmp', 'seq')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__annotations__ = {'filename': 'str | list[str]', 'backend': 'VideoBackend | None', 'backend_metadata': 'dict[str, any]', 'source_video': "'Video | None'", 'open_backend': 'bool', '_exists_cache': 'dict[tuple[str, str | None], tuple[bool, float]]', '_url_headers': 'dict[str, str] | None', '_url_stream_mode': 'str'}
class-attribute
¶
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)
__attrs_own_setattr__ = False
class-attribute
¶
Returns True when the argument is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
__attrs_props__ = ClassProps(is_exception=False, is_slotted=True, has_weakref_slot=True, is_frozen=False, kw_only=<KeywordOnly.NO: 'no'>, collected_fields_by_mro=True, added_init=True, added_repr=False, added_eq=False, added_ordering=False, hashability=<Hashability.LEAVE_ALONE: 'leave_alone'>, added_match_args=True, added_str=False, added_pickling=True, on_setattr_hook=<function pipe.<locals>.wrapped_pipe at 0x7f08471ce840>, field_transformer=None)
class-attribute
¶
Effective class properties as derived from parameters to attr.s() or
define() decorators.
This is the same data structure that attrs uses internally to decide how to construct the final class.
Warning:
This feature is currently **experimental** and is not covered by our
strict backwards-compatibility guarantees.
Attributes:
| Name | Type | Description |
|---|---|---|
is_exception |
bool
|
Whether the class is treated as an exception class. |
is_slotted |
bool
|
Whether the class is |
has_weakref_slot |
bool
|
Whether the class has a slot for weak references. |
is_frozen |
bool
|
Whether the class is frozen. |
kw_only |
KeywordOnly
|
Whether / how the class enforces keyword-only arguments on the
|
collected_fields_by_mro |
bool
|
Whether the class fields were collected by method resolution order.
That is, correctly but unlike |
added_init |
bool
|
Whether the class has an attrs-generated |
added_repr |
bool
|
Whether the class has an attrs-generated |
added_eq |
bool
|
Whether the class has attrs-generated equality methods. |
added_ordering |
bool
|
Whether the class has attrs-generated ordering methods. |
hashability |
Hashability
|
How |
added_match_args |
bool
|
Whether the class supports positional |
added_str |
bool
|
Whether the class has an attrs-generated |
added_pickling |
bool
|
Whether the class has attrs-generated |
on_setattr_hook |
Callable[[Any, Attribute[Any], Any], Any] | None
|
The class's |
field_transformer |
Callable[[Attribute[Any]], Attribute[Any]] | None
|
The class's |
.. versionadded:: 25.4.0
__doc__ = '`Video` class used by sleap to represent videos and data associated with them.\n\nThis class is used to store information regarding a video and its components.\nIt is used to store the video\'s `filename`, `shape`, and the video\'s `backend`.\n\nTo create a `Video` object, use the `from_filename` method which will select the\nbackend appropriately.\n\nAttributes:\n filename: The filename(s) of the video. Supported extensions: "mp4", "avi",\n "mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",\n "tiff", "bmp", "seq". If the filename is a list, a list of image filenames\n are expected. If filename is a folder, it will be searched for images.\n backend: An object that implements the basic methods for reading and\n manipulating frames of a specific video type.\n backend_metadata: A dictionary of metadata specific to the backend. This is\n useful for storing metadata that requires an open backend (e.g., shape\n information) without having access to the video file itself.\n source_video: The source video object if this is a proxy video. This is present\n when the video contains an embedded subset of frames from another video.\n open_backend: Whether to open the backend when the video is available. If `True`\n (the default), the backend will be automatically opened if the video exists.\n Set this to `False` when you want to manually open the backend, or when the\n you know the video file does not exist and you want to avoid trying to open\n the file.\n _exists_cache: Per-instance TTL cache for the result of `exists()` when the\n `filename` is a remote URL. Keyed by `(filename, dataset)` and storing\n `(exists_bool, monotonic_timestamp)`. This avoids issuing a network probe\n on every call (e.g. from the `is_open` property, which GUIs poll on each\n render). The TTL defaults to 60 seconds and can be overridden via the\n `SLEAP_IO_EXISTS_TTL` environment variable. The cache is cleared on\n `replace_filename`.\n\nNotes:\n Instances of this class are hashed by identity, not by value. This means that\n two `Video` instances with the same attributes will NOT be considered equal in a\n set or dict.\n\nMedia Video Plugin Support:\n For media files (mp4, avi, etc.), the following plugins are supported:\n - "opencv": Uses OpenCV (cv2) for video reading\n - "FFMPEG": Uses imageio-ffmpeg for video reading\n - "pyav": Uses PyAV for video reading\n\n Plugin aliases (case-insensitive):\n - opencv: "opencv", "cv", "cv2", "ocv"\n - FFMPEG: "FFMPEG", "ffmpeg", "imageio-ffmpeg", "imageio_ffmpeg"\n - pyav: "pyav", "av"\n\n Plugin selection priority:\n 1. Explicitly specified plugin parameter\n 2. Backend metadata plugin value\n 3. Global default (set via sio.set_default_video_plugin)\n 4. Auto-detection based on available packages\n\nSee Also:\n VideoBackend: The backend interface for reading video data.\n sleap_io.set_default_video_plugin: Set global default plugin.\n sleap_io.get_default_video_plugin: Get current default plugin.\n'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__firstlineno__ = 102
class-attribute
¶
int([x]) -> integer int(x, base=10) -> integer
Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.
If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.
int('0b100', base=0) 4
__match_args__ = ('filename', 'backend', 'backend_metadata', 'source_video', 'open_backend')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__module__ = 'sleap_io.model.video'
class-attribute
¶
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.
__slots__ = ('filename', 'backend', 'backend_metadata', 'source_video', 'open_backend', '_exists_cache', '_url_headers', '_url_stream_mode', '__weakref__')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__static_attributes__ = ('backend', 'filename')
class-attribute
¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable's items.
If the argument is a tuple, the return value is the same object.
__weakref__
property
¶
list of weak references to the object
crop_fill
property
¶
The out-of-bounds fill value for this video's crop (0 if uncropped).
crop_rect
property
¶
Crop rect (x1, y1, x2, y2) in source coords, or None if uncropped.
fps
property
¶
Return the frames per second of the video.
For MediaVideo backends, this reads FPS from the video container metadata. For other backends (ImageVideo, HDF5Video, TiffVideo), this returns the explicitly set value or None if not set.
Returns:
| Type | Description |
|---|---|
|
The FPS if known, or None if unavailable/unknown. |
grayscale
property
¶
Return whether the video is grayscale.
If the video backend is not set or it cannot determine whether the video is grayscale, this will return None.
is_cropped
property
¶
Whether this video is a virtual crop of another video.
is_open
property
¶
Check if the video backend is open.
original_video
property
¶
The root video in the provenance chain.
For embedded videos, this returns the ultimate source video by traversing the source_video chain. Returns None if this video has no source_video (i.e., it IS an original).
This property is computed by following the source_video chain to find the root. For a single-level embedding (A embeds from B), original_video returns B. For multi-level embedding (A <- B <- C), it returns C.
shape
property
¶
Return the shape of the video as (num_frames, height, width, channels).
If the video backend is not set or it cannot determine the shape of the video, this will return None.
__attrs_post_init__()
¶
Post init syntactic sugar.
Source code in sleap_io/model/video.py
__deepcopy__(memo)
¶
Deep copy the video object.
Source code in sleap_io/model/video.py
def __deepcopy__(self, memo):
"""Deep copy the video object."""
if id(self) in memo:
return memo[id(self)]
reopen = False
if self.is_open:
reopen = True
self.close()
new_video = Video(
filename=self.filename,
backend=None,
backend_metadata=self.backend_metadata.copy(),
source_video=self.source_video,
open_backend=self.open_backend,
)
memo[id(self)] = new_video
if reopen:
self.open()
return new_video
__getitem__(inds)
¶
Return the frames of the video at the given indices.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inds
|
int | list[int] | slice
|
Index or list of indices of frames to read. |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Frame or frames as a numpy array of shape |
See also: VideoBackend.get_frame, VideoBackend.get_frames
Source code in sleap_io/model/video.py
def __getitem__(self, inds: int | list[int] | slice) -> np.ndarray:
"""Return the frames of the video at the given indices.
Args:
inds: Index or list of indices of frames to read.
Returns:
Frame or frames as a numpy array of shape `(height, width, channels)` if a
scalar index is provided, or `(frames, height, width, channels)` if a list
of indices is provided.
See also: VideoBackend.get_frame, VideoBackend.get_frames
"""
if not self.is_open:
if self.open_backend:
self.open()
else:
raise ValueError(
"Video backend is not open. Call video.open() or set "
"video.open_backend to True to do automatically on frame read."
)
return self.backend[inds]
__init__(filename, backend=None, backend_metadata=NOTHING, source_video=None, open_backend=True)
¶
Method generated by attrs for class Video.
__len__()
¶
__repr__()
¶
Informal string representation (for print or format).
Source code in sleap_io/model/video.py
def __repr__(self) -> str:
"""Informal string representation (for print or format)."""
dataset = (
f"dataset={self.backend.dataset}, "
if getattr(self.backend, "dataset", "")
else ""
)
return (
"Video("
f'filename="{self.filename}", '
f"shape={self.shape}, "
f"{dataset}"
f"backend={type(self.backend).__name__}"
")"
)
__str__()
¶
apply_crop(path, *, frame_inds=None, fps=None, video_kwargs=None)
¶
Bake this video's virtual crop into a new physical video file.
Materializes the cropped frames (self[i], already cropped by the
virtual :class:~sleap_io.io.video_reading.CropVideoBackend) to path
via :class:~sleap_io.io.video_writing.VideoWriter. The crop becomes
physical: the returned video has no CropVideoBackend / /video_crops
entry. baked.shape equals this video's cropped shape when the cropped
width and height are multiples of 16; otherwise the H.264 encoder pads the
bottom/right edges up to the next multiple of 16 (the macro-block size),
so baked.shape may exceed the cropped shape on those edges. The
top-left content is preserved, so coordinates stay aligned regardless.
This operation is coordinate-neutral. A virtual crop already presents
cropped-frame coordinates, so baking the cropped pixels does not change
any point coordinates (unlike sio transform --crop, which applies a
new crop and adjusts coordinates).
Provenance is preserved: the returned video's source_video is the
uncropped original — self.source_video (the parent a virtual crop is
created against), or, for a manually-built crop with no parent, an
uncropped view reconstructed from the crop backend's inner. So
baked.source_video.shape is the uncropped shape while baked.shape
is the cropped shape, and baked.grayscale is carried from this video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Path to the new video file. Should end in MP4. |
required |
frame_inds
|
list[int] | ndarray | None
|
Frame indices to bake. Can be specified as a list or array of frame integers. If not specified, bakes all video frames. |
None
|
fps
|
float | None
|
Frames per second for the output video. If not specified, uses this video's FPS if available, otherwise defaults to 30. |
None
|
video_kwargs
|
dict[str, Any] | None
|
A dictionary of keyword arguments to provide to
|
None
|
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Raises:
| Type | Description |
|---|---|
ValueError
|
If this video has no virtual crop to apply (i.e.,
:meth: |
Source code in sleap_io/model/video.py
def apply_crop(
self,
path: str | Path,
*,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Bake this video's virtual crop into a new physical video file.
Materializes the cropped frames (``self[i]``, already cropped by the
virtual :class:`~sleap_io.io.video_reading.CropVideoBackend`) to ``path``
via :class:`~sleap_io.io.video_writing.VideoWriter`. The crop becomes
physical: the returned video has no ``CropVideoBackend`` / ``/video_crops``
entry. ``baked.shape`` equals this video's cropped shape when the cropped
width and height are multiples of 16; otherwise the H.264 encoder pads the
bottom/right edges up to the next multiple of 16 (the macro-block size),
so ``baked.shape`` may exceed the cropped shape on those edges. The
top-left content is preserved, so coordinates stay aligned regardless.
This operation is coordinate-neutral. A virtual crop already presents
cropped-frame coordinates, so baking the cropped pixels does not change
any point coordinates (unlike ``sio transform --crop``, which applies a
new crop and adjusts coordinates).
Provenance is preserved: the returned video's ``source_video`` is the
uncropped original — ``self.source_video`` (the parent a virtual crop is
created against), or, for a manually-built crop with no parent, an
uncropped view reconstructed from the crop backend's inner. So
``baked.source_video.shape`` is the uncropped shape while ``baked.shape``
is the cropped shape, and ``baked.grayscale`` is carried from this video.
Args:
path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to bake. Can be specified as a list or array
of frame integers. If not specified, bakes all video frames.
fps: Frames per second for the output video. If not specified, uses
this video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
``sio.save_video`` for video compression.
Returns:
A new ``Video`` pointing to the baked file, with ``source_video`` set
to the uncropped original (or this video) and ``grayscale`` carried
from this video.
Raises:
ValueError: If this video has no virtual crop to apply (i.e.,
:meth:`_crop_tuple` returns ``None``). Use :meth:`save` to
re-encode an uncropped video.
"""
if self._crop_tuple() is None:
raise ValueError(
"apply_crop requires a cropped video (a virtual crop created via "
"Video.crop / Video.from_crop), but this video has no crop to "
"apply. Use Video.save to re-encode an uncropped video."
)
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
if frame_inds is None:
# A crop over a SPARSELY embedded video (frame_map keys are not the dense
# range 0..N-1, e.g. {5, 9}) cannot be baked by default: writing the frames
# compacts them to 0..k-1, so any labeled frame referencing a source index
# (5, 9) would dangle. Refuse with a clear error rather than crash or
# silently misalign. An explicit frame_inds bypasses this for advanced use.
inner = getattr(self.backend, "inner", None)
frame_map = getattr(inner, "frame_map", None)
if frame_map:
keys = sorted(frame_map.keys())
if keys != list(range(len(keys))):
raise ValueError(
"Cannot bake a virtual crop over a video with sparsely "
f"embedded frames (frame_map keys {keys}): baking would "
"compact frames to a contiguous range and break frame_idx "
"references. Pass explicit frame_inds to override, or "
"materialize from the original source video."
)
frame_inds = np.arange(len(self))
# Use this video's FPS if not explicitly specified.
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
baked = Video.from_filename(path, grayscale=self.grayscale)
# Provenance: the uncropped original. Walk past any still-virtual crop
# ancestors (a flattened crop-of-crop's source_video may itself be a crop)
# to the first uncropped ancestor. For a manually-built crop with no parent,
# reconstruct an uncropped view from the crop backend's inner, so
# source_video is never a cropped video.
source = self.source_video
while source is not None and source._crop_tuple() is not None:
source = source.source_video
if source is None:
inner = getattr(self.backend, "inner", None)
source = (
Video(filename=inner.filename, backend=inner)
if inner is not None
else self
)
baked.source_video = source
return baked
close()
¶
Close the video backend.
Source code in sleap_io/model/video.py
def close(self):
"""Close the video backend."""
if self.backend is not None:
# Try to remember values from previous backend if available and not
# specified.
try:
self.backend_metadata["dataset"] = getattr(
self.backend, "dataset", None
)
self.backend_metadata["grayscale"] = getattr(
self.backend, "grayscale", None
)
self.backend_metadata["shape"] = getattr(self.backend, "shape", None)
self.backend_metadata["fps"] = getattr(self.backend, "fps", None)
# Persist the crop so a Video cropped in-memory (never loaded
# from disk) survives a close()->open() and deepcopy: open()
# re-wraps from these keys (the closed-path shape above is
# already the cropped shape).
from sleap_io.io.video_reading import CropVideoBackend
if isinstance(self.backend, CropVideoBackend):
self.backend_metadata["crop"] = list(self.backend.crop)
self.backend_metadata["crop_fill"] = self.backend.fill
except Exception:
pass
# Deterministically release the backend's open handles (the cached
# reader and, for a remote HDF5Video, the fsspec URL file-like)
# rather than relying on garbage collection.
try:
self.backend.close()
except Exception:
pass
del self.backend
self.backend = None
crop(crop=None, *, bbox=None, roi=None, center=None, size=None, margin=0, fill=0, share_decode=True)
¶
Return a virtual, on-read cropped view of this video.
Exactly one region spec must be given: crop (explicit
(x1, y1, x2, y2) rect), bbox, roi (its axis-aligned bounds +
margin), or (center, size) for a fixed-size centered/
centroid-following window. The returned Video shares no pixels with
this one; frames are decoded on read and cropped (byte-identical to
:func:sleap_io.transform.frame.crop_frame). Out-of-bounds regions are
pad-filled with fill (never clamped), so the output shape is always
exactly (y2 - y1, x2 - x1).
The crop composes (FLATTENS when fills agree and the region is in-bounds)
with any existing crop on this video via
:meth:CropVideoBackend.wrap. source_video is set to this video for
provenance. When share_decode (the default), the new crop reuses this
video's backend instance as the shared inner so a mosaic of tiles over
one file decodes each source frame once; in that case the new tile does
NOT own the shared decoder (this video does).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
crop
|
tuple[int, int, int, int] | None
|
Explicit crop region |
None
|
bbox
|
tuple[float, float, float, float] | None
|
A bounding box |
None
|
roi
|
object | None
|
Any object exposing axis-aligned |
None
|
center
|
tuple[float, float] | None
|
Window center |
None
|
size
|
tuple[int, int] | None
|
Fixed output |
None
|
margin
|
int
|
Pixels added around the |
0
|
fill
|
int | tuple[int, ...]
|
Fill value for out-of-bounds regions. |
0
|
share_decode
|
bool
|
If |
True
|
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Source code in sleap_io/model/video.py
def crop(
self,
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
) -> "Video":
"""Return a virtual, on-read cropped view of this video.
Exactly one region spec must be given: ``crop`` (explicit
``(x1, y1, x2, y2)`` rect), ``bbox``, ``roi`` (its axis-aligned bounds +
``margin``), or (``center``, ``size``) for a fixed-size centered/
centroid-following window. The returned ``Video`` shares no pixels with
this one; frames are decoded on read and cropped (byte-identical to
:func:`sleap_io.transform.frame.crop_frame`). Out-of-bounds regions are
pad-filled with ``fill`` (never clamped), so the output shape is always
exactly ``(y2 - y1, x2 - x1)``.
The crop composes (FLATTENS when fills agree and the region is in-bounds)
with any existing crop on this video via
:meth:`CropVideoBackend.wrap`. ``source_video`` is set to this video for
provenance. When ``share_decode`` (the default), the new crop reuses this
video's backend instance as the shared inner so a mosaic of tiles over
one file decodes each source frame once; in that case the new tile does
NOT own the shared decoder (this video does).
Args:
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2``
exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: Any object exposing axis-aligned ``.bounds`` as
``(minx, miny, maxx, maxy)`` (e.g. a shapely geometry).
center: Window center ``(cx, cy)`` (used with ``size``).
size: Fixed output ``(width, height)`` (used with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (the default), reuse this video's backend
as the shared inner so tiles decode each frame once; the new tile
does not own the shared decoder.
Returns:
A new ``Video`` exposing the cropped view.
"""
from sleap_io.io.video_reading import CropVideoBackend
rect = _resolve_crop_rect(crop, bbox, roi, center, size, margin)
if self.backend is None and self.open_backend:
self.open()
if self.backend is None:
raise ValueError(
"Cannot crop a video with no open backend. Open it first (set "
"open_backend=True or call .open()) before cropping."
)
inner = self.backend
cropped_backend = CropVideoBackend.wrap(
inner=inner, crop=rect, fill=fill, owns_inner=not share_decode
)
cropped = Video(
filename=self.filename,
backend=cropped_backend,
source_video=self,
open_backend=self.open_backend,
)
x1, y1, x2, y2 = cropped_backend.crop
src_shape = self.shape
cropped.backend_metadata = {
**self.backend_metadata,
"shape": (src_shape[0], y2 - y1, x2 - x1, src_shape[3])
if src_shape is not None
else None,
# The uncropped source shape, so a closed re-serialize keeps videos_json
# describing the full frame even without a live source_video (D-120/DI-2).
"source_shape": list(src_shape) if src_shape is not None else None,
# COMPOSED source rect from wrap (D-120): keeps open/closed crop keys
# identical and root-canonical, and survives close()->open().
"crop": list(cropped_backend.crop),
"crop_fill": cropped_backend.fill,
}
return cropped
deduplicate_with(other)
¶
Create a new video with duplicate images removed.
This method is specifically for ImageVideo backends (image sequences).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to deduplicate against. Must also be ImageVideo. |
required |
Returns:
| Type | Description |
|---|---|
Video
|
A new Video object with duplicate images removed from this video, or None if all images were duplicates. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If either video is not an ImageVideo backend. |
Notes
Only works with ImageVideo backends where filename is a list. Images are considered duplicates if they have the same basename. The returned video contains only images from this video that are not present in the other video.
Source code in sleap_io/model/video.py
def deduplicate_with(self, other: "Video") -> "Video":
"""Create a new video with duplicate images removed.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to deduplicate against. Must also be ImageVideo.
Returns:
A new Video object with duplicate images removed from this video,
or None if all images were duplicates.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
Images are considered duplicates if they have the same basename.
The returned video contains only images from this video that are
not present in the other video.
"""
if not isinstance(self.filename, list):
raise ValueError("deduplicate_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get basenames from other video
other_basenames = set(Path(f).name for f in other.filename)
# Keep only non-duplicate images
deduplicated_paths = [
f for f in self.filename if Path(f).name not in other_basenames
]
if not deduplicated_paths:
# All images were duplicates
return None
# Create new video with deduplicated images
return Video.from_filename(deduplicated_paths, grayscale=self.grayscale)
exists(check_all=False, dataset=None)
¶
Check if the video file exists and is accessible.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
check_all
|
bool
|
If |
False
|
dataset
|
str | None
|
Name of dataset in HDF5 file. If specified, this will function will
return |
None
|
Returns:
| Type | Description |
|---|---|
bool
|
|
Source code in sleap_io/model/video.py
def exists(self, check_all: bool = False, dataset: str | None = None) -> bool:
"""Check if the video file exists and is accessible.
Args:
check_all: If `True`, check that all filenames in a list exist. If `False`
(the default), check that the first filename exists.
dataset: Name of dataset in HDF5 file. If specified, this will function will
return `False` if the dataset does not exist.
Returns:
`True` if the file exists and is accessible, `False` otherwise.
"""
if isinstance(self.filename, list):
if check_all:
for f in self.filename:
if not is_file_accessible(f):
return False
return True
else:
return is_file_accessible(self.filename[0])
# URL fast path: must run BEFORE `is_file_accessible`, which treats the
# filename as a local path and would spuriously return False for a URL.
from sleap_io.io._remote import _is_url
if _is_url(self.filename):
return self._url_exists(dataset)
file_is_accessible = is_file_accessible(self.filename)
if not file_is_accessible:
# Check if it's a directory (ImageVideo source)
if Path(self.filename).is_dir():
return True
return False
if dataset is None or dataset == "":
dataset = self.backend_metadata.get("dataset", None)
if dataset is not None and dataset != "":
has_dataset = False
if (
self.backend is not None
and type(self.backend) is HDF5Video
and self.backend._open_reader is not None
):
has_dataset = dataset in self.backend._open_reader
else:
with h5py.File(self.filename, "r") as f:
has_dataset = dataset in f
return has_dataset
return True
frame_to_seconds(frame_idx)
¶
Convert a frame index to timestamp in seconds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame_idx
|
int
|
Zero-indexed frame number. |
required |
Returns:
| Type | Description |
|---|---|
float | None
|
Time in seconds, or None if FPS is unknown. |
Notes
This assumes constant frame rate. For variable frame rate videos, the returned timestamp may be approximate.
Source code in sleap_io/model/video.py
def frame_to_seconds(self, frame_idx: int) -> float | None:
"""Convert a frame index to timestamp in seconds.
Args:
frame_idx: Zero-indexed frame number.
Returns:
Time in seconds, or None if FPS is unknown.
Notes:
This assumes constant frame rate. For variable frame rate videos,
the returned timestamp may be approximate.
"""
if self.fps is None or self.fps <= 0:
return None
return frame_idx / self.fps
from_crop(video, crop=None, *, bbox=None, roi=None, center=None, size=None, margin=0, fill=0, share_decode=True, **kwargs)
classmethod
¶
Open video (path or Video) and return a virtual crop.
Accepts the same region specs as :meth:crop (crop/bbox/roi/
center+size); extra keyword arguments are forwarded to
:meth:from_filename when video is a path (ignored when it is already
a Video).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
video
|
str | Path | Video
|
A path/filename to open, or an existing |
required |
crop
|
tuple[int, int, int, int] | None
|
Explicit crop region |
None
|
bbox
|
tuple[float, float, float, float] | None
|
A bounding box |
None
|
roi
|
object | None
|
An object exposing axis-aligned |
None
|
center
|
tuple[float, float] | None
|
Window center |
None
|
size
|
tuple[int, int] | None
|
Fixed output |
None
|
margin
|
int
|
Pixels added around the |
0
|
fill
|
int | tuple[int, ...]
|
Fill value for out-of-bounds regions. |
0
|
share_decode
|
bool
|
If |
True
|
**kwargs
|
Forwarded to :meth: |
required |
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Source code in sleap_io/model/video.py
@classmethod
def from_crop(
cls,
video: "str | Path | Video",
crop: tuple[int, int, int, int] | None = None,
*,
bbox: tuple[float, float, float, float] | None = None,
roi: object | None = None,
center: tuple[float, float] | None = None,
size: tuple[int, int] | None = None,
margin: int = 0,
fill: int | tuple[int, ...] = 0,
share_decode: bool = True,
**kwargs,
) -> "Video":
"""Open ``video`` (path or ``Video``) and return a virtual crop.
Accepts the same region specs as :meth:`crop` (``crop``/``bbox``/``roi``/
``center``+``size``); extra keyword arguments are forwarded to
:meth:`from_filename` when ``video`` is a path (ignored when it is already
a ``Video``).
Args:
video: A path/filename to open, or an existing ``Video`` to crop.
crop: Explicit crop region ``(x1, y1, x2, y2)``, ``x2``/``y2`` exclusive.
bbox: A bounding box ``(x1, y1, x2, y2)``; bounds may be float.
roi: An object exposing axis-aligned ``.bounds`` (e.g. a shapely
geometry); ``margin`` is applied around it.
center: Window center ``(cx, cy)`` (with ``size``).
size: Fixed output ``(width, height)`` (with ``center``).
margin: Pixels added around the ``roi`` bounds on every side.
fill: Fill value for out-of-bounds regions.
share_decode: If ``True`` (default), reuse the source decoder.
**kwargs: Forwarded to :meth:`from_filename` for a path input.
Returns:
A new ``Video`` exposing the cropped view.
"""
if isinstance(video, (str, Path)):
video = cls.from_filename(video, **kwargs)
return video.crop(
crop,
bbox=bbox,
roi=roi,
center=center,
size=size,
margin=margin,
fill=fill,
share_decode=share_decode,
)
from_filename(filename, dataset=None, grayscale=None, keep_open=True, source_video=None, **kwargs)
classmethod
¶
Create a Video from a filename.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filename
|
str | list[str]
|
The filename(s) of the video. Supported extensions: "mp4", "avi", "mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif", "tiff", "bmp". If the filename is a list, a list of image filenames are expected. If filename is a folder, it will be searched for images. |
required |
dataset
|
str | None
|
Name of dataset in HDF5 file. |
None
|
grayscale
|
bool | None
|
Whether to force grayscale. If None, autodetect on first frame load. |
None
|
keep_open
|
bool
|
Whether to keep the video reader open between calls to read frames. If False, will close the reader after each call. If True (the default), it will keep the reader open and cache it for subsequent calls which may enhance the performance of reading multiple frames. |
True
|
source_video
|
Video | None
|
The source video object if this is a proxy video. This is present when the video contains an embedded subset of frames from another video. |
None
|
**kwargs
|
Additional backend-specific arguments passed to VideoBackend.from_filename. See VideoBackend.from_filename for supported arguments. |
required |
Returns:
| Type | Description |
|---|---|
VideoBackend
|
Video instance with the appropriate backend instantiated. |
Source code in sleap_io/model/video.py
@classmethod
def from_filename(
cls,
filename: str | list[str],
dataset: str | None = None,
grayscale: bool | None = None,
keep_open: bool = True,
source_video: "Video | None" = None,
**kwargs,
) -> VideoBackend:
"""Create a Video from a filename.
Args:
filename: The filename(s) of the video. Supported extensions: "mp4", "avi",
"mov", "mj2", "mkv", "h5", "hdf5", "slp", "png", "jpg", "jpeg", "tif",
"tiff", "bmp". If the filename is a list, a list of image filenames are
expected. If filename is a folder, it will be searched for images.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
source_video: The source video object if this is a proxy video. This is
present when the video contains an embedded subset of frames from
another video.
**kwargs: Additional backend-specific arguments passed to
VideoBackend.from_filename. See VideoBackend.from_filename for supported
arguments.
Returns:
Video instance with the appropriate backend instantiated.
"""
backend = VideoBackend.from_filename(
filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
**kwargs,
)
# If filename is a directory, VideoBackend.from_filename will expand it
# to a list of paths to images contained within the directory. In this
# case we want to use the expanded list as filename
return cls(
filename=backend.filename,
backend=backend,
source_video=source_video,
)
has_overlapping_images(other)
¶
Check if this video has overlapping images with another video.
This method is specifically for ImageVideo backends (image sequences).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if both are ImageVideo instances with overlapping image files. False if either video is not an ImageVideo or no overlap exists. |
Notes
Only works with ImageVideo backends where filename is a list. Compares individual image filenames (basenames only).
Source code in sleap_io/model/video.py
def has_overlapping_images(self, other: "Video") -> bool:
"""Check if this video has overlapping images with another video.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to compare with.
Returns:
True if both are ImageVideo instances with overlapping image files.
False if either video is not an ImageVideo or no overlap exists.
Notes:
Only works with ImageVideo backends where filename is a list.
Compares individual image filenames (basenames only).
"""
# Both must be image sequences
if not (isinstance(self.filename, list) and isinstance(other.filename, list)):
return False
# Get basenames for comparison
self_basenames = set(Path(f).name for f in self.filename)
other_basenames = set(Path(f).name for f in other.filename)
# Check if there's any overlap
return len(self_basenames & other_basenames) > 0
matches_content(other)
¶
Check if this video has the same content as another video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the videos have the same shape and backend type. |
Notes
This compares metadata like shape and backend type, not actual frame data.
Source code in sleap_io/model/video.py
def matches_content(self, other: "Video") -> bool:
"""Check if this video has the same content as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same shape and backend type.
Notes:
This compares metadata like shape and backend type, not actual frame data.
"""
# Compare shapes
self_shape = self.shape
other_shape = other.shape
if self_shape != other_shape:
return False
# Compare backend types
if self.backend is None and other.backend is None:
return True
elif self.backend is None or other.backend is None:
return False
return type(self.backend).__name__ == type(other.backend).__name__
matches_path(other, strict=False)
¶
Check if this video has the same path as another video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
strict
|
bool
|
If True, require exact path match. If False, consider videos with the same filename (basename) as matching. |
False
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the videos have matching paths, False otherwise. |
Notes
For HDF5 video backends (e.g., embedded videos in .pkg.slp files), matching prioritizes the source_filename attribute since multiple videos can share the same HDF5 file path but reference different source videos. Falls back to dataset name matching if source_filename is not available.
Source code in sleap_io/model/video.py
def matches_path(self, other: "Video", strict: bool = False) -> bool:
"""Check if this video has the same path as another video.
Args:
other: Another video to compare with.
strict: If True, require exact path match. If False, consider videos
with the same filename (basename) as matching.
Returns:
True if the videos have matching paths, False otherwise.
Notes:
For HDF5 video backends (e.g., embedded videos in .pkg.slp files),
matching prioritizes the source_filename attribute since multiple
videos can share the same HDF5 file path but reference different
source videos. Falls back to dataset name matching if source_filename
is not available.
"""
# Handle HDF5 backends specially - prioritize source_filename matching
self_is_hdf5 = isinstance(self.backend, HDF5Video)
other_is_hdf5 = isinstance(other.backend, HDF5Video)
if self_is_hdf5 and other_is_hdf5:
# Both are HDF5 videos - must match by BOTH source_filename AND dataset
# to distinguish different videos embedded in the same pkg.slp file
self_source = self.backend.source_filename
other_source = other.backend.source_filename
self_dataset = self.backend.dataset
other_dataset = other.backend.dataset
# If both have datasets, they must match
if self_dataset is not None and other_dataset is not None:
if self_dataset != other_dataset:
return False # Different datasets = different videos
# If both have source_filenames, compare them
if self_source is not None and other_source is not None:
if strict:
# For HDF5 videos, just compare normalized path strings
# (avoid slow resolve() on network paths)
return Path(self_source).as_posix() == Path(other_source).as_posix()
else:
return Path(self_source).name == Path(other_source).name
# If only datasets available (no source_filename), they must match
if self_dataset is not None and other_dataset is not None:
return self_dataset == other_dataset
# If neither source_filename nor dataset available, cannot match
return False
if isinstance(self.filename, list) and isinstance(other.filename, list):
# Both are image sequences
if strict:
return self.filename == other.filename
else:
# Compare basenames
self_basenames = [Path(f).name for f in self.filename]
other_basenames = [Path(f).name for f in other.filename]
return self_basenames == other_basenames
elif isinstance(self.filename, list) or isinstance(other.filename, list):
# One is image sequence, other is single file
return False
else:
# Both are single files - use resolve() for symlink handling
if strict:
p1, p2 = Path(self.filename), Path(other.filename)
# Fast string comparison first
if p1.as_posix() == p2.as_posix():
return True
# Only resolve if both exist locally (avoid slow network timeouts)
try:
if p1.exists() and p2.exists():
return p1.resolve() == p2.resolve()
except OSError:
pass
return False
else:
return Path(self.filename).name == Path(other.filename).name
matches_shape(other)
¶
Check if this video has the same shape as another video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to compare with. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the videos have the same height, width, and channels. |
Notes
This only compares spatial dimensions, not the number of frames.
Source code in sleap_io/model/video.py
def matches_shape(self, other: "Video") -> bool:
"""Check if this video has the same shape as another video.
Args:
other: Another video to compare with.
Returns:
True if the videos have the same height, width, and channels.
Notes:
This only compares spatial dimensions, not the number of frames.
"""
# Try to get shape from backend metadata first if shape is not available
if self.backend is None and "shape" in self.backend_metadata:
self_shape = self.backend_metadata["shape"]
else:
self_shape = self.shape
if other.backend is None and "shape" in other.backend_metadata:
other_shape = other.backend_metadata["shape"]
else:
other_shape = other.shape
# Handle None shapes
if self_shape is None or other_shape is None:
return False
# Compare only height, width, channels (not frames)
return self_shape[1:] == other_shape[1:]
merge_with(other)
¶
Merge another video's images into this one.
This method is specifically for ImageVideo backends (image sequences).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Video
|
Another video to merge with. Must also be ImageVideo. |
required |
Returns:
| Type | Description |
|---|---|
Video
|
A new Video object with unique images from both videos. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If either video is not an ImageVideo backend. |
Notes
Only works with ImageVideo backends where filename is a list. The merged video contains all unique images from both videos, with automatic deduplication based on image basename.
Source code in sleap_io/model/video.py
def merge_with(self, other: "Video") -> "Video":
"""Merge another video's images into this one.
This method is specifically for ImageVideo backends (image sequences).
Args:
other: Another video to merge with. Must also be ImageVideo.
Returns:
A new Video object with unique images from both videos.
Raises:
ValueError: If either video is not an ImageVideo backend.
Notes:
Only works with ImageVideo backends where filename is a list.
The merged video contains all unique images from both videos,
with automatic deduplication based on image basename.
"""
if not isinstance(self.filename, list):
raise ValueError("merge_with only works with ImageVideo backends")
if not isinstance(other.filename, list):
raise ValueError("Other video must also be ImageVideo backend")
# Get all unique images (by basename) preserving order
seen_basenames = set()
merged_paths = []
for path in self.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
for path in other.filename:
basename = Path(path).name
if basename not in seen_basenames:
merged_paths.append(path)
seen_basenames.add(basename)
# Create new video with merged images
return Video.from_filename(merged_paths, grayscale=self.grayscale)
open(filename=None, dataset=None, grayscale=None, keep_open=True, plugin=None)
¶
Open the video backend for reading.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filename
|
str | None
|
Filename to open. If not specified, will use the filename set on the video object. |
None
|
dataset
|
str | None
|
Name of dataset in HDF5 file. |
None
|
grayscale
|
str | None
|
Whether to force grayscale. If None, autodetect on first frame load. |
None
|
keep_open
|
bool
|
Whether to keep the video reader open between calls to read frames. If False, will close the reader after each call. If True (the default), it will keep the reader open and cache it for subsequent calls which may enhance the performance of reading multiple frames. |
True
|
plugin
|
str | None
|
Video plugin to use for MediaVideo files. One of "opencv", "FFMPEG", or "pyav". Also accepts aliases (case-insensitive). If not specified, uses the backend metadata, global default, or auto-detection in that order. |
None
|
Notes
This is useful for opening the video backend to read frames and then closing it after reading all the necessary frames.
If the backend was already open, it will be closed before opening a new one. Values for the HDF5 dataset and grayscale will be remembered if not specified.
Source code in sleap_io/model/video.py
def open(
self,
filename: str | None = None,
dataset: str | None = None,
grayscale: str | None = None,
keep_open: bool = True,
plugin: str | None = None,
):
"""Open the video backend for reading.
Args:
filename: Filename to open. If not specified, will use the filename set on
the video object.
dataset: Name of dataset in HDF5 file.
grayscale: Whether to force grayscale. If None, autodetect on first frame
load.
keep_open: Whether to keep the video reader open between calls to read
frames. If False, will close the reader after each call. If True (the
default), it will keep the reader open and cache it for subsequent calls
which may enhance the performance of reading multiple frames.
plugin: Video plugin to use for MediaVideo files. One of "opencv",
"FFMPEG", or "pyav". Also accepts aliases (case-insensitive).
If not specified, uses the backend metadata, global default,
or auto-detection in that order.
Notes:
This is useful for opening the video backend to read frames and then closing
it after reading all the necessary frames.
If the backend was already open, it will be closed before opening a new one.
Values for the HDF5 dataset and grayscale will be remembered if not
specified.
"""
if filename is not None:
self.replace_filename(filename, open=False)
# Try to remember values from previous backend if available and not specified.
if self.backend is not None:
if dataset is None:
dataset = getattr(self.backend, "dataset", None)
if grayscale is None:
grayscale = getattr(self.backend, "grayscale", None)
else:
if dataset is None and "dataset" in self.backend_metadata:
dataset = self.backend_metadata["dataset"]
if grayscale is None:
if "grayscale" in self.backend_metadata:
grayscale = self.backend_metadata["grayscale"]
elif "shape" in self.backend_metadata:
grayscale = self.backend_metadata["shape"][-1] == 1
if not self.exists(dataset=dataset):
from sleap_io.io._remote import _is_url, _redact_url
# Redact credential-bearing URLs (e.g. presigned ``?token=`` links)
# so they never surface in tracebacks/logs. Local paths are shown
# verbatim.
name = (
_redact_url(self.filename)
if isinstance(self.filename, str) and _is_url(self.filename)
else self.filename
)
msg = f"Video does not exist or cannot be opened for reading: {name}"
if dataset is not None:
msg += f" (dataset: {dataset})"
raise FileNotFoundError(msg)
# Close previous backend if open.
self.close()
# Handle plugin parameter
backend_kwargs = {}
if plugin is not None:
from sleap_io.io.video_reading import normalize_plugin_name
plugin = normalize_plugin_name(plugin)
self.backend_metadata["plugin"] = plugin
if "plugin" in self.backend_metadata:
backend_kwargs["plugin"] = self.backend_metadata["plugin"]
# Create new backend. Forward the URL auth context so a reopened remote
# HDF5Video stays authenticated (the previous backend, and its headers,
# were dropped by self.close() above).
self.backend = VideoBackend.from_filename(
self.filename,
dataset=dataset,
grayscale=grayscale,
keep_open=keep_open,
url_headers=self._url_headers,
url_stream_mode=self._url_stream_mode,
**backend_kwargs,
)
# Re-wrap as a crop view if this video records a crop in its metadata.
# The rebuilt backend above is always a plain backend, so this wraps
# exactly once (idempotent across close()->open() and deepcopy).
if "crop" in self.backend_metadata:
from sleap_io.io.video_reading import CropVideoBackend
self.backend = CropVideoBackend.wrap(
inner=self.backend,
crop=tuple(self.backend_metadata["crop"]),
fill=self.backend_metadata.get("crop_fill", 0),
)
replace_filename(new_filename, open=True)
¶
Update the filename of the video, optionally opening the backend.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_filename
|
str | Path | list[str] | list[Path]
|
New filename to set for the video. |
required |
open
|
bool
|
If |
True
|
Source code in sleap_io/model/video.py
def replace_filename(
self, new_filename: str | Path | list[str] | list[Path], open: bool = True
):
"""Update the filename of the video, optionally opening the backend.
Args:
new_filename: New filename to set for the video.
open: If `True` (the default), open the backend with the new filename. If
the new filename does not exist, no error is raised.
"""
if isinstance(new_filename, Path):
new_filename = new_filename.as_posix()
if isinstance(new_filename, list):
new_filename = [
p.as_posix() if isinstance(p, Path) else p for p in new_filename
]
# A relink to a different file makes the recorded shape/grayscale/fps in
# ``backend_metadata`` stale: they describe the OLD file but the new file
# may have a different resolution/channels/frame rate. They must not be
# serialized under the new filename (regression from #483, where
# ``save_slp(prefer_metadata=True)`` prefers these recorded values), so
# invalidate them on a real relink and let them be recomputed from the new
# backend. The no-relink path leaves metadata untouched so golden
# byte-identical saves stay byte-identical.
filename_changed = new_filename != self.filename
self.filename = new_filename
self.backend_metadata["filename"] = new_filename
# Invalidate any cached URL existence results for the previous filename.
self._exists_cache.clear()
if open:
if self.exists():
self.open()
else:
self.close()
# Drop stale metadata AFTER (re)opening: ``open()`` internally calls
# ``close()``, which would otherwise re-stamp the OLD backend's
# shape/grayscale/fps back into ``backend_metadata``.
if filename_changed:
for key in ("shape", "grayscale", "fps"):
self.backend_metadata.pop(key, None)
save(save_path, frame_inds=None, fps=None, video_kwargs=None)
¶
Save video frames to a new video file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
save_path
|
str | Path
|
Path to the new video file. Should end in MP4. |
required |
frame_inds
|
list[int] | ndarray | None
|
Frame indices to save. Can be specified as a list or array of frame integers. If not specified, saves all video frames. |
None
|
fps
|
float | None
|
Frames per second for the output video. If not specified, uses the source video's FPS if available, otherwise defaults to 30. |
None
|
video_kwargs
|
dict[str, Any] | None
|
A dictionary of keyword arguments to provide to
|
None
|
Returns:
| Type | Description |
|---|---|
Video
|
A new |
Source code in sleap_io/model/video.py
def save(
self,
save_path: str | Path,
frame_inds: list[int] | np.ndarray | None = None,
fps: float | None = None,
video_kwargs: dict[str, Any] | None = None,
) -> "Video":
"""Save video frames to a new video file.
Args:
save_path: Path to the new video file. Should end in MP4.
frame_inds: Frame indices to save. Can be specified as a list or array of
frame integers. If not specified, saves all video frames.
fps: Frames per second for the output video. If not specified, uses the
source video's FPS if available, otherwise defaults to 30.
video_kwargs: A dictionary of keyword arguments to provide to
`sio.save_video` for video compression.
Returns:
A new `Video` object pointing to the new video file.
"""
video_kwargs = {} if video_kwargs is None else video_kwargs.copy()
frame_inds = np.arange(len(self)) if frame_inds is None else frame_inds
# Use source video FPS if not explicitly specified
if fps is None:
fps = self.fps
if fps is not None and "fps" not in video_kwargs:
video_kwargs["fps"] = fps
with VideoWriter(save_path, **video_kwargs) as vw:
for frame_ind in frame_inds:
vw(self[frame_ind])
new_video = Video.from_filename(save_path, grayscale=self.grayscale)
return new_video
seconds_to_frame(seconds)
¶
Convert a timestamp in seconds to frame index.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seconds
|
float
|
Time in seconds from video start. |
required |
Returns:
| Type | Description |
|---|---|
int | None
|
Zero-indexed frame number (rounded down), or None if FPS unknown. |
Source code in sleap_io/model/video.py
def seconds_to_frame(self, seconds: float) -> int | None:
"""Convert a timestamp in seconds to frame index.
Args:
seconds: Time in seconds from video start.
Returns:
Zero-indexed frame number (rounded down), or None if FPS unknown.
"""
if self.fps is None or self.fps <= 0:
return None
return int(seconds * self.fps)
set_video_plugin(plugin)
¶
Set the video plugin and reopen the video.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
plugin
|
str
|
Video plugin to use. One of "opencv", "FFMPEG", or "pyav". Also accepts aliases (case-insensitive). |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the video is not a MediaVideo type. |
Examples:
Source code in sleap_io/model/video.py
def set_video_plugin(self, plugin: str) -> None:
"""Set the video plugin and reopen the video.
Args:
plugin: Video plugin to use. One of "opencv", "FFMPEG", or "pyav".
Also accepts aliases (case-insensitive).
Raises:
ValueError: If the video is not a MediaVideo type.
Examples:
>>> video.set_video_plugin("opencv")
>>> video.set_video_plugin("CV2") # Same as "opencv"
"""
from sleap_io.io.video_reading import MediaVideo, normalize_plugin_name
if not self.filename.endswith(MediaVideo.EXTS):
raise ValueError(f"Cannot set plugin for non-media video: {self.filename}")
plugin = normalize_plugin_name(plugin)
# Close current backend if open
was_open = self.is_open
if was_open:
self.close()
# Update backend metadata
self.backend_metadata["plugin"] = plugin
# Reopen with new plugin if it was open
if was_open:
self.open()
to_crop_coords(points)
¶
Map source-frame (x, y) into this video's cropped frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
points
|
ndarray
|
Coordinate array of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Coordinates translated into the cropped frame. If this video is not
cropped, a copy of |
Source code in sleap_io/model/video.py
def to_crop_coords(self, points: np.ndarray) -> np.ndarray:
"""Map source-frame ``(x, y)`` into this video's cropped frame.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated into the cropped frame. If this video is not
cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else crop_points(points, crop)
to_source_coords(points)
¶
Map cropped-frame (x, y) back to source-frame coordinates.
Inverse of :meth:to_crop_coords.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
points
|
ndarray
|
Coordinate array of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Coordinates translated back to source coordinates. If this video is
not cropped, a copy of |
Source code in sleap_io/model/video.py
def to_source_coords(self, points: np.ndarray) -> np.ndarray:
"""Map cropped-frame ``(x, y)`` back to source-frame coordinates.
Inverse of :meth:`to_crop_coords`.
Args:
points: Coordinate array of shape ``(..., 2)``. NaN values are
preserved.
Returns:
Coordinates translated back to source coordinates. If this video is
not cropped, a copy of ``points`` is returned unchanged.
"""
crop = self._crop_tuple()
return points.copy() if crop is None else uncrop_points(points, crop)
rodrigues_transformation(input_matrix)
¶
Convert between rotation vector and rotation matrix using Rodrigues' formula.
This function implements the Rodrigues' rotation formula to convert between: 1. A 3D rotation vector (axis-angle representation) to a 3x3 rotation matrix 2. A 3x3 rotation matrix to a 3D rotation vector
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_matrix
|
ndarray
|
A 3x3 rotation matrix or a 3x1 rotation vector. |
required |
Returns:
| Type | Description |
|---|---|
tuple[ndarray, ndarray]
|
A tuple containing the converted matrix/vector and the Jacobian (None for now). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the input is not a valid rotation matrix or vector. |
Source code in sleap_io/model/camera.py
def rodrigues_transformation(input_matrix: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
"""Convert between rotation vector and rotation matrix using Rodrigues' formula.
This function implements the Rodrigues' rotation formula to convert between:
1. A 3D rotation vector (axis-angle representation) to a 3x3 rotation matrix
2. A 3x3 rotation matrix to a 3D rotation vector
Args:
input_matrix: A 3x3 rotation matrix or a 3x1 rotation vector.
Returns:
A tuple containing the converted matrix/vector and the Jacobian (None for now).
Raises:
ValueError: If the input is not a valid rotation matrix or vector.
"""
# Matrix to vector conversion
if input_matrix.shape == (3, 3):
# Get the rotation angle (trace(R) = 1 + 2*cos(theta))
cos_theta = (np.trace(input_matrix) - 1) / 2.0
cos_theta = np.clip(cos_theta, -1.0, 1.0) # Ensure numerical stability
theta = np.arccos(cos_theta)
# Handle small angles or identity rotation
if np.isclose(theta, 0.0, atol=1e-8):
# For small angles or identity, return zero vector
return np.zeros(3), None
# Compute the rotation axis
sin_theta = np.sin(theta)
if np.isclose(sin_theta, 0.0, atol=1e-8):
# Handle 180-degree rotation (sin_theta = 0)
# Find the largest diagonal element
diag = np.diag(input_matrix)
k = np.argmax(diag)
axis = np.zeros(3)
if diag[k] > -1.0:
# Extract the column with largest diagonal
axis[k] = 1.0
v = input_matrix[:, k] + axis
axis = v / np.linalg.norm(v)
rvec = theta * axis
else:
# Normal case: extract the skew-symmetric part
axis = np.array(
[
input_matrix[2, 1] - input_matrix[1, 2],
input_matrix[0, 2] - input_matrix[2, 0],
input_matrix[1, 0] - input_matrix[0, 1],
]
) / (2.0 * sin_theta)
# Ensure the axis is a unit vector
axis_norm = np.linalg.norm(axis)
if axis_norm > 0:
axis = axis / axis_norm
rvec = theta * axis
return rvec, None
# Vector to matrix conversion
elif input_matrix.shape == (3,) or input_matrix.shape == (3, 1):
# Handle both flat and column vectors
rvec = input_matrix.ravel()
theta = np.linalg.norm(rvec)
# Handle small angles
if np.isclose(theta, 0.0, atol=1e-8):
return np.eye(3), None
# Normalize the rotation axis
axis = rvec / theta
# Create the cross-product matrix
K = np.array(
[[0, -axis[2], axis[1]], [axis[2], 0, -axis[0]], [-axis[1], axis[0], 0]]
)
# Rodrigues' formula: R = I + sin(θ)K + (1-cos(θ))K²
sin_theta = np.sin(theta)
cos_theta = np.cos(theta)
K_squared = np.dot(K, K)
rotation_matrix = np.eye(3) + sin_theta * K + (1.0 - cos_theta) * K_squared
return rotation_matrix, None
else:
raise ValueError(
f"Input must be a 3x3 matrix or a 3-element vector, got shape "
f"{input_matrix.shape}"
)