Instance Segmentation

On this page

Instance segmentation predicts each object separately. One object can have a mask, a bounding box, a class label, and keypoints. After augmentation, all of these targets must still describe the same object.

Semantic segmentation assigns a class ID to each pixel. Instance segmentation also says which object owns that pixel. This extra object ID is what the pipeline must preserve.

Instance segmentation sample with per-object masks, bboxes, and labels

The difficult part is keeping every target for object N together. Its mask, box, class label, keypoints, and keypoint names must survive, move, or disappear as one unit. If these arrays drift apart, the model trains on the wrong annotations without raising an error.

Keep one row per object

Treat each object as one row:

object i = bbox i + bbox label fields i + mask i + keypoints i + keypoint label fields i

If object i survives, every target in its row survives. If a crop or a box filter removes object i, every target in that row disappears.

Shared geometry and shared object IDs solve different problems. Applying the same crop keeps pixels and coordinates in the same frame. Instance binding also keeps row N attached to object N after filtering.

What breaks when targets filter separately

Suppose a crop removes object B's box. The bbox processor also removes B's class label. A separate mask stack can still keep mask B. Row 2 now pairs box C with mask B, and every array still has a valid shape.

The crop removes object B's box. Separate filtering leaves mask B in row 2 beside box C. Binding by object ID removes every row for B.

Compose(instance_binding=[...]) binds each object's targets before transforms run. When a crop drops object B, the pipeline removes B's mask, box, labels, and keypoints together. Surviving objects keep the same per-object format: out["instances"] is one dictionary per object.

You no longer need to track matching array indices by hand. Mosaic, CopyAndPaste, crops, and box filters use the same object rows.

The instances input format

Pass instances as a list of per-object dicts to the instances keyword:

instances = [
    {
        "mask": mask_uint8,                   # (H, W) binary
        "bbox": np.array([x1, y1, x2, y2]),   # pascal_voc, pixel coords
        "bbox_labels": {"class_name": "bowl"},
    },
    ...
]

Two rules to know up front:

  • bbox_labels and keypoint_labels inside instance dicts are reserved keys. They must be dicts keyed by BboxParams.label_fields / KeypointParams.label_fields (e.g. {"class_name": "bowl"}). A bare list raises TypeError.
  • mask and masks are mutually exclusive in instance_binding — pick one.

Three supported bindings cover the common cases.

Format A — masks (N, H, W)

instance_binding=["masks", "bboxes"] packs the per-instance masks into a stacked (N, H, W) tensor before transforms run, then unpacks them back into per-object dicts. This is the layout Mask R-CNN-style heads and panoptic decoders want.

import albumentations as A

transform = A.Compose(
    [
        A.HorizontalFlip(p=1.0),
        A.RandomCrop(height=400, width=500, p=1.0),
        A.Affine(rotate=(-30, 30), p=1.0),
    ],
    bbox_params=A.BboxParams(coord_format="pascal_voc", label_fields=["class_name"]),
    instance_binding=["masks", "bboxes"],
    seed=137,
)

out = transform(image=image, instances=instances)
# out["instances"] is the same list-of-dicts shape you fed in,
# with dropped objects removed.

Format B — multi-channel mask (H, W, N)

Same call, one-line change: instance_binding=["mask", "bboxes"]. The per-instance masks are stacked along the last axis instead of the first, producing a single mask tensor of shape (H, W, N). This is what older Keras / TF segmentation pipelines often expect.

transform = A.Compose(
    [A.HorizontalFlip(p=1.0), A.RandomCrop(height=400, width=500, p=1.0)],
    bbox_params=A.BboxParams(coord_format="pascal_voc", label_fields=["class_name"]),
    instance_binding=["mask", "bboxes"],
    seed=137,
)

Format C — bboxes + keypoints (no masks)

For pose tasks, bind bboxes and keypoints and skip masks entirely. Each instance dict gets a keypoints array and a keypoint_labels dict.

transform = A.Compose(
    [A.HorizontalFlip(p=1.0), A.Affine(rotate=(-20, 20), p=1.0), A.RandomCrop(height=400, width=500, p=1.0)],
    bbox_params=A.BboxParams(coord_format="pascal_voc", label_fields=["class_name"]),
    keypoint_params=A.KeypointParams(coord_format="xy", label_fields=["kp_kind"]),
    instance_binding=["bboxes", "keypoints"],
    seed=137,
)

instances = [
    {
        "bbox": np.array([x1, y1, x2, y2]),
        "keypoints": np.array([[cx, cy], [x1, y1]], dtype=np.float32),
        "bbox_labels": {"class_name": "person"},
        "keypoint_labels": {"kp_kind": ["center", "tl"]},
    },
    ...
]

When keypoints are bound, KeypointParams.remove_invisible and check_each_transform are forced to False. Out-of-frame keypoints stay in the dict — only instance survival (driven by the bbox processor) decides what gets dropped. Each surviving instance keeps a constant keypoint count, which is what pose models want.

Instance survival under cropping and filtering

Survival is decided by the bbox processor — min_visibility, min_area, filter_invalid_bboxes, and related options on BboxParams. Crops and geometric transforms can shrink, clip, or move a bbox out of frame. When the bbox processor drops that row, the bound mask, bbox label fields, keypoints, and keypoint label fields disappear with it; surviving instances keep their object identity intact.

transform = A.Compose(
    [A.RandomCrop(height=180, width=180, p=1.0)],
    bbox_params=A.BboxParams(
        coord_format="pascal_voc",
        label_fields=["class_name"],
        min_visibility=0.1,
    ),
    instance_binding=["masks", "bboxes"],
    seed=137,
)

out = transform(image=image, instances=instances)
# len(out["instances"]) <= len(instances), and every dict in the
# output corresponds to exactly one surviving input object.

Six random crops with surviving instance counts; masks and bboxes track the same objects

Mixing transforms: Mosaic and CopyAndPaste

The two staple instance-segmentation mixers — popularized by YOLO — are first-class transforms, both wired into instance_binding:

  • Mosaic tiles your primary image with neighbors into a single output canvas. Without per-instance binding, mosaic'd masks and bboxes drift apart at the cell seams; with it, every surviving instance — primary or neighbor — appears in out["instances"] in the same per-object shape.
  • CopyAndPaste pastes donor objects onto the primary image with mask-tight cropping, optional shrink-fit, scale jitter, and Gaussian blending at the seam. Pasted instances get fresh ids and slot in alongside the surviving primaries. Without it, you write the cut/paste compositing, the mask and bbox propagation, the visibility recompute, and the id assignment yourself — this is the canonical place where in-house augmentation stacks have bugs.

Metadata shapes are independent of instance_binding:

  • mosaic_metadata is a list of per-image dicts with stacked masks/bboxes (one entry per neighbor image).
  • copy_paste_metadata is a list of per-object dicts (one entry per donor instance).

Both feed back into the per-object output format.

Primary image, donor pool, and result of CopyAndPaste with three pasted instances

Mosaic input example

Each mosaic_metadata item is another image-level sample. Use the stacked target layout there; instance_binding converts the final result back to per-object dicts.

mosaic_metadata = [
    {
        "image": neighbor_image,
        "masks": (
            np.stack([inst["mask"] for inst in neighbor_instances])
            if neighbor_instances
            else np.empty((0, *neighbor_image.shape[:2]), dtype=np.uint8)
        ),
        "bboxes": (
            np.stack([inst["bbox"] for inst in neighbor_instances])
            if neighbor_instances
            else np.empty((0, 4), dtype=np.float32)
        ),
        "bbox_labels": {"class_name": [inst["bbox_labels"]["class_name"] for inst in neighbor_instances]},
    }
    for neighbor_image, neighbor_instances in neighbor_samples
]

out = transform(
    image=primary_image,
    instances=primary_instances,
    mosaic_metadata=mosaic_metadata,
)

Cell placement can clip objects, and box filters can remove them after the mosaic is built. Every surviving neighbor still keeps its box, mask, and box labels together.

CopyAndPaste input example

Each copy_paste_metadata item is one donor object. The transform pastes donor instances into the primary sample, then runs the same survival and filtering rules on the combined instance set.

copy_paste_metadata = [
    {
        "image": donor_image,
        "mask": donor_instance["mask"],
        "bbox_labels": {"class_name": donor_instance["bbox_labels"]["class_name"]},
    }
    for donor_image, donor_instance in donor_objects
]

out = transform(
    image=primary_image,
    instances=primary_instances,
    copy_paste_metadata=copy_paste_metadata,
)

Pasted objects enter out["instances"] in the same format as primary objects. If a pasted object is too small, outside the paste region, or hidden below min_visibility_after_paste, the whole donor row is removed.

A YOLO-style training pipeline

End-to-end pipeline equivalent to what Ultralytics ships for YOLO11-seg, assembled out of pure Albumentations primitives:

transform = A.Compose(
    [
        A.Mosaic(
            grid_yx=(2, 2),
            target_size=(640, 640),
            cell_shape=(640, 640),
            fit_mode="cover",
            p=1.0,
        ),
        A.Affine(
            scale=(0.9, 1.1),
            rotate=(-10, 10),
            shear=(-2, 2),
            translate_percent=(-0.05, 0.05),
            p=0.5,
        ),
        A.Perspective(scale=(0.02, 0.05), keep_size=True, p=0.3),
        A.CopyAndPaste(
            scale_range=(0.4, 1.0),
            blend_mode="gaussian",
            blend_sigma_range=(1.0, 2.0),
            min_visibility_after_paste=0.1,
            p=0.7,
        ),
        A.HorizontalFlip(p=0.5),
        A.HueSaturationValue(
            hue_shift_range=(-15, 15),
            sat_shift_range=(-40, 40),
            val_shift_range=(-25, 25),
            p=1.0,
        ),
        A.RandomBrightnessContrast(
            brightness_range=(-0.2, 0.2),
            contrast_range=(-0.2, 0.2),
            p=0.5,
        ),
    ],
    bbox_params=A.BboxParams(
        coord_format="pascal_voc",
        label_fields=["class_name"],
        min_visibility=0.1,
        min_area=8.0,
    ),
    instance_binding=["masks", "bboxes"],
    seed=137,
)

Per-call data assembly, sampling neighbors from a pool:

mosaic_metadata = [
    {
        "image": img,
        "masks": (
            np.stack([i["mask"] for i in instances])
            if instances
            else np.empty((0, *img.shape[:2]), dtype=np.uint8)
        ),
        "bboxes": (
            np.stack([i["bbox"] for i in instances])
            if instances
            else np.empty((0, 4), dtype=np.float32)
        ),
        "bbox_labels": {"class_name": [i["bbox_labels"]["class_name"] for i in instances]},
    }
    for img, instances in neighbor_samples
]

copy_paste_metadata = [
    {
        "image": img,
        "mask": inst["mask"],
        "bbox_labels": {"class_name": inst["bbox_labels"]["class_name"]},
    }
    for img, inst in donor_objects
]

out = transform(
    image=primary_image,
    instances=primary_instances,
    mosaic_metadata=mosaic_metadata,
    copy_paste_metadata=copy_paste_metadata,
)

out["instances"] is the surviving primaries plus pasted donors, in the same per-object dict format you fed in.

Six samples from the YOLO-style pipeline showing mosaic tiling, copy-paste, and color jitter

API conventions to know

Two breaking changes you'll hit if you copy code from older snippets:

  • Every sampling-range constructor argument ends in _range (e.g. Rotate.angle_range, Blur.blur_range, HueSaturationValue.hue_shift_range). The old *_limit names are gone.
  • Range arguments are tuples only — A.Rotate(angle_range=(-30, 30)), not A.Rotate(angle_range=30).

Pin albumentationsx>=2.2.2 for the pipeline above — earlier 2.2.x versions had ordering bugs around Mosaic + CopyAndPaste + Perspective under instance_binding.

Where to Go Next