albumentations.augmentations.mixing.mosaic

Combine multiple images and annotations into one image via a mosaic grid. Uses metadata for additional images; common in object detection training. Mosaic creates a grid of images by placing the primary image and additional images from metadata into cells of a larger canvas, then crops a region to produce the final output. This is commonly used in object detection training to increase data diversity and help models learn to detect objects at different scales and contexts. The transform takes a primary input image (and its annotations) and combines it with additional images/annotations provided via metadata. It calculates the geometry for a mosaic grid, selects additional items, preprocesses annotations consistently (handling label encoding updates), applies geometric transformations, and assembles the final output. Args: grid_yx (tuple[int, int]): The number of rows (y) and columns (x) in the mosaic grid. Determines the maximum number of images involved (grid_yx[0] * grid_yx[1]). Default: (2, 2). target_size (tuple[int, int]): The desired output (height, width) for the final mosaic image. after cropping the mosaic grid. cell_shape (tuple[int, int]): cell shape of each cell in the mosaic grid. fit_mode (Literal['cover', 'contain']): How to fit images into mosaic cells. - "cover": Scale image to fill the entire cell, potentially cropping parts. - "contain": Scale image to fit entirely within the cell, potentially adding padding. Default: "cover". metadata_key (str): Key in the input dictionary specifying the list of additional data dictionaries for the mosaic. Each dictionary in the list should represent one potential additional item. Expected keys: 'image' (required, np.ndarray), and optionally 'mask' (np.ndarray), 'masks' (np.ndarray, stacked instance masks), 'bboxes' (np.ndarray), 'keypoints' (np.ndarray), and label fields supplied via the `bbox_labels` and `keypoint_labels` wrapper dicts (see Metadata Format below). Default: "mosaic_metadata". center_range (tuple[float, float]): Range [0.0-1.0] to sample the center point of the mosaic view relative to the valid central region of the conceptual large grid. This affects which parts of the assembled grid are visible in the final crop. Default: (0.3, 0.7). interpolation (int): OpenCV interpolation flag used for resizing images during geometric processing. Default: cv2.INTER_LINEAR. mask_interpolation (int): OpenCV interpolation flag used for resizing masks during geometric processing. Default: cv2.INTER_NEAREST. fill (tuple[float, ...] | float): Value used for padding images if needed during geometric processing. Default: 0. fill_mask (tuple[float, ...] | float): Value used for padding masks if needed during geometric processing. Default: 0. p (float): Probability of applying the transform. Default: 0.5. Workflow (`sample_parameters`): 1. Validate Raw Additional Metadata: Filter canonical-incompatible records, then validate active image and semantic mask aliases across the valid donor pool. 2. Calculate Geometry & Visible Cells: Determine which grid cells are visible in the final `target_size` crop and their placement coordinates on the output canvas. 3. Select Subset of Raw Additional Metadata: Choose a subset of the valid raw items based on the number of visible cells requiring additional data. 4. Preprocess Selected Raw Additional Items: Preprocess bboxes/keypoints for the *selected* additional items *only*. This uses shared processors from `Compose`, updating their internal state (e.g., `LabelEncoder`) based on labels in these selected items. 5. Prepare Primary Data: Extract preprocessed primary data fields from the input `data` dictionary into a `primary` dictionary. 6. Determine & Perform Replication: If fewer additional items were selected than needed, replicate the preprocessed primary data as required. 7. Combine Final Items: Create the list of all preprocessed items (primary, selected additional, replicated primary) that will be used. 8. Assign Items to VISIBLE Grid Cells 9. Process Geometry & Shift Coordinates: For each assigned item: a. Apply geometric transforms to image/mask based on `fit_mode`: - "cover": Resize to smallest dimension covering the cell, then crop to cell size - "contain": Resize to largest dimension fitting in the cell, then pad to cell size b. Apply geometric shift to the *preprocessed* bboxes/keypoints based on cell placement. 10. Return Parameters: Return the processed cell data (image, mask, shifted bboxes, shifted kps) keyed by placement coordinates. Label Handling: - The transform relies on `bbox_processor` and `keypoint_processor` provided by `Compose`. - `Compose.preprocess` initially fits the processors' `LabelEncoder` on the primary data. - This transform (`Mosaic`) preprocesses the *selected* additional raw items using the same processors. If new labels are found, the shared `LabelEncoder` state is updated via its `update` method. - `Compose.postprocess` uses the final updated encoder state to decode all labels present in the mosaic output for the current `Compose` call. - The encoder state is transient per `Compose` call. Note: Image targets registered with `additional_targets` share the primary image's donor selection and geometry, while retaining their own pixels, dtype and channel count. Each active image target must be present in every valid donor record, aligned in H/W with that record's image, and match its primary target's dtype and channel count. HW and HWC1 donor arrays are compatible. uint8 and float32 targets can be combined; use a scalar fill for targets with different channel counts. Active image aliases may have at most 128 channels. Wider aliases and incomplete pairs raise ValueError before geometry and donor selection, including surplus donors. An alias absent from the primary call does not require corresponding donor fields. Semantic mask aliases registered as 'mask' also retain their own pixels, dtype and channels through the same donor selection and geometry. They use mask_interpolation and fill_mask. Each supplied, non-None alias requires a non-empty HW/HWC NumPy array in every valid donor, matching that donor's image H/W and its primary alias's dtype and channel count (1-128). HW and HWC1 are compatible. Mask dtypes follow the canonical mask operations; for example, int32 supports nearest resizing but not linear resizing. No dtype conversion is added. Every valid donor must also provide a non-None canonical mask when a semantic mask alias is active. Missing canonical masks or incompatible pairs raise ValueError before geometry and donor selection, including surplus donors and donors unused by a 1x1 mosaic. Registered aliases absent or None in the primary call impose no donor requirement. Without active semantic mask aliases, a missing donor mask keeps the usual fill_mask behavior. Tuple fill_mask must match the channels of every supplied semantic mask target; a scalar supports mixed channel counts. Instance 'masks' aliases are outside this paired-target contract. If fewer additional images are provided than needed to fill the grid, the primary image will be replicated to fill the remaining cells. For example, with a 2x2 grid, if only one additional image is provided, the mosaic will contain the primary image in two cells and the additional image in one cell, with one visible cell selected from these three. Stacked instance masks on the `masks` key (N, H, W) are transformed via `apply_to_masks`. Targets: image, mask, masks, bboxes, keypoints Image types: uint8, float32 Supported bboxes: hbb, obb Reference: YOLOv4: Optimal Speed and Accuracy of Object Detection: https://arxiv.org/pdf/2004.10934 Metadata Format: Each dict in the metadata list represents one additional image and must contain: - image (np.ndarray): Additional image. Required. - Registered image aliases (np.ndarray): For example, 'thermal' when the call includes `thermal` and Compose registers `additional_targets={"thermal": "image"}`. Required for each active image alias; see Note for alignment and dtype constraints. - mask (np.ndarray): Semantic mask for the additional image. Required and non-None when a semantic mask alias is active in the primary call; otherwise optional. - Registered semantic mask aliases (np.ndarray): For example, 'auxmask' registered as 'mask'. Required for each supplied non-None primary alias; see Note for pair constraints. - masks (np.ndarray): Stacked instance masks (N, H, W) for the additional image. Optional; same geometry as image. Use with instance_binding / pipeline masks target. - bboxes (np.ndarray): Bounding boxes in the **same coordinate format** as `BboxParams.coord_format` declared in `Compose`. Optional. - keypoints (np.ndarray): Keypoints in the **same format** as `KeypointParams.coord_format` declared in `Compose`. Optional. - bbox_labels (dict[str, Any]): Label lists for bboxes, keyed by label field name as declared in `BboxParams.label_fields`. Each value must be a list with one entry per bbox. E.g. `{"class_id": [3, 7], "is_crowd": [0, 1]}`. - keypoint_labels (dict[str, Any]): Label lists for keypoints, keyed by label field name as declared in `KeypointParams.label_fields`. Each value must be a list with one entry per keypoint. E.g. `{"joint_name": ["left_eye", "nose"]}`. Examples: >>> import numpy as np >>> import albumentations as A >>> import cv2 >>> >>> # Prepare primary data >>> primary_image = np.random.randint(0, 256, (100, 100, 3), dtype=np.uint8) >>> primary_mask = np.random.randint(0, 2, (100, 100), dtype=np.uint8) >>> primary_bboxes = np.array([[10, 10, 40, 40], [50, 50, 90, 90]], dtype=np.float32) >>> primary_bbox_classes = [1, 2] >>> primary_keypoints = np.array([[25, 25], [75, 75]], dtype=np.float32) >>> primary_keypoint_classes = ['eye', 'nose'] >>> >>> # Prepare additional images for mosaic. >>> # bbox_labels and keypoint_labels are dicts mapping field name -> list of values. >>> mosaic_metadata = [ ... { ... 'image': np.random.randint(0, 256, (100, 100, 3), dtype=np.uint8), ... 'mask': np.random.randint(0, 2, (100, 100), dtype=np.uint8), ... 'bboxes': np.array([[20, 20, 60, 60]], dtype=np.float32), ... 'bbox_labels': {'bbox_classes': [3]}, ... 'keypoints': np.array([[40, 40]], dtype=np.float32), ... 'keypoint_labels': {'keypoint_classes': ['mouth']}, ... }, ... { ... 'image': np.random.randint(0, 256, (100, 100, 3), dtype=np.uint8), ... 'mask': np.random.randint(0, 2, (100, 100), dtype=np.uint8), ... 'bboxes': np.array([[30, 30, 70, 70]], dtype=np.float32), ... 'bbox_labels': {'bbox_classes': [4]}, ... 'keypoints': np.array([[50, 50], [65, 65]], dtype=np.float32), ... 'keypoint_labels': {'keypoint_classes': ['eye', 'eye']}, ... }, ... ] >>> >>> transform = A.Compose([ ... A.Mosaic( ... grid_yx=(2, 2), ... target_size=(200, 200), ... cell_shape=(120, 120), ... center_range=(0.4, 0.6), ... fit_mode="cover", ... p=1.0 ... ), ... ], bbox_params=A.BboxParams(coord_format='pascal_voc', label_fields=['bbox_classes']), ... keypoint_params=A.KeypointParams(coord_format='xy', label_fields=['keypoint_classes'])) >>> >>> transformed = transform( ... image=primary_image, ... mask=primary_mask, ... bboxes=primary_bboxes, ... bbox_classes=primary_bbox_classes, ... keypoints=primary_keypoints, ... keypoint_classes=primary_keypoint_classes, ... mosaic_metadata=mosaic_metadata, ... ) >>> >>> mosaic_image = transformed['image'] >>> mosaic_bboxes = transformed['bboxes'] >>> mosaic_bbox_classes = transformed['bbox_classes'] >>> mosaic_keypoint_classes = transformed['keypoint_classes'] >>> >>> # Paired RGB and single-channel thermal inputs use the same donor records. >>> paired = [ ... {"image": np.full((8, 8, 3), index, dtype=np.uint8), ... "thermal": np.full((8, 8), 100 + index, dtype=np.uint8)} ... for index in range(4) ... ] >>> paired_transform = A.Compose( ... [A.Mosaic(cell_shape=(8, 8), target_size=(16, 16), p=1)], ... additional_targets={"thermal": "image"}, seed=137, ... ) >>> paired_result = paired_transform(**paired[0], mosaic_metadata=paired[1:]) >>> assert paired_result["thermal"].shape == (16, 16) >>> assert np.array_equal(paired_result["thermal"], paired_result["image"][..., 0] + 100)

Parameters