Contents

Isolating an object from a tap, box, or scribble

Generate and refine a mask that isolates an object in an image based on a person’s selection.

Overview

When people interact with images in your app, they might want to select an object, such as an item on a desk or a subject in a photo. Vision provides GenerateIterativeSegmentationRequest to generate an image mask that separates any selected object from its background.

Unlike automatic segmentation requests that detect specific categories like people or documents, iterative segmentation works with arbitrary objects based on coordinates your app provides:

  • Point: A normalized coordinate at a tap location inside the object.

  • Bounding box: A normalized rectangle enclosing the object, useful for regional selections or grouping nearby items.

  • Scribble: A pixel buffer containing drawn strokes across one or more objects.

Vision evaluates the image at those coordinates using an on-device machine-learning model and produces an alpha mask as a PixelBufferObservation. In the resulting mask, pixel values define the object’s silhouette against the background. Your app can then use this mask to highlight the selection, separate the object from its surroundings, or render a visual effect over the subject.

The following figure depicts an object in a photo isolated from its surroundings with an active mask overlay:

[Image]

Because an initial selection might miss parts of an object or include unwanted background areas, you can let people refine the mask:

  1. Initial selection: The person selects an object by tapping it, drawing a rectangle around it, or scribbling across it. The request generates an initial mask.

  2. Refinement: The person adds include points (addIncludedPoint(_:)) to expand the mask over missed parts of the object, or exclude points (addExcludedPoint(_:)) to trim background areas that the mask covers.

  3. Updated mask: Performing the request again with the same handler incorporates the new points and recalculates the mask.

To generate the mask, the request uses a machine-learning model that Vision downloads on demand to the device. For a working example that shows how to select and refine objects in a photo, see Segmenting objects using taps, scribbles or rectangles.

Download the model on first use

Before your app performs the request the first time on a device, check whether the model is ready and download it if it isn’t. Read the request’s assetStatus property to get the current state, and call downloadAssets() if the state is DownloadableAssetsRequestStatus.notReady. The request throws an error if you perform it before Vision downloads the model.

let request = GenerateIterativeSegmentationRequest(seedPoint: NormalizedPoint(x: 0.5, y: 0.5))

switch await request.assetStatus {
case .ready:
    break
case .notReady:
    try await request.downloadAssets()
case .downloading, .error:
    return
@unknown default:
    return
}

The switch handles each DownloadableAssetsRequestStatus case so the request only proceeds after the model is on the device. To report download progress in your app’s UI, use downloadAssets(progress:) instead.

Create the request from the person’s selection

Choose one of three initializers based on how the person selected the object:

The point and rectangle initializers use the Vision framework’s normalized coordinate system, where values run from 0 to 1 and the origin sits at the lower-left corner. This differs from UIKit and SwiftUI, where the y-axis points down from the top left. Convert view coordinates to normalized coordinates and flip the y-axis before passing them to the request.

let pointRequest = GenerateIterativeSegmentationRequest(
    seedPoint: NormalizedPoint(x: 0.5, y: 0.5)
)

let boxRequest = GenerateIterativeSegmentationRequest(
    seedBox: NormalizedRect(x: 0.2, y: 0.3, width: 0.4, height: 0.5)
)

let scribbleRequest = GenerateIterativeSegmentationRequest(
    seedScribbleBuffer: CVReadOnlyPixelBuffer(unsafeBuffer: scribblePixelBuffer)
)

Each initializer accepts an optional revision parameter that pins the request to a specific algorithm revision. If you don’t pass one, the request uses the SDK’s default revision. To find the available revisions, check supportedRevisions.

Perform the request

To generate the mask, run the request against the image using an ImageRequestHandler. Create a handler for the image and call perform(_:) with the request. The result is a PixelBufferObservation whose pixelBuffer property contains the mask.

let request = GenerateIterativeSegmentationRequest(
    seedPoint: NormalizedPoint(x: 0.5, y: 0.5)
)
let handler = ImageRequestHandler(URL(filePath: "/path/to/image.jpg"))

let observation = try await handler.perform(request)
let mask = observation?.pixelBuffer

Reuse the same ImageRequestHandler instance when refining the mask. Reusing the handler preserves intermediate image representations in memory, allowing Vision to update the mask quickly as you add points without reloading the image.

Refine the mask

To expand the mask, add an include point inside the object with addIncludedPoint(_:). To trim the mask, add an exclude point outside the object with addExcludedPoint(_:). Perform the request through the same handler to get the updated mask.

The following figures show how adding include points expands the mask over omitted parts of an object:

try request.addIncludedPoint(NormalizedPoint(x: 0.6, y: 0.4))
try request.addExcludedPoint(NormalizedPoint(x: 0.2, y: 0.8))

let refinedObservation = try await handler.perform(request)
let refinedMask = refinedObservation?.pixelBuffer

The request holds up to 13 total points when you create it with a point or scribble, or 11 when you create it with a bounding box. Adding a point past this limit throws an error. To isolate a different object, create a new request and a new handler.

Choose a mask quality level

Configure qualityLevel before performing the request to balance mask resolution against processing time:

The following figures compare mask edge fidelity across each quality level:

The following code sets the quality level to accurate:

request.qualityLevel = .accurate

See Also

Image segmentation and subject lifting