Image Filters

Apply high-performance filters to, and extract statistical and histogram data from images.

Overview

The MPSUnaryImageKernel and MPSBinaryImageKernel base classes define several properties common to all image kernels:

Cliprect and Cliprect	A clip rectangle is available to all image kernels that write to a destination texture. It describes the sub-rectangle of the destination texture overwritten by the filter. If the clip rectangle is larger than the destination texture, then the intersection between the clip rectangle and the destination texture bounds is used instead. A clip rectangle may be used to avoid doing work to obscured regions of the destination image, or to manage tiling and limit operations to parts of an image—for example, if a user draws a rectangle on the screen and asks your app to just apply the filter there.
Offset, Primaryoffset, and Secondaryoffset	An offset is available to all image kernels that use a source texture from which pixel data is read. It describes the positioning of the source image relative to the result texture. An offset of `{0, 0, 0}` indicates that the top left pixel of the source texture is the center pixel used to create the top left corner of the destination texture clip rectangle (as a further example, an offset of `{1, 2, 0}` positions the top left corner of the clip rectangle at position `x=1`, `y=2`, and `z=0` of the source image). The offset is the position of the top left corner of the clip rectangle in the source coordinate frame. It can be used for tiling and for translating an image up, down, left, or right by pixel increments. If there is no clip rectangle, then the offset is the top left corner of the region read by the filter. If there are multiple source textures, then the primary offset describes the top left corner of the region read in the primary source texture and the secondary offset describes the top left corner of the region read in the secondary source texture.
Edgemode, Primaryedgemode, and Secondaryedgemode	An edge mode describes the behavior of texture reads that stray off the edge of the source image. This can happen if the offset is negative, meaning a read off the top or left edge of the image. This can also happen if the sum of the clip rectangle size and the offset is larger than the source image, meaning a read off the bottom or right edge of the image. Furthermore, it is also possible for image filters to have a kernel window that stretches to examine neighboring pixels beyond the image bounds (such as convolution, morphology, and resampling filters). If there are multiple source textures, then the primary edge mode describes the mode to use with the primary source texture and the secondary edge mode describes the mode to use with the secondary source texture.

A clip rectangle is available to all image kernels that write to a destination texture. It describes the sub-rectangle of the destination texture overwritten by the filter. If the clip rectangle is larger than the destination texture, then the intersection between the clip rectangle and the destination texture bounds is used instead. A clip rectangle may be used to avoid doing work to obscured regions of the destination image, or to manage tiling and limit operations to parts of an image—for example, if a user draws a rectangle on the screen and asks your app to just apply the filter there.

Offset, Primaryoffset, and Secondaryoffset

An offset is available to all image kernels that use a source texture from which pixel data is read. It describes the positioning of the source image relative to the result texture. An offset of {0, 0, 0} indicates that the top left pixel of the source texture is the center pixel used to create the top left corner of the destination texture clip rectangle (as a further example, an offset of {1, 2, 0} positions the top left corner of the clip rectangle at position x=1, y=2, and z=0 of the source image). The offset is the position of the top left corner of the clip rectangle in the source coordinate frame. It can be used for tiling and for translating an image up, down, left, or right by pixel increments. If there is no clip rectangle, then the offset is the top left corner of the region read by the filter. If there are multiple source textures, then the primary offset describes the top left corner of the region read in the primary source texture and the secondary offset describes the top left corner of the region read in the secondary source texture.

Edgemode, Primaryedgemode, and Secondaryedgemode

An edge mode describes the behavior of texture reads that stray off the edge of the source image. This can happen if the offset is negative, meaning a read off the top or left edge of the image. This can also happen if the sum of the clip rectangle size and the offset is larger than the source image, meaning a read off the bottom or right edge of the image. Furthermore, it is also possible for image filters to have a kernel window that stretches to examine neighboring pixels beyond the image bounds (such as convolution, morphology, and resampling filters). If there are multiple source textures, then the primary edge mode describes the mode to use with the primary source texture and the secondary edge mode describes the mode to use with the secondary source texture.

In-Place Operation

Some kernels can operate in place. This means that the same texture is used to hold both the input image and the result image. Operating in place is a great way to save memory, time, and energy. You can perform an in-place operation by using the encode(commandBuffer:inPlaceTexture:fallbackCopyAllocator:) method.

Unfortunately, it is not always possible for kernels to run in place. Whether a particular kernel can operate in place can vary according to the hardware it is running on, the OS version, and the parameters and properties passed to it. You may not assume that because a kernel works in place today on a particular device that it will do so in the future.

To simplify error handling with failed in-place operation, the encode(commandBuffer:inPlaceTexture:fallbackCopyAllocator:) method takes an optional MPSCopyAllocator object. It is used to create a new texture when in-place operation is not possible so as to allow the operation to proceed out of place in a reliable fashion instead. When this happens, the input texture is released and replaced with a new texture. To make use of the feature, you will need to write a copy allocator block.

The following code listing shows a minimal copy allocator implementation. For more information, see the MPSCopyAllocator reference.

Listing 1. Minimal MPSCopyAllocator Implementation

Supported Pixel Formats for Image Kernels

All Metal Performance Shaders image kernels support source and destination textures with the following ordinary and packed pixel formats:

MTLPixelFormat.r8Unorm, MTLPixelFormat.r8Unorm_srgb: Ordinary formats with one 8-bit normalized unsigned integer component.
MTLPixelFormat.rg8Unorm, MTLPixelFormat.rg8Unorm_srgb: Ordinary formats with two 8-bit normalized unsigned integer components.
MTLPixelFormat.rgba8Unorm, MTLPixelFormat.rgba8Unorm_srgb, MTLPixelFormat.bgra8Unorm, MTLPixelFormat.bgra8Unorm_srgb: Ordinary formats with four 8-bit normalized unsigned integer components.
MTLPixelFormat.r16Float, MTLPixelFormat.rg16Float, MTLPixelFormat.rgba16Float: Ordinary format with 16-bit floating-point components.
MTLPixelFormat.r32Float, MTLPixelFormat.rg32Float, MTLPixelFormat.rgba32Float: Ordinary format with 32-bit floating-point components.
MTLPixelFormat.r16Unorm, MTLPixelFormat.rg16Unorm, MTLPixelFormat.rgba16Unorm: Ordinary format with 16-bit normalized unsigned integer components.
MTLPixelFormat.b5g6r5Unorm, MTLPixelFormat.a1bgr5Unorm, MTLPixelFormat.abgr4Unorm, MTLPixelFormat.bgr5A1Unorm: Packed 16-bit format with normalized unsigned integer color components.
MTLPixelFormat.rgb10a2Unorm: Packed 32-bit format with normalized unsigned integer color components.
MTLPixelFormat.rg11b10Float, MTLPixelFormat.rgb9e5Float: Packed 32-bit format with floating-point color components.

Some compressed pixel formats can be used as source textures. They cannot be used as destination textures because they cannot be written to. Metal Performance Shaders image kernels support the following compression families:

PVRTC
EAC/ETC
ASTC

The following Metal Performance Shaders image kernels also support source and destination textures with ordinary signed and unsigned integer pixel formats:

The ordinary signed and unsigned integer pixel formats supported by these image kernels:

MTLPixelFormat.r8Sint, MTLPixelFormat.rg8Sint, MTLPixelFormat.rgba8Sint: Ordinary format with 8-bit signed integer components.
MTLPixelFormat.r8Uint, MTLPixelFormat.rg8Uint, MTLPixelFormat.rgba8Uint: Ordinary format with 8-bit unsigned integer components.
MTLPixelFormat.r16Sint, MTLPixelFormat.rg16Sint, MTLPixelFormat.rgba16Sint: Ordinary format with 16-bit signed integer components.
MTLPixelFormat.r16Uint, MTLPixelFormat.rg16Uint, MTLPixelFormat.rgba16Uint: Ordinary format with 16-bit unsigned integer components.
MTLPixelFormat.r32Sint, MTLPixelFormat.rg32Sint, MTLPixelFormat.rgba32Sint: Ordinary format with 32-bit signed integer components.
MTLPixelFormat.r32Uint, MTLPixelFormat.rg32Uint, MTLPixelFormat.rgba32Uint: Ordinary format four 32-bit unsigned integer components.

For more information on pixel formats, see MTLPixelFormat and Pixel Format Capabilities.

Sample Code

Listing 2. Metal Performance Shaders Sample Code

#import <MetalPerformanceShaders/MetalPerformanceShaders.h>
 
// Blur the input texture (in place if possible) on MTLCommandQueue q, and return the new texture.
// This is a trivial example. It is not necessary or necessarily advised to enqueue a MPSKernel on
// its own MTLCommandBuffer or using its own MTLComputeCommandEncoder. Group work together.
 
// Here we assume that you have already gotten a MTLDevice using MTLCreateSystemDefaultDevice() or
// MTLCopyAllDevices(), used it to create a MTLCommandQueue with MTLDevice.newCommandQueue, and
// similarly made textures with the device as needed.
void  MyBlurTextureInPlace(id <MTLTexture> __strong *inTexture, float blurRadius, id <MTLCommandQueue> q)
{
    // Create the usual Metal objects.
    // MPS does not need a dedicated MTLCommandBuffer or MTLComputeCommandEncoder.
    // This is a trivial example. You should reuse the MTL objects you already have, if you have them.
    id <MTLDevice> device = q.device;
    id <MTLCommandBuffer> buffer = [q commandBuffer];
 
    // Create a MPS filter.
    MPSImageGaussianBlur *blur = [[MPSImageGaussianBlur alloc] initWithDevice: device];
    if( nil == blur )
        MyHandleError(kOutOfMemory);
 
    blur.sigma = blurRadius;
    // Defaults are okay here for other MPSKernel properties (clipRect, origin, edgeMode).
 
    // Attempt to do the work in place.  Since we provided a copyAllocator as an out-of-place
    // fallback, we don’t need to check to see if it succeeded or not.
    // See the "Minimal MPSCopyAllocator Implementation" code listing for a sample myAllocator.
    [blur encodeToCommandBuffer: commandBuffer inPlaceTexture: inTexture copyAllocator: myAllocator];
    [blur release];
 
    // The usual Metal enqueue process.
    [buffer waitUntilCompleted];
 
    return result;
}

Image Filters

Overview

In-Place Operation

Supported Pixel Formats for Image Kernels

Sample Code

Topics

Morphological Image Filters

Convolution Image Filters

Histogram Image Filters

Image Threshold Filters

Image Integral Filters

Image Manipulation Filters

Image Statistics Filters

Image Reduction Filters

Image Arithmetic Filters

Euclidean Distance Transform Filter

Fast Guided Filter

Keypoints

Image Filter Base Classes