VisionKit
Identify and extract information in the environment using the device’s camera, or in images that your app displays.
Overview
VisionKit analyzes pixel information and isolates important data such as text of a given language, URLs, street addresses, phone numbers, shipment tracking numbers, flight numbers, dates, times, durations, and barcodes of various formats. The framework provides this analysis to your app through user interfaces your app displays, which enable people to interact with the analyzed data (ImageAnalyzer.AnalysisTypes) and return the data of interest back to your app. In the interfaces, people can highlight, tap to focus, copy and extract data to the clipboard, or invoke a menu option that runs an app-defined action. VisionKit offers the following user interfaces:
DataScannerViewController presents a camera pass-through view that enables the user to interact with any of the recognized content types (DataScannerViewController.RecognizedDataType) as seen in the environment, and provides captured information to the app for processing.
The Image Analysis interface (ImageAnalysisInteraction on iOS, and ImageAnalysisOverlayView on macOS) displays on top of an image and enables people to interact with content types (ImageAnalysisInteraction.InteractionTypes) that the framework recognizes in the image. For example, the Live Text interface enables them to select any text present in the image (textSelection), or invoke a URL (dataDetectors). Also, the text selection UI offers framework-standard buttons for copying selected text, or looking up the subject on the web for more information.
[Image]
VisionKit’s Document Camera view controller (VNDocumentCameraViewController) is a camera pass-through experience that enables users to scan physical documents. The user scans the document page by page by tapping a camera interface in the view, which provides your app with the resulting images by page number after the scan completes. With the collection of scanned images, your app can create a digital version of the physical document, such as by exporting the scanned images to PDF.
Interact with image subjects
In iOS 17 and macOS 14 and later, VisionKit identifies subjects within an image (see ImageAnalysisInteraction.Subject). A subject may be the focal point of a picture, such as an object around which a photograph centers. Or, the framework may identify several objects that it recognizes in an image. VisionKit enables your app to extract, or lift, subjects to a separate image with the background removed (see image), or present a button that gives more information on the subject (visualLookUp).
Topics
Content recognition and interaction in images
Enabling Live Text interactions with imagesImageAnalyzerImageAnalysisImageAnalysisInteractionImageAnalysisInteractionDelegateImageAnalysisOverlayViewImageAnalysisOverlayViewDelegateCameraRegionView
Barcode and text scanning through the camera
Scanning data with the cameraDataScannerViewControllerDataScannerViewControllerDelegateRecognizedItem