Camera and Photo Technologies Group Lab

Transcript

Engineers from the camera capture, Core Image/Core Graphics, and Photos frameworks teams introduce themselves and set up a session spanning AVCapture, computational photography, RAW and ProRAW, gain maps, depth, the Photos framework, and metadata.

Yes. The file's IPTC metadata (alongside EXIF) is updated to reflect which AI modification was applied — Spatial Reframe or Clean Up — so the edit is discoverable from the metadata. In the Photos app, this information is also surfaced in the Info panel when you swipe up on the image.

Keywords (long available on Mac) are now editable and searchable in the iOS Photos app Info panel, and they're reflected in IPTC metadata on export. However, there's currently no PhotoKit-level API to fetch or query assets by keyword.

On the Core Graphics side, open images with the thumbnail option so it reuses an embedded thumbnail if present, or decodes and downscales as fast as possible. On the Core Image side, request a scale factor when opening images so you decode only what your UI needs rather than the full-resolution image.

Deferred Start just pushes initialization until after preview is running, so on its own you could still miss an early shot. Pair it with the photo output's responsiveCaptureEnabled property: the system then buffers so you can launch quickly and still capture the moment even before the output is fully initialized. Covered in the "Build a responsive camera app that launches quickly" session.

If you only need a live depth effect in the preview, there's a shortcut: turn on the Cinematic Video Capture API (introduced last year) with a video preview layer to get the cinematic-mode effect for free. If you want the actual depth samples and to render it yourself, enable depth data output on the preview stream and use the AVCaptureDataOutputSynchronizer to align depth and video at the same timestamp in one callback, then combine them with a Core Image filter. For the still, enable depth data delivery on the photo output and set depthDataDeliveryEnabled to true in the photo settings, then apply a depth-based blur in Core Image.

With both .quality and .balanced, the system runs Fusion-style processing that captures multiple exposures (e.g. an underexposed and a normal frame), so it can't honor manual controls or predict a single final exposure ahead of time. Manual exposure and ISO are only supported when you capture with .speed prioritization.

It's a new API for an existing feature. When you import RAW+JPEG (or compressed-inside-RAW) assets, Photos lets you choose whether edits apply to the RAW or the JPEG, and lets users swap between them. OriginalResourceChoice expresses which of those resources is treated as the original.

The way to get scene-referred linear data in camera capture is through the Log formats — Log and the wider-gamut Log2 introduced last year. There isn't a separate untone-mapped Bayer preview path that matches a linear CIRAWFilter DNG conversion.

For an app whose experience lives and dies by capture, the throughline is performance and reliability — users don't like to wait, and a missed moment can't be recovered. The top three: fast launch, responsive capture, and deferred processing. Beyond that: don't start from scratch — use the sample code, and never run AVCaptureSession on the main thread (use a dedicated serial queue) so lengthy operations don't block the UI. Tailor the experience to who your users actually are, since an optimal capture app for social video differs greatly from a pro photography one. Finally, plan your PhotoKit integration and permissions early, since captured photos need somewhere to go.

Take A/V sync seriously rather than relying on coincidental alignment. On modern iPhones, video and audio share the same clock; if you don't handle sync explicitly, an app that seems fine can drift (for example on iPad with an external camera). Use AVCaptureDataOutputSynchronizer and carry the sample buffers' presentation timestamps through your pipeline.

Bayer RAW stores the sensor's Bayer data directly, so a Quad Bayer sensor's native RAW is the binned output. ProRAW goes through a demosaic and Photonic Engine merge of multiple frames, producing already-linearized RGB that no longer depends on the sensor's mosaic pattern — which is what lets ProRAW deliver full 48 MP where native Bayer RAW cannot.

Two frameworks handle ISO gain-map data — Core Graphics and Core Image — and a WWDC session from two years ago drills into exactly this, showing the APIs and how to prepare inputs to produce gain-map-compatible outputs.

Think of Deferred Start as simply moving initialization from before preview to after preview to speed up launch. You can still call setPreparedPhotoSettingsArray to warm the photo output after preview is running — the two aren't in conflict; use deferral to get preview up fast, then prepare the output afterward.

It uses a PHAssetChangeRequest. The new Rating property is set per asset via a new enum with values of unset and one through five, so you modify a PHAsset's rating through that change request.

Check that you're specifying the correct UT Type when setting up the Transferable, rather than leaving it as a default image type — that's a common cause of getting PNG. Sample code for PhotosPicker on developer.apple.com covers this. Note the picker can also convert in some cases, e.g. when the user has restricted metadata or location access for your app.

There's no API for this — the adjustment plist can't be decoded, and there's no supported path to re-import those adjustments. If it would help your app, file a Feedback Assistant request.

The most common mistake is running AVCaptureSession work on the main thread. It's designed to be driven from a dedicated background serial queue and will block and wait during graph reconfiguration — calling start/configure on the main thread freezes the UI and produces stutters. Follow the documented threading model and keep session work off the main thread.

Use Core Image, which is built for interactive UI movement like zoom and pan and caches work throughout the decode pipeline. There's a documentation on the developer portal showing how to set up a CIFilter for opening images so you can pan and zoom smoothly up to full resolution.

On-device use is unlimited, but those captures don't go to the normal photo library — they're saved to the Siri AI app at screen resolution and aspect ratio, not full photo-mode quality. Don't rely on it for your best-looking photos; it's meant for having a conversation with Siri about what's captured, and there may be a limit on the number of conversations.

There's no developer API for the Photos Reframe capability. For related work, parts of ARKit support 3D scene capture, reconstruction, and manipulation, and the Pro phones' LiDAR depth-sensing camera is available as an AVCaptureDevice — but none of these is an out-of-the-box equivalent of the Photos spatial reframe edit.

Orientation is genuinely hard because a device's cameras are physically mounted differently (some portrait, some landscape) and vary across devices. The AVCaptureDevice.RotationCoordinator APIs are the supported way to handle orientation more consistently rather than hand-tuning transforms per app.

Yes. 24-megapixel processing is time-consuming enough that you must opt into deferred processing, and you should opt into the maximum photo dimensions so the selected device actually supports 24 MP. With those in place, if you select quality prioritization and enable depth data delivery, you'll get 24-megapixel captures with depth. This was covered in one of this year's sessions on capturing high-resolution images.

Yes — these labs are shot on iPhone. For multi-camera work, two features from last year stand out. Locked frame duration guarantees an exact frame rate (29.97, 24, 25, 60) and keeps audio synchronized to it — more reliable than matching min and max frame rate. Genlock builds on that, letting multiple iPhones synchronize to an external sync source (via a Blackmagic Pro Dock or genlock generator) so recordings share one timeline with no tearing. Timecode generation lets you bring timecode from an external source in through your capture session and store it as a timecode track for easier post. For writing the resulting ProRes data, the AV Pro Video Storage API gives deterministic file-write speeds to avoid dropped frames.

Two picks. The AV Pro Video Storage API is a strong performance optimization for anyone capturing ProRes — it gives you a pre-allocated file on disk so that even on older phones with fragmented storage you won't drop frames, with a settings UI that lets users control how much space to dedicate; it's also being retrofitted to older phones. The other is the new RAW 9 engine in this year's OS — an ML-based solution for processing RAW files, including from third-party cameras. It's covered in depth in David Hayward's session on enhanced RAW image processing with Core Image, which compares RAW 8 and RAW 9.