WWDC2019 Session 506

Transcript

[ Music ]
>> Hello everyone.
My name is Shiva Sundar, and I'm
extremely excited to talk about
HEVC Video with Alpha.
You're probably familiar with
Alpha channels in image files today.
Alpha channels are ubiquitous
in image files, in our apps and on
the web.
When there's an Alpha channel
in an image file, every pixel in the
image gets opacity along with
its color.
This allows us to have an image
that has a complex,
non-rectangular shape but looks
natural when composited against
many different backgrounds.
So Alpha and transparency are
common and used all over the web
principally by still image
formats like PNG.
Alpha in video has been less
common.
It has been a professional video
workflow tool.
The formats used in professional
video workflows like Apple
ProRes 4444 are very high data
rate and not appropriate for
delivery.
In 2017, Apple introduced
support for the HEIF image file
format, which lets us take
advantage of the advanced
lossy compression
technologies in HEVC to compress
still images.
HEIF also lets you include an
Alpha channel in image files.
In fact, this year we are
adding support for HEIF
sequences, which lets you have
multiple images in a single
file.
HEIF sequences also support
Alpha channels.
For some time we've recognized
the need for a lower bitrate way
to include Alpha in motion
video.
And so we said let's use that
advanced loosy compression
technology to support Alpha
channels in video files as well.
Playback of HEVC video with
Alpha is supported in iOS 13,
tvOS 13 and macOS Catalina on
all supported devices and
computers.
Encoding is supported on all of
those devices and computers that
have an HEVC encoder.
Having a video with Alpha format
that works well at bit rates low
enough for distribution opens up
many opportunities.
You can have rich,
non-rectangular motion video
content that you incorporate as
a layer as part of your app's
expressive experience.
It could be pre-rendered like an
animation.
Such an element can move around
on top of a complicated
background and you could even have
several of them to give a
feeling of depth or complex
interactions.
Because these are movies they
can have synchronized sound
tracks so the characters in
these non-rectangular movies can
talk.
>> Highly professional.
>> Alpha channel.
>> Content.
>> For you.
>> Woo-hoo.
>> They could be integrated into
games using SpriteKit, SceneKit
or Metal.
In this context, we have
provided you with some sample
code on how to record a SceneKit
scene into a HEVC video with
Alpha and how to render another
in a SpriteKit scene.
On web pages, they could bring
interactive elements to life.
HEVC video with Alpha is
supported by Safari in iOS 13
and macOS Catalina.
Safari can also tell the web
page whether this feature is
supported.
For more information on using
the media capabilities API, be
sure to check out the What's New
in Authentication, Safari and
WebKit video.
The ability to distribute video
with Alpha at low bitrates opens
opportunities for authoring apps
to provide background removal.
Here I am in front of a green
sheet hung from the wall
recorded using an iPhone.
I wish I was here.
I used a custom video compositor
with a co-image filter to
replace the green pixels with
transparency and exported it to
a video with Alpha.
I wish I was here.
And now I can overlay this video
on any background I like.
I wish I was here.
We are providing sample code
showing how to do this and write
the resulting HEVC video with
Alpha movie.
So how does this work?
As you know, movie files contain
tracks of various media types.
A video track contains video
frames.
HEVC video with Alpha is
contained inside a single video
track in the movie.
In the single track, each frame
contains 2 parts called layers
and each layer is compressed
using the HEVC codec.
The base layer contains the
colored image of the frame as
usual.
The Alpha layer contains just
the Alpha channel of the frame.
This is done using special HEVC
syntax which identifies the 2
layers.
The HEVC rules for this syntax
mean that the HEVC decoder that
does not know how to handle the
Alpha layer will ignore it and
display the base layer alone.
Let's move along and talk about
how to use it.
HEVC with Alpha encoding is
integrated into numerous AV
Foundation APIs.
Let's look at content creation,
for example.
You can render video frames with
Alpha in your application and
use AVAssetWriter to encode them
and write them to a movie file.
You can use new presets
available with
AVAssetExportSession to
transcode video formats
containing Alpha including Apple
ProRes 4444 into HEVC with
Alpha.
These new HEVC presets have the
suffix with Alpha.
You can also use
AVAssettExportSession to burn
in a specific background color
converting a movie with Alpha
into a movie without Alpha, that
will play on players that don't
support HEVC with Alpha.
Or you can also use the HEVC
with Alpha video encoder
directly via
VTCompressionSession.
Looking at AVFoundation's
playback APIs, HEVC with Alpha
is supported by AVPlayer and
AVPlayerLayer for display in
your apps view hierarchy via
Core Animation, AppKit and UIKit.
The video in AVPlayerLayer will
be displayed with a transparent
background and composed with the
rest of the layers and views.
Alternatively you can access a
sequence of decoded video frames
directly using
AVPlayerItemVideoOutput.
This is a great choice when you
want to incorporate those video
frames into a custom rendering
you're doing using Metal,
SpriteKit and other GPU-based
rendering APIs.
If you need to decode frames for
another purpose, you can use
AVAssetImageGenerator directory
and individual frame of the
movie as a CGImage with Alpha.
You can use AVAssetReader to
extract video frames for use in
a non-playback workflow.
You can also use the HEVC with
Alpha video decoder directly via
VTDecompressionSession.
In short because HEVC with Alpha
is integrated into AVFoundation
at the level of a video codec,
it fits in well into all these
familiar API workflows.
Now let us see how we can
accomplish this in code.
When encoding a movie, when
configuring your AVAssetWriter
input or VTCompressionSession,
use the video codec type HEVC
with Alpha.
This is a special signal that
the Alpha channel should be
encoded.
The encoded file will have the
standard HEVC codec type.
It will be backwards compatible.
An HEVC player that does not
understand Alpha will display
only the base video layer.
It will ignore the Alpha
portion.
If you work with Alpha channels,
you may know that there are 2
strategies for how to write
video data to a file,
premultiplied Alpha or straight
Alpha, also known as
unassociated Alpha or
non-premultiplied Alpha.
The tradeoffs between these
choices are beyond the scope of
this talk.
If you're familiar with these,
what you need to know is that
both are supported.
If you're not similar with
these, we recommend the default,
which is premultiplied Alpha.
It's the natural choice in most
GPU-based rendering parts and it
scales better.
Now looking at the code, you may
specify the Alpha mode by either
configuring the compression
session or as a buffer
attachment.
As a safety mechanism if the
setting and the buffer
attachment are both specified
and they do not match, the
encoding operation will fail.
Our APIs require the
Alpha channel to be encoded with
fixed quality because video
encoding artifacts in the Alpha
channel can be bothersome in
ways that the same artifacts
wouldn't be in the base layer.
Our APIs let you
control the base layers bitrate
and the Alpha channel's quality
separately if you need to.
Now, looking at the code you can
specify the Alpha layer quality
using target quality for Alpha
parameter.
It is arranged between 0 and 1
where 1 means it is close to
lossless.
The bitrate parameter you
specify only applies to the base
layer and is ignored by the
Alpha layer.
To detect the presence of an
Alpha channel, you can either
test AV media characteristic
containsAlphaChannel or you
can query the format description
extension contains Alpha
channel.
If you are trying to export HEVC
with Alpha movie, use to
determine compatibility API to
validate that Alpha is present
in the source asset.
This validation only takes the
source movie into account not
the video composition.
So you must skip it if you are
using a video composition to
generate a new Alpha channel
from sources that do not have
Alpha channels.
There are a lot of video players
in the ecosystem.
Many of them don't support HEVC
let alone HEVC with Alpha.
You can remove the Alpha channel
by burning in a solid background
color.
We have added some convenience
API and sample code to help
simplify this conversion.
If you're looking to build
support into a non-Apple media
framework, we have published a
specification for an
interoperability profile that
you should read, but if you
can't wait, here is a sneak
peek.
To summarize, HEVC video with
Alpha is a cool new feature.
It takes advantage of HEVC's
state of the art compression
technology.
It is supported on iOS 13, tvOS
13 and macOS Catalina.
It takes advantage of hardware
acceleration on recent devices.
It is well integrated into
AVFoundation for use in your apps.
It is also integrated in Safari
for use in your web experiences.
HEVC with Alpha brings us new
opportunities for creative
expression for both developers
and users alike.
We can't wait to see the ways
you can take advantage of it in
your apps.
Thank you for your time.