Transcript
[ Music ]
>> Hi. I'm Jacob.
I'm here to tell you about the
Semantic Segmentation Mattes.
So, first, I'm going to go
through what are these new types
of mattes.
And then, David is going to talk
you through how to leverage a
core image to work with these
new mattes.
So, remember in iOS 12, we
introduced the Portrait Effect
Matte?
So, this was a matte designed
explicitly to provide effects
for portraits.
So, we use it internally to
render beautiful looking
portrait mode photos and
portrait lighting photos.
So, in taking a closer look at
the Portrait effect in matte,
you can see how that clearly
delineates the foreground
subject from the background.
So, this is beautifully
represented here as a black and,
black and white matte.
So, values of 1 indicating
foreground and values of zero
indicating background.
In iOS 13, we are taking this a
step further with Semantic
Segmentation Mattes.
So, we're introducing hair,
skin,
and teeth.
So, taking a closer look at the
hair matte, for instance, you
can see how this is beautifully
separating the hair region from
the non-hair regions.
So, we get great hair details
against the background.
And we get great separation
between the non-hair regions
and-- and the hair.
Similarly, for the skin regions
where now we have alpha values
indicating how much of a pixel
is of type skin.
So, so an alpha value of .7, for
instance, would indicate that,
that a pixel is 70% of type
skin.
So, we hope these new three
types, three new types of mattes
will give you the creative
freedom to, to render some cool
effects and beautiful looking
photos.
So, a few things to note is that
the mattes are half size of the
original image.
That means they are half in each
dimension of the original image.
And that means quarter
resolution.
So, another thing to remember is
that-- that these segmentation
mattes can actually overlap.
So, this is particularly true
for the Portrait Effects Matte
and the Skin Matte that will
inherently overlap.
So, these mattes do not come for
free.
So, we heavily leveraged the
Apple Neural Engines for machine
learning spectral graph theory.
And looking a bit under the hood
what we do is we take the
original size image.
We feed it through the Apple
Neural Engine and together with
the original sized image, we, we
render these high-resolution,
high-quality with high
consistency segmentation mattes.
These are then ready to be
embedded into the HEIF or JPG
files as you know them, together
with the original sized image
and the Depth, as you know, from
iOS 11.
So, there are two distinct ways
to generate these two types of
mattes.
So, so one is that they're
embedded in all Portrait Mode
captures.
So, you can grab them from those
files.
Or even better, you can write
your won capture app and opt
into these mattes on capture.
So, if you have files with the
segmentation mattes in them, you
can work with them through Core
Image and Image I/O.
David's going to talk more about
that.
But first, I'm going to talk you
through how to capture with AV
Foundation API.
There are four phases we're
going to go through here that
relates to the extension.
So, first, is when we set up the
AVCapturePhotoOutput.
So, again, is when the capture
request is being initiated and
any point in the lifecycle of
your app.
Then, two of the callbacks.
So, one is when the settings are
resolved for your capture.
And the final one is the when
the Photo did finish processing.
So, for, for full details on
this, please, refer to Brad's
2017 talk on this exact topic.
Let's go through how to set up
AVCapturePhotoOutput.
So, this usually happens when
you're, you're setting, you're
configuring your session.
So, you've already at this point
on session dot begin
configuration, you've set your
presets.
You've added your device inputs,
you add your
AVCapturePhotoOutput.
At this point is when you, when
you tell the API what superset
of segmentation mattes are you
going to ask for at any point in
the lifecycle of your app.
When you actually want to
initiate your capture request,
you need to specify the
AVCapturePhoto setting.
So, this, this is where you tell
the API this is what I really
want in this particular capture.
So, here again, you can specify
all the ones that you already
enabled.
Or you can specify a subset, say
hair or skin.
Now, you initiate your capture
request.
So, you give it the
AVCapturePhoto settings and you
give it the delegate where you
want to have your callbacks.
So, time passes and soon after
you get that, get a
willBeginCaptureFor callback.
This is when, this is when the
API tells you, you may have
asked for something but this is
what you're actually going to
get.
So, this is important for the
Portrait Effects Matte and the
Semantic Segmentation Mattes
because if there are no people
in the scene, you'll actually
not get a matte here.
So, you need to check for the
dimension of the-- dimensions of
the Semantic Segmentation
Mattes.
They'll be zero, in such case.
More time passes.
The photo did finish processing.
So, this is when you get your
AV-- AVSemanticSegmentationMatte
back.
This is the variable matte, in
this case.
So, this new class have the same
type of methods and properties
as you know from the Portrait
Effects Matte.
That means you can rotate
according to Exif data.
You can get your CVPixelBuffer
reference or you can get a
dictionary representation for
each file I/O.
So, for a full walkthrough of
the lifecycle of, of how to make
these captures, please refer to
the AVCam sample app.
It has been updated with the
Semantic Segmentation Mattes and
will take you through all these
different steps.
I'm going to hand it over to
David who is going to talk about
the Core Image.
[ Applause ]
>> All right.
Thank you very much.
Now that we've learned how to
capture images with Semantic
Segmentation Mattes, we get to
have some fun and learn how we
can leverage Core Image to apply
some fun effects.
Now, I'm going to have a demo
next.
But I should warn you if--
there's clowns in this image.
So, if you have any coulrophobia
or irrational fear of clowns,
you know, avert your eyes.
All right.
So, here we have an image that
was captured in Portrait Mode on
a device.
And what we can see in this
application is that we can now
very easily see all the
different Semantic Segmentation
Mattes that are present in this
file.
We can use the-- the traditional
Portrait Effects Matte or we can
also see the Skin Matte or we
can see-- the Hair Matte or the
Teeth Matte.
And it's also possible to use
Core Image to combine these
various mattes into other
mattes, such as this one I've
synthesized by using logical
operations to create a matte of
just eyes and mouth.
If we go back to the main image,
however, we see this is a
picture of me in Apple Park.
And one of the great things you
could do with Semantic-- with
Portrait Effects Mattes is you
could add a background very
easily.
As you can see here, we can put
me in a circus tent.
And while that really does look
like a circus tent, I don't look
like I fit in.
So, now we can use some fun
effects.
For example, we can make it look
like I've got some clown makeup
on.
Or if we want to go a little
further, we can give myself some
green hair.
And lastly, we can use some of
these other mattes to give
myself some makeup.
So, that's what I'd like to talk
to you about today, is how we
can do these kinds of fun
effects in your application.
[ Applause ]
All right.
So, most of the clown references
are gone now so it's safe to
look back.
All right.
So, we're going to be talking
about three things today.
One is how you create matte
images using Core Image, how you
can apply filters to those
images, and lastly, how you can
save these into files.
So, firstly, let's talk about
creating matte images using Core
Image.
There are two ways.
One if you can create matte
images by using the
AVCapturePhoto APIs.
And then, from that, you can
create a Core Image.
So, the code for this is very
simple.
What we're going to be doing is
using the Semantic Segmentation
Matte API and specifying that we
want either the hair or the skin
or the teeth.
And that returns an
AVSemanticSegmentationMatte
object.
And from that, it's trivial to
create a CIImage or we can just
substantiate a CIImage from that
object.
The other common way you're
going to want to create matte
images is by loading them from a
HEIF or a JPG file.
These files have a main image
which you're familiar with, a
typical RGB image.
But they also have auxiliary
images such as the Portrait
Effects Matte as well as the new
mattes that we're talking about,
the Skin Segmentation Matte and
the Hair and the Teeth.
The code for this is very
simple.
The traditional code to create a
CIImage from a HEIF file is just
to say CIImage and specify a
URL.
To create these auxiliary
images, all you do is make the
same call and provide an options
dictionary specifying which
matte image you want to return.
So, we can specify the
auxiliarySegmentationHairMatte.
Or if we want, we can get the
mattes for the other Semantic
Segmentations.
So, very simple.
Just couple lines of code.
The next thing we want to do is
talk about how you can apply
effects to these images.
So, I showed a bunch of effects.
I'm going to talk about one in a
little bit of detail.
What we're going to do is we're
going to start with a base RGB
image.
And then, we're going to apply
some effect to that.
Let's say we want to do the
washed-out clown white makeup.
So, I'm going to apply some
adjustments to that.
Those adjustments, however,
apply to the entire image so we
want those to be limited to just
the skin area.
So, we're going to use the Skin
Matte and then, we're going to
combine these three images to
produce the result we want.
Let me walk you through the code
for it because it's actually
quite simple.
But first, I want to talk about
the top feature request we've
had for Core Image which is to
make it easier for people to
discover and use the 200 plus
built-in filters we have.
And that is the new header
called Core mage
CIFilterBuiltins.
And these allow you to use all
of the built-in filters without
having to remember the names of
the filters for the names of the
input.
So.
[ Applause ]
It's-- it's really great.
So, let me show you some code
that will use this new header.
So, the first thing we're going
to do is create the base image.
And we're just going to call it
image with contents of URL.
And that will produce the
traditional RGB image.
Now, we're going to start
applying some effects.
So, the first thing I want to do
is I'm going to convert it to
grayscale and I'm going to use a
filter called the
maximumComponent.
And I'm going to give that
filter an input image of the
base image.
And then, I'm going to ask for
that filter's output.
And that produces the image that
looks grayscale like this.
This doesn't look quite bright
enough to look like clown
makeup.
So, we're going to apply an
additional filter.
We're going to say use the gamma
adjustment filter.
And the input to this will be
the previous filter's output.
And then, we're going to specify
the power for the gamma function
and ask for the output image.
And you'll notice it's now very
easy to specify the power for
the gamma filter.
It's a Float rather than having
to remember to use NSNumber.
So, that's the first part of our
effect.
The next thing we want to do is
start by getting this Skin
Segmentation Matte.
So, again, as I described
earlier, we're going to start
with the URL to specify that we
want the Skin Matte.
However, when we get this image,
notice it's smaller than the
other image.
As we mentioned before, these
are half size by default.
So, we need to scale it up to
match the image, main image
size.
So, we're going to create a
CGAffineTransform that scales
from the matte's size to the
base image size.
And then, we're going to apply a
transform to the image.
And that produces a new image
which, as you expect, matches
the correct size.
The next step we're going to do
is start combining these two.
And we're going to use the
blendWithMask filter.
And this is great and we use
this throughout the sample I
just showed.
We're going to specify the
background image to be the base
RGB image which looks like this.
Next, we're going to specify the
input image which will be the
foreground image which is the
image which has the-- the white
makeup applied.
And lastly, we're going to
specify a mask image which is
the image that I showed
previously.
Given these three inputs, you
can ask the blend filter for its
output and the result looks like
this.
Now, as you can see, this is
just the starting point.
You can combine all sort of
interesting effects to produce
great results in your
application.
Once you're done applying these
effects, you want to save them.
And, most typically, you want to
save them as a HEIF or a JPG
file which supports saving
auxiliary images, as well.
So, in addition to the main
image, you can also store the
Semantic Segmentation Mattes so
that either your application or
other applications can apply
additional effects.
The code for this is very
simple.
You use the Core Image API
writeHEIFRepresentation.
And typically, you specify the
main image, the URL that you
want to save it to, and then the
pixel format that you want it to
be saved as, and the colorSpace
you want it to be saved as.
And what I want to highlight
today is another set of options
that you can provide when you're
saving the image.
So, for example, you can specify
the key
semanticSegmentationSkinMatte
and specify the skinImage or the
hairImage or the teethImage.
And all four of these images
will be saved into the resulting
HEIF or JPG file.
Now, there's an alternate way of
getting this result, which is,
if you want, you can save the
main images and specify the
segmentation mattes as
AVSemanticSegmentationMatte
objects.
This again, the API is very
simple.
You specify the-- the URL, the
primary image, the pixel format,
and the color space.
In this case, if you want to
specify these objects to be
saved in the file, you just say
AVSemanticSegmentationMattes and
you provide an array of mattes.
So, that's what you can do using
Core Image with these mattes.
What I talked about today is how
to create images for-- for
mattes, how to apply filters,
and how to save them.
I will, however, mention that
the sample app I showed you has
been written as a Photos App
plug-in.
And if you want to learn about
how you can do that in your
application so that you can save
these images, not just to HEIFs
but, also, into the user's Photo
library, I recommend you consult
these earlier presentations.
Especially, the "Introduction to
the Photos Frameworks" from WWDC
in 2014.
All right.
And thank you all, very much, I
really look forward to seeing
what you do with these great
features.
Thanks.
[ Applause ]