WWDC2014 Session 514

Transcript

X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
>> My name is David Hayward
and welcome to our first
of two discussions
about Core Image.
And we'll be talking
today about what's new
in Core Image on
both iOS and OS X.
So what is Core Image?
Core Image is a fast,
easy, flexible framework
for doing image processing.
And it supports all
of our supported devices
on both iOS and OS X.
It's also used by several of our
key applications such as photos
on both platforms
and it allows you
to get very good
performance results
and very flexible output.
For those of you who may
be new to Core Image,
I just want to take a few
slides to talk about some
of the key concepts because
those will be relevant
for the rest of the
discussion today.
So first off, filters --
Core Image filters allow you
to perform per-pixel
operations on an image.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to perform per-pixel
operations on an image.
In a simple example, you have
an original image and you want
to apply a sepia
tone filter to it.
And you'll get a
resulting image.
Obviously, that's fun
but where things start
to get interesting is where
you combine multiple filters
in either chains
or complex graphs.
And here an example, you can see
a very interesting result you
can get by just chaining
three filters together,
sepia tone plus a hue
rotation to turn it
into a blue tone
image plus a contrast
to make it more dramatic.
One thing to keep in mind is
these intermediate images are
actually lightweight objects.
So there need not necessarily
even be memory associated
with these of any
significant amount.
Another thing that's
important to keep
in mind is each filter may have
one or more kernels associated.
So these kernels are
the actual algorithm
that implements each
filter's effect.
And one of the great things
about Core Image is we
can take these kernels
and concatenate them into
programs and this allows us
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and concatenate them into
programs and this allows us
at runtime to minimize the
amount of intermediate results
and with some great
compiler technology,
both at the image processing and
at the low-level compiler level,
we're able to get the
best possible performance
out of a complex graph.
So that's the basics in
terms of how it works.
These are the four key
object types that you need
to be familiar with if you
want to use Core Image.
The first which we'll be talking
about a lot today is CIKernel.
And this represents a
program that's written in CI's
or Core Image's Kernel language.
Second object type is a
CIFilter and this is an object
that has mutable
input parameters
and those parameters
can be images or numbers
or vectors or other types.
And it also allows you to
use one or more kernels
to create a new image based on
the current state of the output
or of the input parameters.
Third key type is a CIImage
and this is an immutable object
that represents the
recipe to produce an image.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Just the act of creating an
image does not necessarily do
any real work.
The actual work occurs
when you render a CIImage
into a CIContext and
that's the object
through which you
render results.
So those are the basics.
What I want to talk
about today is what's new
in Core Image this year and
we have a lot to talk about.
We have a bunch of new
things that are on iOS.
For example, we have our
most requested feature
which is Custom CIKernels.
We also like to talk about
how you can do Photo Editing
Extensions using Core Image
and also how we can now
support large images on iOS.
We also made some improvements
to how the GPU render is used.
We also have some
API modernization.
We have some new
built-in filters.
We have some new CIDetectors
and then lastly, we will talk
about some new things that
we have on the Mac OS X side,
improve RAW support and
how to use a second GPU.
So first and most interesting,
I think is Custom
CIKernels on iOS.
As I mentioned, this has been
our top requested feature.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
As I mentioned, this has been
our top requested feature.
Core Image already has 115
great built-in filters on iOS.
But now you can easily
create your own.
So this is a terrific
feature for developers.
When you're writing
CIKernels on iOS,
you can use the same
CIKernel language
that you use today on OS X.
There are a few extensions
which allow making
typical kernels even easier
and we'll talk about
that in much more detail
in our next presentation.
Where can your CIKernels live?
Well, they can live
in your application.
The kernel code can
either be a text resource
or it can just be an
NSString, if you'd like.
The kernel is wrapped
up in CIFilter subclass
that you provide that
applies to kernels
to produce an output image.
Another great place for
your Custom CIKernels
to go is inside an
App Extension.
For example, Photo Editing
Extensions can use CIKernels
and CIFilter subclasses
very effectively.
And you can use them to modify
either photos or videos.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And you can use them to modify
either photos or videos.
So again, we'll be talking
our next presentation
in much more detail about
how to use CIKernels on iOS
but let me just give
you a little teaser now
of how simple it is.
Here we have in just
basically two lines of code,
how to use a Custom CIKernel.
We create an NSString which has
some CI Core Image source code
in it.
This is a very simple kernel
that takes a pixel
value and inverts it.
You'll notice it's actually
subtracting it not from 1
but from the alpha value.
That's the correct way to invert
if you've got premultiplied data
which is what Core
Image receives.
Once you have the program
written then all you need
to do is create a
CIKernel object
from the string and
then apply it.
You can specify two things.
One is the resulting
extent of the produced image
and also the arguments that
will be passed to that kernel.
In this particular example,
there is only a single argument
which is the input image and
so as a result, our arguments
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
which is the input image and
so as a result, our arguments
down below is just an array
with a single image in it.
So to give you a little bit
of idea of what that looks
like in practice, I
have a quick demo.
This is a fun example
that we wrote.
And it's kind of an
example of something
that you wouldn't necessarily
have as a built-in filter.
Let's see.
But it would be fun for
a presentation like this.
So we have an application
called Core Image Funhouse
and this allows you to explore
all the built-in filters
and also allows you to see
some sample code for how
to write Custom CIKernels.
So the image starts out as gray.
The first thing we need to do is
provide an image to start with.
And we're going to say that we
want the video feed to come in.
And then I'm going to add
a filter and you can see,
we're seeing a list
of all the filters
that are part of Core Image.
And we created one down
here called WWDC 2014
and I hope you can see this
so that I can kind of wave
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and I hope you can see this
so that I can kind of wave
in front of the camera.
What we're doing here is
actually algorithmically taking
the luminance from the
video feed and then using
that to control the size
of the geometrically
generated rounded rectangle.
And we can change the size
of that larger or smaller
or we can change the amount
of the rounded radius here.
It's actually a little easier
to see that it's a video feed
when it's smaller but it looks
more cool when it's bigger.
So and we're getting about
30 frames per second on that
which is probably the frame
rate of the camera right now.
So that's our short example and
we'll have that code available
for download at some point soon.
So again, that's Custom
CIKernels and please come
to our second session to see
all you can learn about that.
The next thing I'd like to talk
about briefly is the Photo
Editing Extensions on iOS.
There are whole talks on
that this year at WWDC.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
There are whole talks on
that this year at WWDC.
I'd like to talk a little
bit about how that works
in relationship to Core Image.
So here's just a
little short video run
through of how this
works in practice.
What we have is an image
the user wanted to edit.
They brought up a
list of extensions.
We picked the Core Image one
and this particular Core Image
based extension has two sliders.
One is the amount of sepia
tone which we can slide
and we're getting
very good frame rates
to the screen as we do this.
And then the second slider
is a vignette amount.
So it starts out with
a large radius and then
as you bring the radius
smaller, you get more
of the vignette effect
as you bring it down.
And all of this is
happening right now
on a display-sized image.
Later on, when you
hit Save, it's applied
on a full-sized image which
is the 12 megapixel image
in this case.
And it goes back into your
library with your edits.
So that's how it
looks in practice.
I'm not going to go into too
much detail on how to code this
but I'll give you
some good advice here.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
but I'll give you
some good advice here.
First off, you can start to
create a Photo Editing Extension
by going into the
templates in Xcode.
We'll also provide some
sample code as well
so that'll be a good
starting point.
But as I said, I
wanted to talk a bit
about how you can use
Core Image effectively
within Photo Editing Extensions.
There's basically three steps.
The first step is when your
extension is initialized,
what you want to do is you want
to ask the editing input object
for a display-sized image.
Initially, that is a UI
image object and from
that you can create a CGImage
and then from that CIImage.
That sounds like
a couple of steps
but it's actually those are
just lightweight wrappers.
Once you've created that
CIImage, we're going to store
that in a property
for our delegate.
The other thing, it's a good
time to do at that time is
to create your view that you're
going to be rendering into.
We recommend using a GLKView
and also create a CIContext
that is associated
with that view.
And it's good to store that
away in the property as well.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Step two is what you do every
time the user makes an edit
in your extension so every
time the slider moves.
And this is very simple.
What you do is you recall
the display-sized CIImage
that we created in step one.
We apply the filters
that correspond
to those slider adjustments.
So in that previous example,
it was the sepia tone filter
and the vignette effect filter.
And then once you've
chained those together,
you get the output
image from that.
And then you're going to
draw that using the CIContext
that we also created
in step one.
And Step three is what happens
when the user clicks
the Done button.
And this is slightly
different because in this case,
you want to apply the effect
on the full-sized image.
So what we have here is we can
ask the editing input object
for its fullSizeImageURL.
From that, we create a CIImage
and we apply the
filters to this as well.
Now, for the most part, this is
the same as we did in step two.
Some parameters however such as
radiuses may need to be scaled
in accordance to the fact
that you're now working
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
in accordance to the fact
that you're now working
on a full-sized image.
Once you have chained
together your filters,
you ask the output
image and then you --
the way this API works is you
return the CGImage so you can do
that very easily
with Core Image.
You ask a CIContext to create a
CGImage and this will work even
on the full-sized image.
So that brings me to the
next subject I want to talk
about today which is working
on large images on iOS.
So we've made some great
improvements here in addition
to the supporting kernels, this
is I think our second key thing
that we've added
this year on iOS.
So now you can -- we have
full support for images
that are larger than
the GPU texture limits.
And this means that input
images can now be larger than 4K
and output renders
can be larger than 4K.
We refer to this as large
images but in practice,
4K images are not
that large these days.
Many of our devices'
cameras are bigger than that.
So this is actually a
really critical feature
to support this size
image as well.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The way we achieve
this automatically is
that we have automatic
tiling support in Core Image.
And this among other
things leverages some great
improvements that were made
in ImageIO and they're JPEG
to improve how the
decoder and encoder works.
And also, there's
some great features
in the Core Image language
that allows supporting
of large image as well.
So let me talk about that last
item in a little bit of detail.
So the CIKernel language
allows your kernels
to just work automatically
regardless
of whether tiling happens
or at what size it happens.
So this is a great feature
that makes writing
CIKernels very flexible.
The way this is achieved is by
two key extensions that we have
in our language and
these are available both
on OS X and on iOS now.
The first is a function called
desk coordinate or deskCoord
and that allows Core Image
to support tiled
output automatically.
It basically allows your kernel
to see the desk coordinate
in the native images space even
though we may only be rendering
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
in the native images space even
though we may only be rendering
a given tile at a time.
Similarly, there's a function
called samplerTransform
and that allows Core
Image to support tiling
of large input images
automatically.
So this is the two key things
about the CIKernel language
that we'll talk about
in much more detail
in our second presentation.
So another great thing about our
large image support is how we
work together with CGImageRef
and how we get some
great improvements
on iOS 8 by being lazy.
So one thing to keep in mind is
if you have a small
input CGImage
that you create a CIImage from,
then this image is fully decoded
at the time you call
CIImage initWith CGImage.
And that's actually usually
the right thing to do
for small images
because you may be using
that image multiple
times and you want
to take the performance impact
of decoding the JPEG once early.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to take the performance impact
of decoding the JPEG once early.
However, for large images,
that's not a good strategy
because you don't want to
require all of that memory
to be -- for that JPEG
to be compressed unless
you know you need it.
So if you have a large
input CGImage, that image,
that JPEG image behind that
CGImage is decoded only
as needed when you
call CIContext render.
So that's a very
important detail.
Similarly, when you're
producing a CGImage as an output
of CIImage, when you call
CIContext createCGImage,
if the output CGImage is small,
then the image is fully rendered
when CGImage is called.
However, if you're producing
a large CGImage as an output
such as an example of
the photo extensions,
the image is only
rendered as needed
when the CGImage is rendered.
This is also important because
a very common situation is you
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
This is also important because
a very common situation is you
pass the CGImage, the
CGImage DestinationFinalize
to encode it back as a JPEG.
So what all this means is that
if you have a very large JPEG,
you can take that large JPEG,
decode it, apply a filter to it
and re-encode it back into
a JPEG with minimal memory
and great performance
and this is a huge win
for Core Image on iOS.
So let's take a quick example.
You're applying a sepia tone
effect to a 4K by 6K JPEG,
so 100 megabytes of image.
That on iOS 7 took 17 seconds
to decode, apply the filter
and re-encode it as a JPEG.
On iOS 8, that's 1 second.
[ Applause ]
And just as important on iOS
is the memory high water mark
because that can really
force your application
into an unhappy place.
And our high water mark
on iOS 7 was 200 megabytes
which makes sense.
We have a source image that was
fully decompressed and we need
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We have a source image that was
fully decompressed and we need
to produce a whole new image
which is the same size.
However because we
now have tiling,
our high water mark
is now 25 megabytes.
[ Applause ]
And just to summarize, on iOS
7, we worked on the full image
at a time and because it
was large, we often had
to use a CPU renderer.
On iOS 8, we have automatic
tiling and as a result,
we can use the GPU
which is a huge win.
So we've also made
some other improvements
to how GPU rendering
works with Core Image
on iOS which are important.
So your application
sometimes needs
to render in the background.
Often either when the app is
just transitioning to background
or when it's fully in
the background state.
On iOS 7, that is supported.
However all background renders
used the slower Core Image CPU
Rendering path.
On iOS 8, we have an improvement
in this regard which is
that renders that occur
within a short time
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that renders that occur
within a short time
of switching the background will
now use the faster GPU renderer.
Now, it is serviced with a lower
GPU priority and the advantage
to that is that any
foreground renderers that happen
at that time will not be --
have any performance impact
because Core Image will be
using a lower priority renderer.
So this is another
great advantage.
There are some restrictions
on this GPU usage.
It is not allowed if you use
CIContext drawImage inRect
fromRect because in that case,
Core Image needs to render
into the client's
[inaudible] context.
However, any of the other render
methods calling createCGImage
or render toCVPixelBuffer
or render toBitmap will
all work in this way.
Another great improvement we
have is oftentimes you want
to do rendering in the
foreground when your app is
in the foreground but do it
from a secondary thread
in a polite manner.
So if your application
is showing one thing
and then doing something
on a secondary thread
using Core Image, on iOS 7,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
on a secondary thread
using Core Image, on iOS 7,
that required care
in order to avoid --
in order for the secondary
thread to avoid causing glitches
for the foreground thread.
And of course, the only
sure-fire way to avoid that was
to use Core Image's
slower CPU renderer.
On iOS 8, we have a new feature
which is the secondary thread
can now render into a context
that has had this
new option specified
which is CIContext
PriorityRequestLow.
And the idea now is that
context renders using
that context will not
interrupt any foreground higher
priority renders.
So this is also great
for your application.
So this brings me to
some final thoughts
on Core Image's CPU rendering.
Basically, there were three key
reasons why an app would need
to use the CPU renderer
on iOS 7.
For example, the CPU
renderer was used
when GPU texture
limits were exceeded.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Well, starting on iOS 8, that's
no longer a limit in Core Image
so that's not a reason anymore.
Similarly, the application might
have needed to render briefly
when in the background, that's
also been improved in iOS 8.
And lastly, if your
application wanted to render
from a secondary thread
when in the foreground,
you might have used the
CPU renderer and now
that is no longer a limitation.
So we have some great
ways to keep us
on Core Image's much
faster GPU rendering path.
The next subject I want to
talk about this afternoon is
about some API modernizations
that have been made
both on OS X and on iOS.
These are small conveniences
but they add up in total.
First off, Core Image
filter subclasses
on OS X can now use
properties instead of ivars.
One thing to be aware of is
that Core Image filter
subclasses do not need
to release the object associated
with input ivars or properties.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to release the object associated
with input ivars or properties.
So it's a little bit nonstandard
as a class in that regard.
By supporting properties,
that means that code that used
to look like this where you
have output image equals filter
valueForKey kCIOutputImageKey
can now be a little cleaner
and just look like outImage
equals filter.outputImage.
We also have a convenience
method if you want
to create a filter
and also set a bunch
of parameters all
in one fell swoop.
This can be now done by
saying filter, filterWithName
and then you could specify some
parameters at the same time.
And in those parameters
are a dictionary
where you can specify all the
inputs in one convenient manner.
There's an even slightly
simpler case
which is very commonly
usable where one
of your inputs is an input
image and you just want
to get the output of a filter.
So this means you can apply a
filter to an image with a set
of parameters without even
creating a filter object.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of parameters without even
creating a filter object.
Lastly, one of the
common questions we get
from developers is, "How do
I correctly orient my image
so the orientation is
correctly upright?"
And the standard TIFF
specification has a set
of 8 possible values that tell
how the image should be flipped
or rotated and we've provided
a code snippet in the past
for that but much easier
is that we provided an API
for that now in iOS 8 and OS X.
So the simplest way
of calling it is
to say
imageByApplyingOrientation
and that gives you
back a new image.
And again, you're specifying
an integer orientation value.
As an alternative to doing the
same thing, we also have an API
that allows you to get
back the fine transform
that is equivalent to that.
And the reason why that's useful
is usually orienting your image
upright is only the
first of several affines
that you may apply
to your image.
You may also be scaling
it to fit or panning it.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
You may also be scaling
it to fit or panning it.
And so by getting this affine
matrix and concatenating
with any other affine matrix,
you can get a little better
performance out of Core Image.
So we've also made some
modernizations on OS X
with regard to color spaces.
The default RGB color space
is now sRGB which is great
because it matches with
the default RGB color space
that we have on iOS.
It also matches what most
modern applications expect
for untagged images.
Similarly, our default working
space has also changed on OS X.
It is now a linearized version
of the Rec.709 chromaticities
and again, this matches
the default we have
for our working space on iOS
and has a great performance
advantage which means
that in most typical scenarios
where you have sRGB
content going into a filter
in its working space and then
going back to sRGB output,
no matrix math is needed at all
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
no matrix math is needed at all
so this is a great,
great advantage.
Next subject, I'd like to talk
about today is some new
built-in Core Image filters.
So we have several I'd
like to talk about.
One is new to iOS 8 is
we've added CIAreaHistogram
and CIHistogramDisplayFilter.
The first filter,
CIAreaHistogram takes an
input image and the rectangle
that you want to generate
the histogram of it
and it'll produce an output
image that's typically 256
by 1 pixels.
So that image is useful
if you want to render
and get the pixel
values out of it
because that'll give you your
histogram data very efficiently.
However, oftentimes
you also want
to display this histogram
to the user.
So we have a second filter which
is CIHistogramDisplayFilter.
And it takes as an input
this 256 by 1 pixel image
and it produces a
pretty graph with red,
green and blue graphs
in it just like this.
It's very easy to use
in your application.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
It's very easy to use
in your application.
You just chain together
these two filters.
This is another great filter
that I'm really pleased with.
This is -- we've always had
filters for doing Gaussian blurs
on an image but we have a new
filter called MaskVariableBlur.
And the idea is you want
to apply a blur to an image
but you want to apply a
different amount of blur
at different locations.
So the way this filter works is
you start with an input image
and you provide a masked image.
In this example, we
have the mask is white
in the lower left-hand
corner, black in the center
and then white again the
upper right-hand corner.
And what this means when
we combine these two images
with masked variable blur
is we get a resulting image
that is defocused at the corners
and then gradually transitions
to a nice sharp image
in the center.
This is not just done with
blends but it's actually done
with variable radius blurs
which is quite a trick.
So there's a couple of
different ways you can use this.
You can use this to
achieve a sort of fake depth
of field effect where
the top and bottom
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of field effect where
the top and bottom
of your image might be blurry
and the center may be sharp.
Or you can actually hand create
a masked image with a person
in the foreground and then
nicely blur the background
with a nice bokeh.
So I hope to see lots
of fun examples of that.
This is another fun one we added
which is AccordionfoldTransition
and this is something
we did for the mail team
but we've also provided
it as a public filter.
You provide two images, a
before and an after and a couple
of parameters like how many
folds and how many pixels
at the bottom are shared.
And what this filter looks
like in practice is this.
And if you actually
look carefully,
that's the actual entire
kernel for this filter.
So it's a nice bit of trickery.
Another filter we've
added, in prior releases,
we've had filters for
generating QR codes.
We've added a new one for
generating code 128 barcodes
and it works in a
similar fashion.
You specify an input message
as NSData and in this case,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
You specify an input message
as NSData and in this case,
there's an additional parameter
which says how many pixels
of quiet space you want
and it'll produce
an image like this.
We've also added another
one for Aztec codes.
Again the same kind
of idea for the API,
you just specify
the input message
and for this particular
generator,
it has an input correction level
which tells how many error
correction bits it will have.
Another new filter which is also
fun is CIPerspectiveCorrection.
And the idea behind this
is you have an input image
and you specify 4 points and
it will create a new image
that is cropped and undistorted
preserving the original
and intended aspect ratio
so this is again very nice
for capturing parts of an
image and distorting them.
We've added a handful of
new blend filters, linear,
dodge and burn, pin
lights, subtract, divide.
Also just to be aware, we've
made a fix to SoftLightBlendMode
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Also just to be aware, we've
made a fix to SoftLightBlendMode
so it better matches the spec.
And then there's a few other new
ones we've added that are new
on iOS such as GlassDistortion,
StretchCrop for anamorphic
correction,
Droste which is a great demo
from our conference show two
years ago, and then who knows,
if we have some more time,
we'll get a few more in.
But what that brings us to today
is over 115 built-in filters
on iOS and of course, that
really is an infinite number now
that you guys can create
your own custom filters.
So we're excited to see
all sorts of new things.
Another area we've
made some improvements
in Core Image is CIDetectors.
So what is a CIDetector?
Well, CIDetector is an
abstract class that allows you
to help find things
within an image.
And prior to iOS 8, we had just
one type which was TypeFace.
But we've added two more.
So we now have
CIDetectorTypeRectangle
and CIDetectorTypeQRCode.
So how does this work?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So how does this work?
Well, creating a detector is
largely the same regardless
of what type of detector
you are creating.
Here we have an example of
creating a detector of TypeFace
where we say detector,
detector of TypeFace
and we can also specify
some options.
There are a couple of
options that are very useful
for all the detectors.
One is you could say whether
you want to have high accuracy
or low accuracy which depending
on your need might allow you
to trade off performance
versus precision.
Also, you can tell a detector
what the smallest feature
to detect is and that also can
greatly improve performance.
And of course, now that we've
added these new detectors,
you can just use
DetectorTypeRectangle
or DetectorTypeQRCode as well.
So just as a reminder, so when
you're using the FaceDetector,
there's a couple of options
that you want to pass
in when you're asking for the
actual features in an image.
One is you can specify what the
orientation of the image is.
That's important because
the FaceDetector looks
for upright faces.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Also you can specify
options to say I want to look
for eye blinks or smiles
and that's specified
in the same options dictionary.
And let me show you
a little bit of code
about how we can now use this
detector to create a sort
of augmented reality
example here.
And the idea we wanted
for this little bit
of a sample code is we wanted
to start with the input image,
find the faces in it
and then put squares
over the image where
we find them.
And so this is a little
clever bit of sample code.
First off, for each face that
we detect in the features array,
we're going to check to see if
the eyes were closed or not.
Then we're going to
create a CIImage WithColor.
And we're going to have
a different color based
on whether the eyes
are closed or not
or whether face is
smiling or not.
Now that API actually
returns an infinite image
so what we then need to do is
to crop that image to the bounds
of the feature that
was detected.
We then take that cropped
image color and we composite
over the previous
resulting image.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
over the previous
resulting image.
And this is also a new
API that we've provided.
It's basically convenience
API that's equivalent
to using the Core Image source
over compositing filter.
And this is what it
looks like in practice.
Here's a little sample
video we shot
where we are detecting
the faces in real time
and then coloring them based
on whether the face is smiling
or blinking or combinations.
And we're getting about
25 frames per second.
We could do something similar
also for rectangle features.
So the idea behind rectangle
features is we understand
that in a lot of cases,
the first step in looking
in an image for something
interesting is to look
for something like a rectangle.
For example, if you're looking
for a sign or if you're looking
for a business card or if you're
looking for a piece of paper,
oftentimes looking for the
rectangle first is a great place
to start.
So we've created a generic
rectangle detector object
and it takes one
option parameter
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and it takes one
option parameter
which is the aspect ratio
that we want to search for.
And again, you can
ask the detector
to return the features array.
Now right now, it just
returns one rectangle
but that may change
in the future.
So here again, we wanted
to do a little sample here,
a little bit fancier
because we want to,
instead of just doing
the bounding box overlay,
we want to make it a
little bit prettier.
So again, we're looping over
all the features in the image.
We're creating a CIImage
WithColor which is infinite.
But we're going to take that
infinite color image and run it
through the
CIPerspectiveTransform
WithExtent filter.
And that filter does two things.
First of all, you
specify an extent
which in this case
we're specifying 0011
so now effectively, we
have a unit square image.
And then the other parameters,
take that unit square
and stretch it to the top-left,
top-right, bottom-left,
bottom-right coordinates.
And then we overlay that
on the previous result.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And then we overlay that
on the previous result.
And here's what that
looks like in practice.
So this is the nameplate from
my office and we are taking,
running it through the Detector,
getting the detected rectangle
and then producing this
overlay tinted red image.
Lastly, we can do the
same thing with QR Codes.
The code here is
exactly the same.
The only difference is
that we're using the QR
Code feature instead.
This example, you could
have also gotten the message
from the QR Code
but in this case,
I'm just going to do an overlay.
So all I needed to do was
use the coordinates and again
as you see in the example,
we can detect this QR Code
and do an overlay in real time.
So that's the bulk of
my conversation there.
The last thing I want to
talk about is improvements
that we've made to
RAW support on OS X.
So let me talk a little
bit about our RAW support.
So I'll talk about our
history, the fundamentals
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So I'll talk about our
history, the fundamentals
of RAW image processing,
some architectural overview
and how you can use this
great filter we have called
the CIRAWFilter.
So history first, so Apple
has been supporting RAW
since back in April of 2005.
Over those years, we have been
continuously adding support
for cameras and improving
the quality.
We have about 350
cameras supported today
and that's not including
all the DNG possibilities.
And one of the improvements
we've made in OS X this year is
that we support the latest
version of DNG specification
so that greatly improves
the number
of images that we can support.
And the other thing that's
wonderful about our support is
that it's provided to the
entire operating system
which means everything
from NSImages
to CGImages will
automatically support RAW files.
System services like Spotlight
and Quick Look support,
these key applications
like Preview, Finder,
even Mail support RAW.
Our photo applications
Aperture, iPhoto and Photos.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Our photo applications
Aperture, iPhoto and Photos.
Also all third-party
app can also get this
for very little effort.
So what is involved in
processing a RAW image?
And this is why, you know, this
subject is actually very dear
to my heart because
it involves a lot
of very advanced
image processing
to produce a RAW file.
So you start off with the fact
that RAW files contain only
a minimally processed data
from the camera sensor image.
And in fact, the image is
actually missing typically 66%
of the actual data because
at each pixel location,
you only have a red or
a green or a blue value.
And that means to produce a
final image, we actually have
to make up good values for
those missing 60% of your data.
And that requires a lot of
advanced image processing
to produce a beautiful
image at the end.
There are several
steps in this process.
They involve extracting
critical metadata from the file,
decoding the raw sensor,
de-mosaic deconstruction
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
decoding the raw sensor,
de-mosaic deconstruction
which is a hugely complex task,
lens correction,
noise reduction.
And then there's a set
of operations that are
in the color domain such as
mapping scene-referred color
values to output-referred
and then adjusting exposure
and temperature and tint
and then adding contrast
and saturation to taste.
So it's a lot of steps and
we've made some significant
improvements to several
of these in OS X Yosemite.
So we've benefitted
for lens correction,
a great new noise reduction
which we'll show in a minute
and also some improvements
to color as well.
So as I said before,
APIs like NSImage
and CGImage will get
RAW support for free.
And that's because our support
provides that default rendering
which is processed according
to all of our parameters
and whatever our
latest algorithm is.
However, we have this API
which is called the CIRAWFilter
which gives your
application much more control.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
which gives your
application much more control.
And it allows you to get a
CIImage with extended range,
floating point precision
and also
on that object are
easy-to-use controls
to control how our RAW
imaging results are processed.
And it gives you fast
interactive performance all
in the GPU.
So it's some great stuff that
you can use in your application.
So this is sort of how it
works as a flow diagram.
You start out with a file
and that can be passed either
as a file URL or NSData.
And that's passed as an input
to create the CIRAWFilter.
Also it can be specified on
that RAW filter are several
of our processing parameters.
Once you've set those correctly,
you can get a CIImage output
which you can then
display on the screen.
And by default, it'll look just
like our default rendering.
However, the great thing
about the CIRAWFilter is
that once the user
has seen those
and if your application
has controls,
you can alter those values, send
them back into the CIRAWFilter
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
you can alter those values, send
them back into the CIRAWFilter
where it can be re-displayed
all in real time.
Another great feature we have on
this is we actually have a place
where you can insert a
custom CIFilter in the middle
of our RAW filter processing
before we've done anything
to change the data
from a linear space.
So this is very useful
if you want
to do certain types
of image processing.
Now of course, you
can also apply filters
after the CIRAWFilter but this
is a great set of functionality
for certain use cases.
And lastly, it doesn't
have to go to the display.
You can also take the CIImage,
create a CGImage from that
and produce a new CG, a
file on disk from that.
And this is an example
of how little code it
takes to use this filter.
Basically, we start
out with a URL.
We create a CIFilter
filterWithImageURL
and that'll return
to CIRAWFilter.
In this particular
example, we want to get
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
In this particular
example, we want to get
from that filter what
our default value
for the luminance
noise reduction was
that returns to us as an object.
We can then make slight changes
to that like say for example,
you want all of your images to
be slightly more noise-reduced.
You can take that
value, add a bit to it
and then set that
as a new value.
And then once you're
done setting values,
you can get an output image.
So with just a few
lines of code,
you can leverage all
of our RAW pipeline.
So to show this in
much more detail,
I'm going to pass the
stage over to Serhan
who will be giving
a live demo of this.
Thanks.
>> In this part of our talk,
I would like to show you some
of the great things
that you can also do
in your applications
using the CIRAWFilter
and OS X's built-in support
for RAW camera files.
To do that, we created a
very basic simple application
that simply puts
up an NSOpenGLView
which is tied up
to a CIRAWFilter.
And another NSView which is tied
up to the controls
of the CIRAWFilter.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
up to the controls
of the CIRAWFilter.
So let me run that and
point it to a RAW image.
Now, by default, when you
actually open up a RAW file,
we will tap into our
own calibration database
and make sure that we
apply the correct set
of calibration settings that
are specific to the make
and model for this RAW file.
And some of the settings are
for you under lens correction,
white balance settings,
noise reduction settings
that we will go into more
detail in just a second,
exposure and boost controls.
So there is not much going
on with this very good
image in the first place.
So let me pull up a
more challenging image
to show the great benefits
of using RAW files.
Now, on this image, by
default when you load it,
you see that parts
of the image is close
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
you see that parts
of the image is close
to clipping point especially the
sky and the mountainous region.
So we're probably losing some
color fidelity in this region.
What's more interesting
is the part of the trees
which are underexposed and we're
probably not getting the right
amount of detail.
So let's see if we can
actually improve this image.
The first thing that I would
like to try is setting
the exposure
to see how it actually
looks like.
Want to probably increase
the exposure to make sure
that I get the detail in
the tree part of the image.
But as you can quickly see,
we're losing all the
detail in the highlights.
And the opposite is also true.
Once you start decreasing
the exposure,
you're getting back
the color in the sky
but you're losing all the
detail in the low lights.
So there is something that can
be done better and the answer
to that is CI Highlights
and Shadows Filters.
Normally, if you were shooting
JPEG, you would tie the output
of the JPEG decoder to this
highlights and shadows filters.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of the JPEG decoder to this
highlights and shadows filters.
But what's interesting
when you're shooting RAW is
that you can actually insert
this filter into the middle
of our RAW processing
pipeline and take advantage
of the linear input space
that we're operating in.
That means that you will be able
to better keep the
color fidelity.
You'll operate on a
linear 16-bit pipeline
and at the end, get
better results.
So let's try that.
The first thing that I want
to do is increase the shadows
and almost immediately I
can see that all the detail
in the shadow part is
kept, is brought back.
Same for the sky.
I want to bring it
down to make sure
that I can see more
of the sky colors.
And I can easily do that
without overblowing any part
of that image.
So that is a good example
of how you can actually use
the CIRAWFilter to make sure
that you can double up your
own images in the best way
that you think is appropriate.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that you think is appropriate.
Next noise filter.
Now noise reduction is a
very challenging problem
and traditionally it
is very computationally
expensive algorithm.
We're very happy to offer you
a new noise reduction algorithm
starting in OS X Yosemite.
That doesn't compromise on the
quality and you can still use it
at an interactive 60
frames per second rate.
To show you that, we have
this very noisy image
of the Moscone Center
and I want to focus
on this part of the image.
Just for fun, I'm going to turn
off all the noise reduction
to see what we are
dealing with initially.
So this is the original --
this is how the original
image looks like.
And using the CIRAW LNR and
CNR noise filter settings,
I can get it to a
state where I feel
that is most comfortable
for my image.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that is most comfortable
for my image.
So probably the first thing
that I want to do is get rid
of all the color noise and I'm
using the CNR slider to do that.
And look how interactive
this process is.
Same for LNR, you have a
wide variety of settings
that you can play with.
You can go with something that
is very smooth or something
which keeps all the
luminance noise.
So I want to probably hit
somewhere in the middle
where I got rid of most of
the noise but still kept some.
Another good thing that you
can do is brought back some
of the fine high
detail back to the image
after you clean up
all the bad noise.
So the detail slider is the one
that you would be
using for that.
And quickly you can get back to
this film grain type of look.
Same is true for high frequency
contrast and if you choose
to do that, you can also play
with it again 60
frames per second.
So that is the noise filter.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So that is the noise filter.
Starting with OS X
Yosemite, you'll also be able
to use this filter for your
JPEG images and this is going
to be a really nice advancement
on top of our offerings.
The last thing that I want
to show you today
is lens correction.
So a lot of the point-and-shoot
cameras
in the market today
are actually relying
on digital signal processing
techniques to fix some
of the compromises that
are made in the lenses.
What I mean by that, the
input image as you can see
by default is looking
correct to us.
But actual, the RAW
image that is coming
in is looking like this.
So whenever that data is
available in the file,
RAW camera will try
to do the right thing
and actually correct
for this aberration.
But for your own application,
you may choose to skip this step
and actually do your
own set of filters
or your own lens correction.
And it's an easy way
to actually go back
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And it's an easy way
to actually go back
to the actual RAW sample
of the file itself.
I'm going to now quickly turn
it back to David who's going
to talk about usages
of the second GPU.
[ Applause ]
>> Thank you, Serhan.
So as you saw, we have this
great new noise reduction
and it's a very complex Core
Image filter that we developed
and it makes great use of
the GPU which brings us
up to talking about
the second GPU.
So a year ago, we announced
at the WWDC our new Mac Pro
which has this great feature
of having a second GPU,
just waiting for
your application
to take advantage of it.
So let's talk a little
of how that can be used.
So we had some thoughts about
for Core Image and for RAWs.
When is a good time where you
might want to use a second GPU?
And a couple of scenarios
come to mind.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And a couple of scenarios
come to mind.
One is if your application
has ability
to do speculative renders.
For example, you may have
a large list of images.
The user might be looking
at one but he may switch
to the previous or the
following image at any time.
Your application could be
speculatively rendering the next
or previous image
on a second thread.
Similarly, your application
may have the ability
to do a large batch
export and you want to do
that in the background and
you want to use the GPU
but you don't want
that background GPU
to affect your foreground
GPU usage.
So these are both great
reasons to use the second GPU
because it allows you to
get the best performance
without causing your
user interface
to stutter for its usage.
So how does one do that?
Well, you could do this
today on Mavericks.
It takes around 80 lines
of OpenGL code to tell,
to create a CIContext
that refers
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to create a CIContext
that refers
to the second offline GPU.
However, we've added a simpler
API in Core Image on Yosemite
which is CIContext
offlineGPUAtIndex
and typically you
just specify zero.
So with one API call, you get
a CIContext and that when using
that all renders will
use the second GPU.
So it's very easy.
And to show that in action,
I'm going to bring Serhan back
up to do a quick demo.
>> Well, in our first demo,
we showed that even the most
computationally expensive noise
filter algorithm can be done
at 60 frames per second.
I'm going to bring that
application back and open
up a very noisy image.
So our LNR controls can be
done at 60 frames per second.
To show you that, we actually
wrote a little bit of code
to display the frames per second
when I'm actually sweeping
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to display the frames per second
when I'm actually sweeping
through all the noise
filter settings.
And as you can see, I'm
getting 60 frames per second all
the time.
Now let's say that you
have a background trait
where you are constantly
exporting images
and for some reason you wanted
to do a GPU pipe
on your first GPU.
To simulate that, we
have written a little bit
of text application which
is using the first GPU.
And when I go back
to my own application
which is now also
using my first GPU,
I can see that the frame rate is
actually suffering a little bit.
I'm going to run my test shoot
one more time to see what type
of frame rate I'm
getting out of this.
And you can quickly see that
it has dropped down to 50%.
I'm getting 24 frames
per second.
So can we do something
better than that?
And the answer is yes.
If we can offload this work
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
If we can offload this work
to our second GPU using
the CIGLOfflineContext,
I will get back to my original
performance in my active app.
And to show you that,
here we go one more time.
I can see that the user controls
are once again very smooth
and the frame rate that I'm
going to get is close to 60.
So once again, this is a
great way to take advantage
of the second GPU if you are
constantly doing computationally
heavy algorithms
in the background.
I'm going to hand it
back once over to David.
>> So to summarize what
we've talked about today.
We've talked about
some key concepts
to understand about Core Image.
We've talked about what's
new in Core Image on iOS 8,
most notably Custom CIKernels
and large image support.
We talked about some
new things in Core Image
on Yosemite notably
some API modernization
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
on Yosemite notably
some API modernization
and some great new noise
reduction and RAW support.
We've also talked about how
to use the latest CIDetectors
and how to work with
RAW images in ways
that you may have
not imagined before.
So this is the usual information
about who to contact.
Allan's a great person to talk
to if you have a request
for more information.
Related sessions, there's one
I really hope you guys can come
to is our second session this
afternoon where we're going
to be talking about how to
write Custom CIKernels on iOS.
And also, we have a lab
session that follows that so
if you have coding questions,
please come and we would love
to hear your questions
or suggestions.
So that's all.
Thank you so much for coming.
[ Applause ]