Transcript
[ Music ]
[ Applause ]
>> Hello and welcome.
I hope that everybody's
having a great WWDC.
My name is Justin Stoyles,
I'm part of our Graphics
and Media Group here at Apple.
And I'm really excited
to talk to you today
about working with wide color.
We're going to cover
a few topics today.
I'm going to start
with discussing some
core color concepts.
And we'll talk a little bit
about what is wide color
and why does it matter.
Then I'm going to hand
it over to Patrick,
who's going to walk you
over some of the processes
of getting wide color assets
into your application.
And then I'm going to
hand it over to Steve,
who's going to talk about
rendering wide color.
who's going to talk about
rendering wide color.
So let's get started.
As I mentioned, I'm
excited to talk to you
about wide color today
and the reason why is
because we're actually
transforming the way
that we handle wide color,
and colors in general
on the Mac and iOS platform.
A good place to start
is in the '90s.
So Apple has always
been at the forefront
of solving the color
problem in computing.
And a good place to
start with ColorSync.
So ColorSync is our color
management platform on the Mac
and it was developed in
the '90s and at the time,
Apple and a number of other
like-minded companies got
together with the idea that
standards in color management
and color reproduction
would begin to solve some
of the problems faced by
engineers and designers
when working with color.
That group later became known
as the International
Color Consortium, or ICC,
which you may be familiar with.
The work of ICC was then
incorporated into ColorSync
and we built that
fundamentally into OS X.
Now Apple has also been at the
forefront of display technology.
We now have these
beautiful retina displays
across our product line.
People really, really love them.
But what really is
the next frontier?
Well, last year we announced
the beautiful new iMacs
with P3 displays.
And the feedback we got
was really tremendous.
They display 25 percent
more color.
And they have the
P3 color space,
which people really,
really love.
And if you fast forward
another year,
we have these new iPad Pro
9.7 displays with True Tones.
So we've taken that
even further.
Now, there's a lot
happening under the hood here,
and we're going to try to get
into some of those topics today.
And I'll start with
Core Color Concepts.
Let's start very,
very, very simple.
So, what is a color space?
Well a color space
is an environment
in which colors can be
compared and represented.
It can be a 1, 2, 3,
or 4 dimensional space
that is defined by the intensity
of its color components.
Those color components
are also often referred
to as color channels.
One representation that you
might be familiar with is RGB.
So, this is very
relevant with displays,
where your subpixels are
red, green, and blue.
And to think about these
color channels you can think
that the intensity of each
of these color channels
defines the color
of these color channels
defines the color
that you see on the other end.
There's many different
types of color spaces.
I mentioned RGB spaces, which
are very popular in displays.
But there's also lots
more, there's gray spaces,
which are used in
monochrome printing.
In color printing we
often use CMYK spaces.
And if you're trying
to do calculations
and transformations, we
generally use color spaces
like LAB, which are
device independent.
So now we have color spaces, and
color channels, and we now want
to create a coordinate system
where we can compare
and compute colors.
So there's the concept
of color primaries.
So color primaries generally
fall at the most intense value
that you can get with that
particular color channel.
So in this example of an RGB
color space, you would see
that the color primaries are
where we anchor 1.0
in our color space.
So for black, I would
have no saturation in each
of my color channels,
so I have 0, 0, 0.
For white I would have 1, 1, 1.
And for red I would only
saturate the red channel,
and I would have 1, 0, 0.
Very simple.
And when we refer to color
gamut, we're actually talking
about all of the colors that
can be defined as a combination
of those individual
color channels.
So now you understand some
of the basic color concepts
that we're going to be
discussing in this presentation.
So then what is wide color?
Well first we should
talk a little bit
about what is the standard
in the industry today.
So standard RGB, or sRGB is the
most widely used color space
in computing today.
It's based on the
BT.709 standard.
We use an approximated
gamma of 2.2.
Typical lighting conditions.
Which in this case is D65.
And it's the default
color space for iOS.
And it's actually the
default color space
for a lot of platforms.
Which is very convenient,
because there are some platforms
out there that are
color managed,
and there are some that are not.
And when a lot of
the content that is
in existence today is sRGB,
you can make some assumptions
on your incoming content
and have it reproduce
faithfully on the other end.
But that's not always
going to be case.
sRGB does a good job of
describing colors that we work
with on a daily basis, and our
systems have been really good
at displaying those
colors to us.
at displaying those
colors to us.
But really there's a lot of
colors that don't fit into sRGB.
A lot of textiles are
designed using inks and dyes
that have lots of colors
that are outside of sRGB,
mainly because these
colors really catch our eye.
They're really vivid
and impressive.
If you're watching
soccer for example,
a lot of the uniforms are
actually outside of sRGB,
because they're striking,
they get our attention.
Whether you're watching
your kid's soccer game,
or you're watching the
Eurocup, you're going
to see a lot these
jerseys with colors
that are actually not
describable within sRGB.
A lot of products are
also designed with colors
that are outside of sRGB.
And these are products that we
interact with on a daily basis.
But really the most compelling
examples are in nature itself.
Sunsets, autumn leaves,
tropical waters.
These are all things in
nature that have colors
that are outside of sRGB.
And we actually want
to show those.
And the nice thing is you
probably have a lot of images
that you've taken
on your camera,
especially if you're
capturing in RAW,
that actually contain a
lot of this color data,
but you're not seeing
it on your display
but you're not seeing
it on your display
if your display is only sRGB.
So what do we do about this?
Well, last year as I mentioned,
we introduced products
with a new color space.
And that color space
is Display P3.
So in our iMacs, and in
our new iPad Pro 9.7,
we use this color space.
And it's based on the
SMPTE standard of DCI-P3.
But it's a little bit different.
DCI-P3 is a color space that is
defined for digital projection.
And it works really great
for those viewing conditions.
But our viewing conditions
are a little bit different.
And sRGB is really great at
defining a standard that works
with our viewing conditions.
So we adopted the same gamma,
and typical lighting conditions
for our white point, as sRGB.
These are the key differences
between Display P3 and DCI-P3.
In order to understand what
colors are actually outside
of sRGB, but can be
described in Display P3,
I'll try to show it
to you in action.
So here's an image
from my photo library,
and I exported it using photos
in a Display P3 color space.
This is a really
beautiful image,
and even though we're viewing
it on an sRGB projector today,
or if you're viewing the video,
you're viewing it in sRGB today.
This image still looks
really, really great.
But there's a lot of
things in this image
that you might not notice right
away, but there's actually a lot
of colors that aren't
actually describable in sRGB.
These are colors that are
outside of the sRGB color gamut.
And in this case what you
end up with is posterization
in those regions, so
those colors are flattened
on the edge of the sRGB gamut.
So how do we fix this?
Well, we're moving more
professional workflows
to our mobile platforms.
And in order to enable that,
it's not just putting a wide
gamut display into your system.
There's a number of other things
that need to come along with it.
One of those things is we need
to upgrade our factory
calibration.
So, now all of our products have
individually calibrated displays
that from device to
device you can count
on those colors being
accurate and consistent.
And then finally, we had
to build full system-wide
color management into iOS.
And that's exactly
what we've done.
But we can't just build the
same color management system
with the same approach
as we did on the Mac.
There's different considerations
and different restrictions
when working with
a mobile platform.
One of those considerations
are the countless applications
that already exist
in the iOS ecosystem.
These are applications that are
built in sRGB, tuned for color
and performance in sRGB.
So how do we make sure that
those applications can still run
on our platforms that
are now color managed,
without any impact
to performance,
without any impact
to your colors.
And what we did is we
built on top of sRGB.
So we've introduced an
extended range sRGB color space.
This is our working space
for working with wide color.
We use the same sRGB primaries.
We use the same gamma 2.2.
The same white point.
But, the interesting
difference here is
that we actually allow values
that are greater than one
and less than zero,
which allows us
to not only keep the same
performance and look of all
to not only keep the same
performance and look of all
of your applications that are
built in sRGB, but using values
that are negative and
values that are greater
than one allow us to express any
color in the visual spectrum,
while still keeping our
anchor points in sRGB.
The best way to describe
this is with a demonstration.
So let's say I want to
take the most saturated red
in Display P3.
So that would be 1, 0, 0.
Now, if want to express
that in extended range sRGB,
it would look a little
bit like this.
So in the red channel,
I actually have a value
that is greater than one.
And in the green
and blue channels,
I'm actually using
negative values.
So I'm subtracting
green and blue,
and adding oversaturated red.
And what that allows me to do
is get a color that is outside
of the sRGB gamut, while using
that same sRGB anchor points
in extended range sRGB.
The interesting thing is if
you're using a pixel format
that allows you to go very
negative, and very positive,
this approach allows
you to express any color
in the visual spectrum.
So this is scalable.
Speaking of pixel formats,
sRGB is largely standardized
in 8 bits.
So 8 bits is largely enough
to describe colors in sRGB,
not perfect, but great.
Now an easy example to
describe the difference
between the precision that
we would want for sRGB,
and the precision
that we would want
for wider gamut, is
with this example.
So say I wanted to build
a staircase in my home
from the basement
to my first floor.
I would use, say 8 stairs, I'd
probably use 12, but let's say
for simplicity I would use 8.
This is the appropriate
height for me.
Now, if I wanted to extend that
staircase up to my second floor,
I wouldn't just use the
same number of stairs.
And when we're using
digital encoding for color,
it makes sense that if you're
trying to express more colors,
you would want to
us more precision.
So if we're talking about using
colors that are outside of sRGB,
So if we're talking about using
colors that are outside of sRGB,
really our recommendation is to
use 16 bits per color channel.
Now, before I hand
it over to Patrick,
I wanted to end with
one more point.
So, we've gone through our
entire system and upgraded all
of our frameworks and
tools to be color savvy.
But if you're building
your application
with open source tools,
or you've built your own
image processing pipeline
from scratch, you need to
take some action to make sure
that the tools that you've
used are color savvy.
Otherwise, we've
got you covered.
And for more on that, I'll
hand it over to Patrick.
[ Applause ]
>> Thank you Justin.
Hi. I'm Patrick Heynen.
I'm a Senior Engineering Manager
in the Cocoa Frameworks Group.
And I'd like to ask where
does wide color come from?
Well the answer is it turns
out, it's not just shirt.
No, it comes from you.
Namely, it comes from
apps and their content,
and the user experiences
that you provide in
your applications.
So what kind of content types
are amenable for wide color?
So what kind of content types
are amenable for wide color?
Well the first one is the one
you're probably most familiar
with and that's static
image resources.
These are the sort
of the PNG, or JPEG,
or individual graphic files
that you may have bundled
into your application
and shipped along
with your application
bundle to your customers.
The next category is document
and network-based
image resources.
These are individual
image content
that you may either download
off of a network service,
or store in your document data.
There's also further
categories, like advanced media,
which refers to things like
live photos, or content acquire
from the build-in
iPhone cameras.
I'm not going to go
into too much detail
about wide color implications
for those kinds of types,
but I refer you to the
Advances in iOS Photography
and Editing Live Photos
and RAW on iOS Sessions
for more details about that.
And lastly, there's also
the concept of GPU textures.
If your program -- if your
application is working
If your program -- if your
application is working
at the GPU level, like your game
engine, or your advance piece
of graphic software, and you're
sending color information
in the form of either
textures or shader values
to the GPU directly, there
are other considerations
as well for wide content.
I'm not going to go into much
detail about those either,
but I refer you to my colleague,
Dan Omachi's excellent treatment
of this in What's
New in Metal Part 2.
Okay. So let's frame
the color problem.
Justin gave a great
explanation earlier.
But how does it apply to apps?
Well app content can come from a
broad range of sources and come
in a broad range
of color richness,
all the way from grayscale,
all the way through to even
16-bit wide color gamut content.
At the same time, devices and
displays come in a broad range
of color capabilities.
For example, in iPhone
5 it was only capable
of representing sRGB, all the
way through the latest iPad Pro
which does a great job
with full P3 colors
which does a great job
with full P3 colors
and extended range sRGB.
So, how do you bridge
the difference?
Well, we're going to
solve this color problem.
And the backbone really
is color management.
So, what is color management?
The job of color
management is to ensure
that an image looks the
same on any output device,
no matter what color
space it is encoded in
or how it was originally
created.
That's pretty much it.
But how does it work?
Color management
starts with every image
or content type having an
associated color space,
sometimes referred
to as color profile.
This informs the system what
the colors actually mean.
And then the color matching
process algorithmically maps
those colors to the
characteristics and output space
of the device that
you're rendering to.
Of course, this is a
computational process,
and it is not for free.
In fact, every single
pixel needs to be touched
In fact, every single
pixel needs to be touched
and converted and matched.
Also, it's important to
note this is a potentially
lossy operation.
Especially if you're going
from a wider color gamut source
down to a narrower one,
say a 16-bit P3 content
down to an sRGB 8-bit,
there's going to be some loss
of color fidelity
in that scenario.
So that's something
to be aware of.
So these are important
aspects, but there's good news.
The first good news is that
the color matching operations,
the way they're algorithmically
defined, I'm not going to go
into the details of the
color science behind them.
But it turns out
that computationally,
they're very easily
hardware accelerated,
either by the CPU or by GPU.
Which leads to the
next good news which is
that we've built
this functionality
into the system itself, so that
it all works automatically via
the Quartz 2D, ColorSync on
the Mac, and Core Animation.
In fact, all you really
need to make sure of is
that your content is tagged
properly, and in that case,
there's no code required to
display your image correctly,
there's no code required to
display your image correctly,
and have the colors
appear correct.
So, platform color management.
Well, macOS, as Justin eluded
to has been color managed
since its inception.
Some might even say since
before its inception,
when it was previously
known as macOS,
during the ColorSync times.
So there's nothing new there,
but there is something
new on the iOS fronts.
We have, since iOS 9.3,
we now have automatic
color management support
on most devices.
So that's color management,
the backbone
of solving the color problem.
But now let's talk about how
some design consideration
and how the toolchain and the
platform services have been
enhanced to accommodate
getting wide content
into your applications.
First, as always it
starts with design.
So, what is important to think
about when you're
approaching the wide color
from a design perspective?
Well, the first thing I would
say is that it's important
Well, the first thing I would
say is that it's important
to use wide gamut content
where it makes sense.
It doesn't mean everywhere,
remember most of the colors
that we interact with on a daily
basis, are in fact contained
within that nice sRGB triangle.
That's where the bulk
of the colors are.
There are however, these nice,
new vivid saturated colors
that may make sense for
your application to use.
So really it's important to
think about wide gamut content
as being a tool that you can use
where vivid colors really
enhance the user experience
and add some value
to your application.
There is no need to go and
update all of your content
and immediately upgrade
it to P3.
This is not that kind
of technology shift.
This is merely a new
creative tool that you use
where you want it,
when you want it.
And the good news is we've
enhanced the toolchain support
to make this gradual opt-in
of wide gamut possible.
So let's say you do want
to upgrade some content
to wide color, what are
some considerations?
It's really important
to be careful
when you upgrade a
design file to wide color.
when you upgrade a
design file to wide color.
A common pitfall is to
just assign a new profile.
It starts out as sRB, assigning
a new wider color profile.
This is a mistake, because this
will just remap the existing
color information
into new color space.
It's a pretty effect, but it's
probably not what you want.
Because it will just stretch
all the colors out to be vivid
and the appearance of the design
file will be inevitably altered.
Instead, it's important to
use convert to P3 instead.
This will actually do a
color match operation,
your resulting design should
not change its appearance,
but it will be prepared
to go and have some
of its content boosted
up into those nice,
vivid saturated P3 colors.
When you're working on wide
gamut designs, it's important
to use, and we recommend
strongly
that you use the
Display P3 color profile
as your working document
profile.
Also to maintain
maximum color precision
and highest quality
it's a good idea to work
and highest quality
it's a good idea to work
in 16-bit per channel
color mode.
And of course, it's nice to
be able to see your designs.
So, if you are going
to be working
with wide color it's good
to do that design work
on a system capable of
rendering wide gamut color
such as the late 2015 iMac,
or other capable hardware.
When it comes time
for production,
for exporting delivering
assets, it's important
to use 16-bit PNG files with an
embedded Display P3 ICC profile.
This is sort of the gold
standard of content interchange
for wide color content.
Now, a brief note.
There are some very popular
content production workflows
out there.
They come under different names.
I'm going to talk specifically
about the Adobe Photoshop
Workflow.
They come under the names of
Save for Web and Export Assets.
Many of these workflows
have not made the transition
to wide color yet, and
they are not compatible
with exporting 16-bit
P3 content.
with exporting 16-bit
P3 content.
So, stay away from them for now.
And instead use the workaround
of using save as PNG format
with 16-bit and embedded
display profile as a workaround.
Okay, so that's the
design process
and how wide color impacts that.
Now, let's talk about tools
and how they can be used
up to incorporate the
content you've produced
out of the design process.
Well the first thing
I'm going to talk about
and really the epicenter
of the content tool story
is Xcode asset catalogs.
Now you may be familiar
with this.
This is the general tool in
Xcode to allow you to organize
and catalog your image
resources, mark it up
and provide metadata,
and deliver it inside
your applications.
What do asset catalogs
do for you?
Well, they're the best
deployment vehicle
for static assets.
We automatically color
correct your source content
and optimize it for the target
that you're building for.
So even if, maybe your
designers are prefect
So even if, maybe your
designers are prefect
and always can deliver their
content with the right profile
and all the right information
in their deliverables,
but that may not
always be the case.
That's where this automatic
color correction comes in handy
to make sure that we
match and normalize it all
into the appropriate working
space for the target device.
Also, we do automatic
pixel format optimization,
making sure that the
right bit depth ends
up in the right device.
And, last, but definitely
not least,
asset catalogs are the
gateway to app slicing
which is an important component
of the thinning feature
that assures that only the
content that's relevant gets
delivered to your customers on
a particular device and none
of the content that doesn't.
So, what have we done in asset
catalogs to make it easier
to work with wide color content?
Well the first important
enhancement is we now support 16
bit source content and
provide a clean end-to-end path
for 16 bit image resources.
This, we store it
in 16 bit half load,
but it is available
for you in Xcode 8.
but it is available
for you in Xcode 8.
The next important thing
we've added is the ability
to catalog a display gamut.
What does this mean?
This means that you really
introduced a new option
in the inspector to provide
and tailor optimized assets
for either the sRGB
or Display P3,
display gamuts matching devices
with those characteristics.
Okay so that's the feature.
That's what Xcode has to
offer for organizing assets.
But how do you use it?
How do you think about using it?
Well, there's really
three easy choices here.
Choice number one, do nothing.
This may seem radical, but it is
in fact a perfectly
valid thing to do.
Because if you make no
changes to your asset catalog,
you may not need any of
these new vivid colors,
we will continue to render your
8 bit sRGB content faithfully
on all devices.
So nothing will change.
You don't need to go
update everything just
You don't need to go
update everything just
to make sure it works
on this new hardware.
So this is a perfectly
valid option.
Of course, this does mean
your app will not contain any
wide colors.
That's a perfectly
fine design choice.
That's the outcome of
this deployment choice.
So let's say you do want
to use some wide color
in your application.
Brings us to choice number two.
Upgrade to P3.
So, the way this
works is something we
like to call Universal P3 Asset.
So, all it involves is taking
a 16 bit Display P3 file
that you may have gotten
from your designer,
and just replace your existing
asset in the asset catalog
with this upgraded asset.
What happens at build
time is we're going
to automatically generate
an sRGB derivative
from that 16 bit
Universal Display P3 master.
We're going to do a
high-quality color match,
and a high quality
dither down to that 8 bit.
And then of course at
thinning and content selection,
we're make sure the right
content variant gets selected
on the appropriate device.
But let's say you're
not really happy
with this automatic conversion
and you want full control.
Well the good news is we
have that for you as well.
And that's choice number
three, optimize assets.
It's a very simple option.
It's basically, you provide
a 16 bit Display P3 content
and your original
8 bit sRGB content.
We provide places for
you to organize both
of those in the asset catalog.
And they'll get built
into your app and selected
and thinned appropriately.
Okay, that's wide color
assets and asset catalogs.
I'd like to now give
a demonstration
of creating content.
Okay, so let's say I've got
this beautiful hue wheel here.
Let's say I have this
beautiful hue wheel here.
There we go.
That's much better.
And I really decided this
really could in fact benefit
from some more vivid
saturated colors.
How is this done?
Let's take a look at this here.
This is just a simple gradient
here, a radial gradient.
And I just want to call
out here this point
in the gradient is a
pure saturated green.
Okay, that's what
I'm starting with.
So what do I do to upgrade it?
Well the first thing
I'm going to do
because I'm an organized type,
is I'm going to create a copy
of this and not just
destroy the existing asset,
I'm going to name it
something new here.
And, okay now I'm ready to work.
So the first thing I'm going to
do is I'm going to update this
to 16 bits per channel.
Now, we're ready
to work in 16 bit.
And now the all-important
convert to profile.
And what I'm going to do
is I'm going to change this
to the Display P3 color profile.
Okay, now we're ready to go.
In fact, I can confirm with
document profile down here,
In fact, I can confirm with
document profile down here,
Display P3, 16 bits
per component.
I'm good to go.
Okay, so now let's take a look
at what happened
to that gradient.
And look at our favorite
green here.
Okay, so that's interesting
right?
That green it didn't change.
Notice that the hue wheel didn't
change its appearance at all.
That was actually the
intended goal of converting,
as opposed to assigning.
But I'm left with
the same green now,
being only 70 percent saturated.
Well this really indicates just
how much head room there really
is to punch out into
that wider gamut
and deploy a more purer green.
So, well let's say I want to
do that, but I don't want to be
on stage all day, I'm just going
to reapply the gradient preset.
Now everything should be
back to what it was before,
100 percent saturated, but now
in that pure P3 wide
color space.
So there we are.
Now I have my beautiful asset,
I'm going to go ahead and save.
And of course this is just a
design file, I can't use it
in my app until I save it.
So I'm going to do that.
I'm going to save it as, whoops
not as a [inaudible] but as PNG.
I'm going to save it as, whoops
not as a [inaudible] but as PNG.
I'm going to make sure to
embed the color profile.
And save it.
And viola I'm done.
Well, actually I'm not done,
I've just finished
the design process.
I haven't incorporated
it into my app.
So, let's go do that.
Where's my app?
There's my app.
So here's my app, unmodified.
Here's my existing
sRGB P3 hue wheel.
Well, what I want to do here
is I want to go over here
to the pop-up here to expose
the sRGB and Display P3 gamut.
This will immediately
reveal a couple
of additional cataloging
options.
And I take my P3 hue wheel
and just drop it
right into that slot.
And I'm ready to go.
Now if I build in run, it
will compile both of these
into my asset catalog.
And if I was running on
an iPad Pro 9.7 inch,
I would be getting the
16-bit Display P3 asset,
whereas on an sRGB
device like an iPhone 6,
I would be getting this asset.
Okay. That's incorporating
content into your application.
Okay so we talked about tools
and how they work
with live content.
Now it's time to talk about some
of the deployment considerations
and what happens after
you've used that tool
and how it might impact the
running of your application.
So, with asset catalog
deployment, app slicing is going
to ensure that the
appropriate variant is delivered
to a given device.
And this is really important
because this is potentially a
lot more content now that's part
of your application.
And, with that slicing,
we ensure that there's
no payload cost
for your actual end-user
customers
for adding this wide gamut
content into your application,
because we make sure
that with app slicing,
wide gamut content only
goes to wide gamut devices,
sRGB content goes
to all the rest.
And it doesn't waste
space on the devices
that it's not relevant for.
On the Mac, there's
actually nothing new here,
On the Mac, there's
actually nothing new here,
NSImage has always been able to
select the best representation
from available representations
in the asset catalog.
It's going to continue to do
so based on the characteristics
of your target display.
So if you are on a wide
gamut iMac for example
and you have P3 content
available in your asset catalog,
it's going to go
ahead and select
that when rendering its content.
Also, just like the behavior
with 1X and 2X assets
on the Mac, NSImage
and NSImage View
and all the related app
kit classes make sure
to refresh the content
automatically
when the display
characteristics change.
Such as when your window
goes before the internal
and the external display, or
the color characteristics,
or back end scale
factor changes.
So that's great.
Okay, but how is this
data actually stored
and what impact can that
have on your application?
Well the good news is that at
build time we tried really hard
to optimize the pixel formats
and storage characteristics
of all your image content
in the asset catalog.
of all your image content
in the asset catalog.
And do so in as efficient
a way as possible,
to the best of our
ability anyways.
We do now use 16 bit
per component storage
for wide color content.
I mentioned this before.
This really allows you to
have end-to-end 16 bit color
precision in your
application resources.
And, we also have compression.
Now this is quite handy because
more data, 16 bit versus 8 bit.
More information is
inevitably going to lead
to large footprint unless
you apply compression.
Now, we've always
had compression,
we've always had
lossless compression.
Akin to what happens
with say PNG files.
But what's new this year is
some lossy compression options
to help counteract, you
know app footprint size.
The first one of these is what
we call basic compression.
Now this is a compression system
that's almost exactly like JPEG,
except the added bonus that it
actually handles transparency
and alpha as well.
It has a similar
visual characteristics
and performance characteristics.
And the great news is it
really performs really well,
on almost all, on
all devices really.
Just like JPEG does.
So it's something you can count
on to give a very slight
reduction in visual quality
to get a really excellent
storage characteristics.
Now, new this year as well,
we have this exciting option
called GPU compression
using ASTC.
So this stands for Advanced
Scalable Texture Compression,
which is a GPU compressed
textured pixel format,
which many modern GPUs
on Apple devices as well
as other systems support.
And we have brought this
option to you in asset catalogs
in two different forms.
One is GPU best quality,
which is a constant bit rate,
4 bits per pixel,
ASTC compression mode.
Which is a great choice,
roughly analogous in terms
of visual performance and visual
fidelity to high quality JPEG.
of visual performance and visual
fidelity to high quality JPEG.
And then we also have
GPU smallest size
where if you're really wanting
to optimize your footprint
and your memory footprint,
greatly then you
choose this size
and that's a 1 bit per pixel
constant bit rate codec
which has excellent
storage characteristics.
Of course, since not
every GPU on every device
that is supported today
can use the ASTC format,
we take an extra step for you
and automatically generate a
software fallback for devices
that lack that capability.
What's great about this is it
means you don't have to worry
about having incompatibilities
of your content
for all the supported devices,
we're going to automatically
generate that fallback,
and use it and thin it,
and route it appropriately
to those devices that
cannot support ASTC.
So you can use GPU compression
without any real risk
of compatibility breakage.
A brief note about how
GPU compression interacts
with wide color assets
specifically.
with wide color assets
specifically.
So, we use the ASTC Low Dynamic
Range, or LDR compression mode,
which really means that
wide content actually needs
to be reduced to 8 bits
before compression,
because LDR is an 8 bits per
sample compression format.
The good news is we
perform this work for you.
We automatically do perform
a high quality dither
down to 8 bit at build time when
we're processing your images.
But we preserve the wide gamut
colors that may be contained
within your original 16 bit
source content by encoding
and compressing in the
Display P3 color space,
thus preserving the ability
to represent all of the colors
within the Display P3 gamut.
Okay, so that's the deployment.
A little talk about
deployment characteristic,
but this talk was
supposedly about colors.
Well what about colors?
Specifically colors in UI.
An important observation is
that most of the pixels you see
on screen drawn by
most applications,
on screen drawn by
most applications,
don't come from images,
even though they tend
to get top billing
in this kind of talk.
Most pixels on screen are
actually solid colors drawn
by your code in your
application.
And wide gamut colors it turns
out, can present new challenges
in just working at
that simple level.
So let's talk about that.
In particular, the first
challenge I'd like to talk
about is actually
talking about colors,
because this is an
under-appreciated problem,
under-appreciate problem.
Usually when designers
and engineers communicate
in written form, or
perhaps even verbal form,
or visual form code is
usually communicated
with an assumed sRGB
color space.
That means you're probably
used to seeing colors written
down as something like
this, RGB 128, 45, 56.
You know very simple.
They don't tell you what
color space they're in.
And it's just assumed everybody
knows what color that is,
because everybody's
using sRGB, aren't they?
Well, not anymore.
This is no longer sufficient for
working with wide gamut colors?
So what do you do?
Well the most important step
you can take is be specific
about what color space you're
working in when you communicate,
or write down, or, you
know, transmit that color.
Use Display P3, instead of
sRGB when you're working
on wide gamut designs.
And indicate that as such.
And if you need more
precision than 0
to 255 8-bit representation
can give you then go ahead
and use floating point.
So as an example, next time
you're sending colors in email,
maybe you'll refer to them
as something like this,
with an annotation, P3
255, 128, 191, etcetera.
Okay that's how you communicate
a color, but where did
that color come from
in the first place.
You probably picked it,
the designer probably
picked it somewhere.
How do they pick it?
Using a color panel.
This is the standard color panel
that gets shipped with the Mac,
it's part of AppKits,
known as NSColorPanel.
This of course is a very
familiar UI, but it also suffers
from some of the
limitations we just talked
about in calling colors.
Typically, you pick red,
green and blue values,
0 to 55 numbers.
The color panel is
always supported selecting
in different color spaces,
but that hasn't always
been a very obvious
or easy user experience.
So I'm excited to say that we've
actually made some enhancements
in the color panel, in
macOS Sierra to make working
with wide color a little easier.
The first thing we've done
is we've put the most common
and important working spaces,
mainly Display P3 and sRGB,
right up there in a
context in the action menu
for the number picker, sRGB,
and of course Display P3.
The next thing we've done is
we've actually allowed an option
to change the numerical
representation of the colors
from integer to floating point.
So that you can standardize
on floating point colors
if that works for your workflow.
Another exciting thing we've
done is in the hue wheel,
or wheel picker, we've actually
changed its implementation
so that it can actually render
the full gamut of P3 colors
when placed on an
appropriate display.
when placed on an
appropriate display.
And we've added a new
context menu that allows you
to either use this
automatic behavior,
where it just basically switches
its capability between P3
and sRGB as you cross displays,
or pin it to a specific
color space,
if that's more what
you'd like to see.
Okay, now we've picked
our colors,
we know how to communicate them.
Well, to actually make
anything happen we have
to code them don't we?
How do we construct wide
gamut colors in code?
We've introduced two new
convenience constructors
in AppKit and UIKit to
take Display P3 color
numbers directly.
This is NSColor Display P3
red, green, blue, alpha,
and UIColor Display P3
red, green, blue, alpha.
Great way to work with P3
colors from your designer.
But of course your code
may actually be working
with extended range sRGB
colors that maybe didn't come
from a designer, but maybe came
from a different subsystem,
or an API that might
have vended,
you know extended
range sRGB to colors.
How do you work with those?
Well the good news is we've
extended the existing standard
red, green, blue alpha
constructors for NSColor
and UIColor color to not
clamp their input components
and allow values to
be expressed greater
than one and less than zero.
So now you can construct
extended range sRGB colors
very easily.
Okay, now storing colors.
What if you actually
have to like put colors
into your document data, or
archive them in some form?
Well special care really needs
to be taken when doing that,
because like many other
parts of the ecosystem,
there has traditionally
been a lot of assumptions
of assumed color spaces and
assuming everybody's talking
about the same thing
when you pass a red,
green and blue number.
All you should ever need
is three numbers, right?
That's all you need
to represent a color.
What could be wrong with that?
Well it turns out there is.
Because you may not be able to
disambiguate between an sRGB
and a P3 color, and you may
make the wrong choice and end
up with the wrong
color in your document.
So I'd like you to
consider, perhaps,
So I'd like you to
consider, perhaps,
encoding a compatible sRGB color
alongside the new wide gamut
color, which allows
your application,
especially older versions
of your application,
because since this is
document data, you have forward
and backward compatibility
to consider.
Allows the older applications to
continue pulling that sRGB data
and treat it as if it was sRGB,
which is the assumption
it's making.
Whereas the newer
software can know
about this new wide gamut code
that you've stored alongside it.
How do you create that
compatible sRGB color?
On iOS you can use the
CGColor.convert API.
And on the macOS you
can use the venerable
NSColor.usingColorSpace
API to convert those colors
to the sRGB color space.
We actually ran into this in
the system itself in macOS.
You may be familiar
with TextEdit.
And the text editor
app on the Mac.
And its document format is
RTF, or rich text format,
which has been around
and is supported
by the Cocoa Text System as its
needed document storage format.
And it turns out that when you
actually applied a color to say,
a range of text, the way
it would actually store
that in the RTF document was
in fact as just simple red,
green, blue, 0 to 55 values.
Which is kind of a problem, I
think you'll understand now,
because we have no idea which
color space those red, green,
blue values are in, and
it doesn't allow you
to express the full range.
So, we had to take action.
What we ended up doing
is revving the RTF spec
and how we write and read
RTF on our platform at least.
By including an expanded
color table
which actually annotates
each of the color tuples,
which have now been
turned into 16 bit integers
with individual color space
information, allowing you
to specify colors in
multiple color spaces.
Okay. That's great.
But this is California,
we're supposed
to be surfing aren't we?
Well, what does that
have to do with colors?
Well, there's colors on the web.
The good news with wide
gamut colors and the web is
The good news with wide
gamut colors and the web is
that as long as your image
content is tagged appropriately
and has appropriate color
profile information,
all of the content
is color matched
when it's rendered on webpages.
So as long as you do
that you're good to go.
Also, there are now
medial queries available
to resolve assets between
P3 and sRGB capable devices,
which is really handy.
And lastly, there's a WebKit
proposal out right now
for specifying CSS colors
in color spaces other
than the assumed sRGB,
which has been the case
since CSS's inception.
And with that I would like to
hand it over to Steve Holt,
who will tell you a little
bit more about rending
with wide color in your apps.
Thank you.
[ Applause ]
>> Thank you Patrick.
So, in your applications
you often have your assets
that you get from
the asset catalog,
but also you might get some
content generated by your users
or pulled down from the internet
from sources you don't control.
And maybe you want to
generate some additional assets
as part of that.
How do you deal with
this with wide color?
Well, when you're
drawing with wide color,
I'm going to take a
simplified example here.
Now, this is a box where
one-half of the box is rendered
with the most saturated
Display P3 red we can have.
And the other half of
the box is rendered
with the most saturated
sRGB color that we can have.
Unfortunately, because
these videos are recorded,
and you're watching it
live on a projector system
that doesn't have the
same color fidelity
as our new displays
have, we have to cheat.
So, we've actually
de-saturated both colors here.
So what you're seeing isn't
exactly what this code
that I'm going to show you
will output, but render it
on a device and it
will be there.
So, I'm a Frameworks Engineer on
the Cocoa, or on the UIKit Team,
so of course I'm going to start
with Cocoa on the desktop.
so of course I'm going to start
with Cocoa on the desktop.
When you're on the desktop,
it already has some great
color management support
for color profiles in your
images and in your drawing.
If you have to do manual drawing
in your code, and it needs
to be done off screen,
the recommended way
to do this is using the
NSImage with drawinghandler API.
It's called with
the current context.
And you can use this anywhere
you use any other NSImage,
it will just work there.
So, let's look at some code.
So, pretty simple,
just set up a size
and we initialize our
NSImage with the size we want.
We don't want this
to be flipped.
And we configure
our drawing handler.
We get the Rect from the
drawRect that gets passed in.
We divide it up in halves,
allocated displayP3Red
and draw it.
allocated displayP3Red
and draw it.
Then allocate our sRGB
red and then draw that.
And then return true,
because we true successfully.
And we're done.
There you go.
It all just works.
If you're using this in
your application today,
you're already doing
the right thing.
So, that's pretty
much the desktop.
What about iOS?
Should be just as simple right?
So you must all be familiar with
UIGraphicsBeginImageContext.
It's a nice little API.
And, let's see the code again.
We get the size, we made
the context with the size.
We divide our Rects, draw our
P3 color, draw our sRGB color.
We get our image and
then end the context.
And this is what we get out.
That's not correct.
Why didn't this work?
Well, if we look
at the documentation we
have this fun little bit.
We've documented that
BeginImageContext API is
in the UIKit will only
every return you a 32-bit
integer context.
That means you only get 8 bits
per every channel per red,
green, blue, and alpha.
This has a pretty big impact
on what we can draw with this.
Since we can't create
contexts that are larger
than 8 bits per color channel,
it can't represent any colors
in the extended sRGB
color gamut.
And, because of the existing
interface written in C,
well we can't expand
the options.
And we didn't want to make
UIGraphics be an image context
with more options.
So, we have some new
API for you in iOS X.
It's called
UIGraphicsImageRenderer.
It's called
UIGraphicsImageRenderer.
And the way this works is
really straightforward.
So I'm just going to jump
into some code with this.
Here we are.
First we instantiate our
image renderer object.
You can reuse this.
And it retains the
same properties
across every implication
of the image you get
out with the drawing handler.
And you just call
renderer image.
You supply your block,
and you just draw
as you would otherwise.
The bounds you can
get off of the format
which describes the
exact properties
that this particular
renderer has.
And then you just
divide the bounds.
You get and render
the P3 red square.
And then you configure and
render the sRGB red square
and just like we had
before, in AppKit,
we have a correctly rendered
extended range half P3,
half sRGB red square.
So just a few other
notes about this new API.
So just a few other
notes about this new API.
It is fully color
managed by default.
And it also supports the
extended range sRGB color space
by default.
And it's pretty smart
about this.
If you have a device
that has a P3 display,
like the new iPad Pro 9.7 inch,
we'll just turn it
on by default.
You don't need to do
any additional work.
But if you're on a device
that does not have the
extended color display,
like every other iPad and
iPhone, then you're going
to get the more standard
8 bits per channel,
sRGB context to draw into.
So we don't waste
the additional memory
for having the 16
bit float context.
The other thing this does
for you is it manages
the CGContext lifetime.
You don't need to worry
about ending the context
or any other management
yourself.
And for any legacy code that
you have in your application,
it works with
UIGraphicsGetCurrentContext.
it works with
UIGraphicsGetCurrentContext.
So if you call into some
function like a draw Rect,
you'll do the right thing.
So, that's rendering
off the screen.
What about to the screen?
Okay we'll start
a new iKit again.
On [inaudible] view we
have the new Swift 3, well,
newly named in Swift 3, Draw.
And we do the right
thing for you here.
When you call draw in
your UI view subclasses,
if you are on an iPad 9.7 inch,
we will call your draw method
in the correct, extended
range sRGB color space
with the floating
point pixel format.
And if you're not on one
of the new devices, then,
well you get what you get
now, 8 bits per channel sRGB.
And of course if you have
images, we have UIImageView,
which has been color managed
since iOS 9.3 and continues
to be color managed today.
If you need to know how
you're rendering, you can look
at a new trait on
UITraitCollection in your views
in view controllers
called display gamut.
It takes a UI display gamut
enum and its properties are P3
if you're on a new, P3 class
display or sRGB if you're
on a new sRGB class display.
And this can be very useful
in the case where you need
to match a particular UI
color with an asset pulled
from your asset catalog.
So you can properly match
colors to your assets.
Now, what if you know
that a view is never going
to need the extended
range sRGB context?
Well, there's a new
property on CALayer
and that is contents format.
And this controls exactly how
deep the context that CA creates
for you when it renders
that view will be.
for you when it renders
that view will be.
By default, on the
iPad Pro 9.7 inch,
it is to use the
extended range context.
And on all of the
devices the default is
to use the traditional,
sRGB context.
And you control it with any one
of these various
format literals.
So on Cocoa, back
on the desktop,
NSView, same thing as UIView.
It will draw with the current
context that NSWindow has.
And you can look at any of the
traditional properties there
with the Window Backing Store,
the screen target
profile, etcetera.
If the window changes displays,
then you get the
view did change back
in properties callback
on your view.
And you can also listen
to the NSWindowDidChange
BackingPropertiesNotification,
which is omitted
from your view's window.
These are the same that
you've been working
with since a very long time now.
And just as a refresher.
The backing properties
on the NSWindow include the
display scale, color space
and the output display gamut.
And just like in iOS, you
can also control how deep
of a context you get in AppKit,
with the WindowDepth property
on NSWindow, and you
can set it to any one
of these depth properties.
So if you are on a wide
gamut display but don't need
that extra precision you can
set this to be 24 bit RGB,
and it will not create the
extended context for you.
So, what did we learn today?
Well, you've seen our vision
for wide color and how to bring
that to the next generation
of displays and your users.
We've talked a little
about color gamuts
and color management,
and how to work
and color management,
and how to work
with that wide gamut
content that you want
to bring to your users.
We've reviewed how to use
colors in your application.
And how to take your drawing
code to the next level of color
by making sure that
it draws properly
in these wide gamut scenarios.
Now for more information you can
of course check out the slides
on the session's website.
And there are a large
number of related sessions.
This is page 1.
So please check them out.
If they've already
passed, I'm not going
to enumerate them,
there are a lot.
See the videos through
the WWDC app
or online at the WWDC website.
Thank you.