WWDC2004 Session 207

Transcript

Kind: captions
Language: en
I'm Travis Browne in the graphics
imaging evangelist and I helped to put
together this for graphics media track
along with help from obviously the
engineers who've created the great
technology and content and I'm sort of
generally your host for a lot of things
going on with graphics and media here at
WWDC the one they wanted to take a
little bit of time is to talk about some
big sort of shifts that are happening in
graphics and particularly the tiger
timeframe and you've noticed they've
been several mentions of the fact that
now we're leveraging floating-point
pipelines and for example the GPU we're
also have floating-point pixel support
and quartz 2d and also new technologies
like core image also are based on
floating-point pixel operations and this
is sort of important shift in graphics
because it used to be that you know
eight bits per component was plenty it
was enough it was the ideal 16.8 million
colors encapsulated at all but as the
sort of imaging market has matured it's
really become obvious that we need more
bits per pixel we also need them encoded
differently and this raises new
challenges that we need to solve inside
the ls not only to be able to pump those
through our graphics subsystem but
additionally be able to do things like
read and write them to to disk so this
session we're going to be talking about
two topics and specific one is high
dynamic range which is going to
basically cover several techniques in
terms of how deep pixel data is encoded
put on disk available file formats Plus
also once you read that deep pixel data
back and you actually make it viewable
what what you need to do to it to
actually you know basically create an
image you can display on your monitor
and then secondary to be talking about
the reciprocal changes in the operating
system the technology such as image i/o
which is an imaging library which is
going to deal with these new data
formats and also changes in areas such
as colors ink and color management
because these changes all together sort
of complete the picture they're going to
Nabal to you to actually step beyond
eight bits per component and start
leveraging the new fantastic
capabilities inside the GPUs are imaging
stack to really do new and interesting
things with high res
illusion high fidelity data on that note
I'd like to invite David Hayward this
stage to take you through the session
thank you thank you travis with
introduction and thank you all for
coming today session on high dynamic
range imaging with image I Oh what I
want to talk about today and what you'll
learn is that the new exciting emerging
field of high dynamic range imaging and
how you can take advantage of it today
in tiger using a new facet of quartz
called Amy Jo but before I talk about
those two fields and the people who'll
be coming up and talking about it in
more detail I want to give a brief
interrupt eight on what's new in color
sync for tiger because color stink is
one of the key pieces of technology that
allows for the proper rendering of both
standard and high dynamic range images
so let me give an update on color sync
for tiger will be talking briefly about
adding floating-point support in color
sync use of core foundation types some
API changes will be making some notes
for developers of custom CMS and some
changes to color sync utilities user
interface so first and foremost is
floating point support the Travis
mentioned earlier one of the things
we're trying to do for Tigers provide a
new high fidelity cinematic graphic
environment for tiger and in order to
achieve that we need full floating-point
support throughout the entire system and
one key piece of that is color stink so
in order to achieve this first thing we
needed to do was to have a new bitmap
structure in color stink for supporting
arbitrary bitmaps of floating point data
that your application can pass to us we
wanted to make the structure as flexible
as possible so that you wouldn't have to
repack the data before you send it to us
so this new structure supports both
chunky and planar arrangement of data
and also allows for the channels to be
in any arbitrary order the way we
achieve this in the structure is it's a
little different from other bitmap
structures you may have seen is that
instead of having a single base address
for the all the pixel data we actually
have a different base address for each
Channel this allows for the channels to
be in any order and then we also allow
for both robots and column bytes to be
specified which allows for you to have
your data be scanned
in reverse order if needed or also if
you've got unusual packing between
channels that you want to make sure to
skip over so it's a fairly basic
structure but in most cases most people
will be passing in a buffer of chunky or
interlayer interleaved data and so we
provided a simple utility function
called CM float bitmap make chunky which
you supply at a single base address and
then it'll fill on the structure
appropriately for you in either case
whether you fill in the structure by
hand or whether you call this helper API
once you have a source and destination
float bitmap you can then call color
sync to match data from one space to
another we have three functions to do
this the first one is CM convert XYZ
float map which allows you to convert
between all the cie related color spaces
so x y z yxy la d and l UV and also is
another function which is convert RGB
float map which allows you to convert
between the RGB derives spaces RGB HSV
and HLS these both of these functions
are based on textbook formulas and so as
a result there's no need to pass in a
profile or color world to do the
transform it just does the math for you
with floating-point precision the last
is probably the most interesting is mrs.
the new API CM match float map which
allows you to pass in a color world
reference to perform the actual
transformation you can create the color
world by concatenating one or more
profiles and then at that point the data
will be sent through the CMM which if it
supports floating point data will be
done in full floating point precision
one of the other changes we've made to
color sync is integrated in more closely
with the core foundation types and the
key way we've done this is that the two
common colors think opaque data types
which are the CM profile ref and the CM
color world rest are now CF types and
this is quite convenient because it
means that you can now call the CF base
functions such as CF retain and CF
release it also means you can add
profiles in color worlds two
dictionaries or arrays this is kind of
handy if you're passing profiles and
dictionaries around to other parts of
your code the other way that we
supported core foundation type is to
actually get the data out of a profile
one of
questions I often hear from new users to
color sink is I've got this profile
reference how do I get the data out of
it in the past that was done with by
calling either CM copy profile or CM
platen flattened profile now it's much
easier you can just call CM profile copy
icc data and it will return you all the
data within the profile as one giant ICM
CF data type ok next thing I want to
talk about are some API changes that
we're making for Tiger way back several
years ago one of the features we added
to color sync both at the API and the
user interface level we're a set of
preferences so that applications could
have one place to go to for specifying
default profiles based on usage or color
space and at the time we hope that this
would be a way of simplifying the user
interface across a wide variety of
applications and so we presented both
API and user interface to help with this
in practice however it turned out that
very few applications have used this API
and what we're left with is the user
interface in the color sync utility
where people say I can't figure out what
this does because nothing I'd change
here make seems to make a difference so
we're listening to the usage and we're
actually beginning the process of
deprecating the API and also the user
interface so we still want the any
applications that we're using this API
to function correctly so what we're
doing is changing the behavior of CM get
default profile by space CM get default
profile by use and CM get preferred CMM
instead of storing their preferences as
a setting that's global across all the
Machine it will now be stored in the
current application current house
current user domain so the api's will
still function but we're deprecating
them this house and the ramifications
also in the UI which I'll talk about
later I also want to make a take this
time to talk a little bit about custom
CMM one of the things that we've been
doing over the last few years is making
even tighter and more powerful
integration between the graphic system
as a whole notably courts and printing
and color management and in order to
achieve this with high performance and
high reliability
we have made made it so that courts and
printing will only use the Apple CMM
that said we have a long tradition of
allowing applications and other other
developers to develop their own CMMS and
for applications to call those as they
wish it's still possible for
applications if they wish to have a
custom CMM and for them to explicitly
create a color world using that CMM and
that can be done using the recommended
API for this now is NCW can cat color
world and this this API has an easy
convenient way for you to specify which
CMM to use the other thing to mention
for CMM developers is that there's a new
entry point for cmms which is CMM match
float map if your CMM supports this then
you can have a full floating-point
support throughout the rest of the court
system and if you don't support this
then the data will be truncated to
16-bit integers and everything will work
sufficiently lastly I want to mention
some changes we're making the color sync
utility as I mentioned earlier we're
deprecating the Preferences api's for
default profile and one visual
manifestation of this is that we were
removing the user interface from color
sync utility however we're adding
something in its place where I'll be
adding a new utility to the colors link
utility which is we call a calculator so
let me give a brief demonstration of
this on demo to rear so as we see in the
color sync utility everything looks
similar except for there's no longer the
preferences pane as the first item but
we have a new item which is calculator
which provides a very simple way to
convert color spaces between in all the
various different color spaces using
floating-point precision this is a
convenience that also provides a good
way to demonstrate our floating point
data pad so obviously we can specify our
source color space in our destination
color space if we're just converting RGB
to HSV we can see the slider values we
can update the sliders on the left and
they update on the right one thing
you'll notice is because the RGB and HSV
are related color spaces their basic
formulas for each other so as a result
the color on the left will be the same
as the color on the right if we switch
to CMYK you'll see something a slightly
different which is now it's going
through a profile and if I go to a
saturated color you'll notice that the
color on the right is g saturated one of
the other things we added is the ability
for it to be fully symmetrical so now
instead of just updating on the Left I
can also update on the right and it'll
show you the values in that order we can
also this is an interesting way to test
out a CMYK profile we can specify that
we want to input la b values and output
to CMYK and as we scroll through all the
possible la b values we can see what the
resulting CMYK values will be so that's
the brief demo of the color calculator
we hope that's a useful function so back
to slides so the next thing I want to
talk about is something that's all new
for tiger which is this new facet of
quartz called image I oh and again as
Travis alluded to earlier we wanted to
provide a new API for doing image
processing or image reading and writing
from a variety of formats and this is
image aya we talking today about its
features its goals what formats it
supports the clients of this API some of
the core concepts you need to understand
for using this API and some advanced
techniques as well so what are the
features of image I well first is we
want to be able to read a wide variety
of file formats and write to a wide
variety of file formats we also want to
support reading and writing metadata and
also we want to support incremental
loading for clients such as web browsers
that get data an incremental fashion
over slow data connection we also want
to support floating-point support
because that's one of the key
initiatives for graphics and in Tiger we
also want to have broad color space
support and something called cacheable
decompression mentioned a little bit on
this now which is typically different
api's for reading and writing image file
formats have one of two behaviors in
terms of decompression in the case of
the existing core graphics AP is every
time you draw the image it's fully
decompressed each time this obviously
has the advantage that you have very
little memory overhead but it's a
performance hit if you draw the image
more than once other api's have the
behavior
that the first time you draw the image
it fully decompresses it which obviously
requires more memory but has the
advantage that subsequent draws will
perform quickly there are merits to both
approaches and so what one of the
approaches we've used with image i/o is
to try to allow for both features not
all file format support both approaches
but we were ever possible we support
both philosophies here are some of the
overarching goals for image iya first
foremost with the reduced code
duplication turns out there was an
embarrassing number of different
variants of JPEG readers and writers and
tiff readers and writers within our
system and they all had different
strengths and weaknesses and if you were
actually trying to write an application
that read and wrote images you had to
make a choice between which strengths
and weaknesses he wanted to use we
wanted to have a single reference
implementation within the system use
that use that means many places as
possible so that we have a single place
to make changes in the future one of the
other goals is willing to leverage open
source so that our havior of our api is
consistent with other implementations
and improved performance this is one of
the other key things we've been spending
a lot of time with the vectorization
team at Apple to make sure that our key
file formats decompress with optimum
speed another feature was lazy
decompression in the sense that if all
you need to do is get the height and
width or metadata out of an image you
shouldn't have to fully decompress the
data so we want to support that as well
and lastly we wanted to make sure we had
a very modern core graphics friendly and
easy to use API so that you could all
easily adopt this in your applications
so one of the first questions I always
get when I'm talking about image i/o is
well what formats you support and we
support all the standards for the
internet TIFF jpg ping gif and JPEG 2000
these are already supported on the
developer CD that you got this week
we're also supporting some exciting new
formats such as some high dynamic range
formats such as openexr radiance and
some important variants on tiffs such as
log lu v and some pics are variants
there's also countless other formats
we're going to be supporting BMP PSD qti
f SGI icns files
and we're considering more both for
tiger and beyond so the clients for
image I obviously we hope that anyone
who wishes to use this API are free to
use them in their application but
there's also lots of places within the
system that are going to be calling
image aya so you may get the benefits of
image era without having to change your
code at all probably the first and most
important client is for image i/o is a
preview application this has been a
great example of how how the power of
the new image I oh and some of the
advantages and get from its making
strong use of this new API also app kit
will be switching over it's not yet
switched over in the current developer
release but app it will be switching
over to using API the new image i/o API
as well WebKit and its clients such as
Safari mail and any of your applications
that are using web kit will be using
image io core image is using image I owe
to load data and floating-point format
spotlight is using it for generation of
thumbnails and getting metadata and some
of our scripting technology such as
sifts and image events are also using
imagej oh so we're trying to use this
everywhere in the system this should I
want to give an outline on the API in
image I oh but before I do that I want
to talk a little bit about how images
are organized so you can get an
understanding for why we designed the
API the way we did in previous systems
the standard way of representing an
image in core graphics was with a CG
image rep and this is a great basic
format for representing images it allows
you to specify three things the geometry
of the image such as its height with
robots and pixel size the color space of
the image which can be a profile or
other equivalent description of the
color space and the actual pixel data
this is the minimum information you need
to describe an image however it turns
out that there's a lot of file formats
out there and they are actually quite
elaborate in many cases and so one of
the things we want to do support an
image i oh was a richard model for
images for one thing we want to be able
to support thumbnails and metadata free
images and also a lot of file formats
support multiple images within the same
file format such as tiff so we want to
make sure we support that as well
and also there's a set of attributes
that apply to the image file as a whole
rather than to the individual file
images contain within the image file
this is the file format of the image
such as it's whether it's tiff or jpeg
and also some properties that apply to
the file as a whole for example tiff
files can be big endian here's an
example of how this works in practice
we're using an example of a tiff file
the file type is public tiff which is a
universal type identifier that describes
this image as being of the type tiff we
have some properties that apply to the
file as a whole the file size and bytes
for example in the Indian asst of the
tip and then we have the standard
information for each image such as its
height width its color space it's pixel
data its thumbnail as possible and its
metadata such as copyright and artist
information you name it so here's how
this model is reflected in our API
through data types what we have is we
use the existing CG image ref to
represent the geometry color space and
pixel data the thumbnail is also
represented by SVG image rest the
metadata and the file properties are
represented as key values in a CF
dictionary f so it's a little bit all
very simple so now I can talk a little
bit about the API what we've added is a
new data type called CG image source and
this is the opaque type used for reading
images from either memory or disk you
can create a CG image source from either
a CF URL ref CF data or with the CG data
provider once you have a CG image source
you can query the image source for
several attributes you can ask for the
properties of the file as a whole using
CG image source get properties you can
ask for its file type by calling cgma
source get type you can get the count of
images using CG image source get count
once you know the count of images you
can then for each image ask for its
image you can ask for its thumbnail and
you can ask for its metadata so it's
pretty simple just to show you how this
works here's a little code sample that
shows you how given a URL to get the
first image out of the file it also
returns some simple metadata in this
case it's just returning the
the DPI of the image in the horizontal
and vertical direction first thing this
code does is call CG image stores create
with URL which creates our data type for
subsequent access to the file then what
we want to do is we want to get the set
of properties for the first image so we
call CG image source get properties at
index and that returns a dictionary we
can then query that dictionary to see if
it has the GPI height and width
properties and return those to the
client lastly we need to actually return
the image so we call CG image stores
create image at index and that will
return the image to the caller here's
another example for getting a thumbnail
out of an image image IO is very
flexible for creating some nails as it
turns out some file formats support
thumbnail some don't also with some file
formats thumbnails can be quite large
your application may need to have
control over how thumbnails are returned
and we provided that with the image io
API be an options dictionary in this
case what we're doing is we're again
creating a CG image source by specifying
a URL and then we're going to be
creating an option to dictionary with
two key value pairs in it the first q is
CG image source creates thumbnail from
image if present what this does is tell
Amy Jo that even if the image doesn't
create a thumbnail return the actual
image instead so we'll always get an
image for the thumbnail the second key
value pair we specify is CG image source
thumbnail max pixel size and this allows
us to make sure that thumbnails are a
reasonable size which is especially
important if you've specified the
previous option so in this case we're
saying that we always want an image to
be returned and we want it to be no
bigger than 160 x 960 pixels once we've
created that dictionary all we do is
call CG image source creates thumbnail
at index specifying the image source 0f
index and the options dictionary and
it's returned this is for example the
way that the spotlight technology
creates thumbnails for images in the
search results field so that's the
basics of reading from an image I oh
here's what we do for writing we have
another data type which is CG image
destination which can be created with
the CF URL CF mutable data or with the
CG data consumer at the time of creation
you also specify the type of
file whether it's a jpeg or tiff for
example and the capacity or the number
of images that that image will hold once
you have a CG image destination you can
specify the properties for the file as a
whole using CG image destination set
properties and then you can repeatedly
add each image with various options and
metadata at the same time using CG image
destination add image lastly you could
flush the file out to either the URL or
to the data by calling cgms destination
finalized and that returns true if the
image was successfully flushed again let
me give a short example just to show how
easy this is to add to your application
we have a function called right jpg data
which takes a URL and an image to right
and a dpi to specify in the metadata
first thing we do is we create an image
destination with a URL specifying that
it's going to be a type jpg and that
it's got one image in it next thing we
do is we specify a dictionary with three
keys and values for options and metadata
one option that we're specifying is the
quality of the JPEG and that's specified
with the key kcg image property equality
in this example we're specifying a
quality of point eight or eighty percent
compression the other two key values are
for metadata and they are the kcg image
property DPI wits and DPI height in this
case we're just creating CF numbers
based on the value that was passed in
once we have this dictionary then we
call CG image destination add image to
add the image and its options in
metadata to the CG image destination and
lastly we call CG image destination
finalized to write the file to disk so
it's pretty easy so those are the basics
of the image I oh I hope I've given the
impression that this is a very simple
and easy API to add to your application
and again some of this benefits you'll
be getting for free if you're using
applicant and other technologies let me
talk for a minute about some of the more
advanced techniques that come up when we
talk about image reading and writing
such as extracting a RGB data requesting
the depth of an image and loading an
image incrementally
so one of the common questions we have
is well I haven't an image has been
returned from image I oh but it's I
don't know what color space it is I
don't know what depth it is I don't know
what pixel format it is and i have an
application that only works in RGB
that's a common scenario and this is an
interesting piece of code that makes it
very easy to convert the data matter
what format it came in into a RGB
basically the technique is to use a CG
bitmap context to render the original
image into an off-screen and one
advantage of this is that it takes care
of all the color management correctly if
the image happen to be an l a b or CMYK
image and had a profile then it'll be
correctly color managed to the RGB
colour space that you're working in
another interesting question is the
depth of image some formats only support
one pixel depth for example JPEGs are
always eight bits per sample other
formats can support arbitrary pixel
depths the rental tips can be 1 2 4 8 or
16 bits per sample as a rule the image
returned by image io will be the same
depth as that indicated by the file so
if you open a 16-bit tiff file you'll
get a 16-bit cgm address however in the
case of high dynamic range file formats
it gets a little bit more complicated
the data in these file formats are
typically encoded in special encoding
formats which can then be decoded in a
variety of ways they can either be
unpacked two floating point values
either 32 or 16 bit formats or two
integers with 16 or eight bit precision
also in the decoding process that you
can either be left as extended range
values or they can be compressed to the
logical 0 to 1 clipped range both of
these are reasonable types of values to
be returned and your application may
want one versus the other by default cg
image io will return an image ref that's
compressed to 16-bit integers this will
give the best results with reasonable
memory for the typical application
however if by request an application can
specify that they want the floating
point unprocessed data returned here's a
brief example that
is how to do this this is a code snippet
that given a URL will request that the
data be returned in floats and if it is
it'll the data is actually returned as a
float a boolean will be returned to
specify that it was actually floats the
way we've done this is as you've seen
from the previous examples we create an
image source and we specify an options
dictionary which has as one of its key
value pairs CG image source of maximum
depth with the value of 32 at this point
we can then ask image I owe to get the
properties of the first image given
those options and this will return a
dictionary we can then query that
dictionary to see if it has floating
point data or not then lastly we can get
the image and return that to the client
another advanced techniques I wanted to
make sure people knew that we supported
was incremental loading his images I
won't go in too much detail on this but
the basic idea is that you create an
image source in an incremental fashion
using CG image source create incremental
and then you repeatedly add updated data
to the image source each time you add
data you can request a new image and it
will give you a partial image or
complete if the image is fully loaded
the and then it when you once you're
done with the image you can release it
and then once you've added more data you
can get a new updated image it's
important that you release it before you
ask for a new image so let me give a
brief demonstration of image I oh and
action so one of the things i want to
show first is the new preview and i've
got a bunch of images here open
and one nice thing in preview is you can
open all the images just by selecting a
folder and I've got a variety of images
in here one of them is an LED image and
we can do that but we can verify that it
is an lav image by going to tools get
info and this shows the metadata that's
been obtained using image I oh and we
can tell in here from the metadata
that's currently returned that the color
model is la b we have a variety of other
images we can zoom in and zoom out the
thumbnails over here were obtained using
image aisle as well we have high dynamic
range images here you can zoom in and
zoom out on that Luke later will show
how we can manipulate these images in
real time here's another interesting
example which I like to show people this
is one of our things that we use for
testing oftentimes people want to know
well how do I know if the profile is
being used what I have here is a
document that's a black-and-white CMYK
document that has a profile on it that
makes values that are gray disappear so
if this image were rendered and the
profile were ignored what you'd see is
the text the prophetic test profile is
not used and that's because you can't
see it here because the profile is being
used but there's actually a gray word
not right here so it provides an
interesting test that you can tell if
your profile is being respected or not
here in this gray version you can kind
of see a little bit of the hint of what
was once there and the word not but this
is a great way of testing images we
really should distribute these at some
point one other example of using image I
oh I have a test application which shows
some of the options so I'm going to go
to open one of the images we just saw
with the desktop images and open up this
image here we can see some information
the height and width and how long it
took to draw we can one one thing we can
do is we can specify that we'd like to
see what this would look like if it was
progressively loaded if I open up
another image if i open the high dynamic
range image this is a big image
unfortunately so it takes a couple
seconds to open
we can if we bring up the metadata on
this you go to window metadata we can
see that it has height and width and its
depth of 16 this is because by default
we return 16-bit integers however if we
want to return it as 32 and again
they'll take a second or so this was
still need this code still need to be
altivec someday soon we need to bring up
the metadata and hat now we can see that
there's a new property in here which is
saying that data is returned it floats
so that's the introduction to image I oh
I'm going to pass the microphone and the
demonstration and all the new stuff over
to Luke Wallace who will be talking
about high dynamic range imaging thank
thank you David so today I will be
talking about Mac os10 support for high
dynamic range imaging which is a new and
exciting feature that we are adding into
the tiger release as many of you know
high dynamic range imaging is generating
a lot of interest and is still a subject
of very active research so we could talk
about high dynamic range imaging from
many different points of view but what I
would like to do today is concentrate on
answering very simple three questions
what is it why use it and how to process
it before we try to answer this question
let's take a quick look at the current
status quo in digital image processing
we can conclude that in majority digital
image processing is dominated by the
what is called output referred approach
what it means is that the requirement of
image reproduction are imposing certain
requirements on the way we acquire and
create images and because most of the
devices we are dealing with like
displays and printers can only handle
8-bit data through color channel we
impose the same requirement on digital
cameras that in fact could produce much
more about an order of magnitude more
data if they were not restricted to that
requirement obviously there are some
advantages this is not done for no
reason the main is that there is a very
minimal image manipulation required
before displaying or printing such an
image but obviously there is an
disadvantage that we are losing a lot of
color and image information that could
be used in further image processing that
could result in much higher
quality of display or fringe oops sorry
infirm direction another requirement
which is sort of hidden in the output
referred approach is that the data is
exchanged in one predefined color space
and in the most difficult case this is
srgb so when you look at this slide you
see I drew the shape of the typical
exchange color space made it be srgb
that color space covers only a part of
visual gamut so everything is fine as
long as the camera is acquiring the
color data we send that triangle but if
we are outside then we are out of luck
we have to do something with this color
and typically we have to push it into
the color space can be done through
different methods but because the
cameras are not very sophisticated in
terms of processing power we are using
very often clipping and as we know from
practice gammas clipping can produce
really bad results like for example hue
shift and here is one of the maybe a
little bit stronger and exaggerated
example of what could happen but this is
a real clipping that in which the white
color because of clipping became a
mixture of completely unrelated colors
so when that is what we can conclude
when we look at the image processing
from the point of view of device
capabilities to reproduce the image and
what I would like to do now is to look
at the image processing from a little
bit different perspective from the
perspective of human vision and as we
know from the very rich research in this
area color and visual acuity are to the
most important are the most important
characteristics of the scene and not
only this these to depend on luminance
and observers visual adaptation we know
that we can measure the war
of luminance and it will cover the range
of the values between 10 to power of
minus 6 all the way to the power of 10
to aid in when measured in Candela per
square meter but what is important for
us that different ranges of luminance
create different illuminations now and
that illumination also can stretch all
the way from very dark environments to
starlight all the way beyond the
sunlight and now I could spend a lot of
time talking about physio psychological
and physiological mechanism mechanism
controlling our vision but what I would
like to do without going to those
details to say that humans have three
types of vision which are dependent of
the type of luminance we have scotopic
vision which works when we are in the
dark environment we have mossop excision
which works in light dark environment
and finally when we are in high
illuminant illumination environment we
switching to photopic vision why this
decision is important because our
quality of vision is related to this
type of vision as we know if we look at
something at the very dark environment
we have no color vision and very poor
acuity everything in the darkness seems
to be just a shade of gray on the other
hand our best vision is in this photopic
range where we can see many colors and
have a good color and visual acuity this
is not everything the what is very
important that human has a limited
simultaneous range which also depends on
the type of illumination and here I'm
showing the widest simultaneous range
which again existing this photopic
vision that can cover the range of
of order of magnitude three to four but
if we try to estimate this simultaneous
range in poor of vision that can the
values can drop by the order of two so
we may ask ourselves why this is all
important well I think there is an
answer because if we want to represent
faithfully the scene that we want to
process through image processing we
should have a mechanist to encode the
data the same way as or at least as
close as possible to the human vision
fidelity so now let's take a look we're
in this picture we can fit the typical
8-bit display and as we know the typical
8-bit display can cover the range of
luminance on the order of the order of
magnitude of two that is big discrepancy
between human simultaneous range and
dynamic range of a display so this is
the biggest challenge that we are facing
when we that we have to map the
relatively wide human simultaneous
ranging too low dynamic range of our
display device there is one solution
which we already know about this is the
output referred digital photography we
are imposed imposing the low result low
resolution low color of small color
space and the only thing we can do is to
choose between different options this is
a simplistic view in which we may say
well if I want to expose the details in
highlight I can use the short exposure
but if I want to see the details in the
shades i can sacrifice the details in
highlights and use the long exposure to
capture or what I wanted the most
important point is that this applied
exposure is permanent
once we burn this into the image there
is no way back so I think that at this
moment I will try to answer the question
what is high dynamic range and I think
that we can define high dynamic range as
a special encoding of the image data
which allows us to preserve the full
fidelity of human vision from the
implementation point of view the high
dynamic range imaging is based on color
values that first of all extend over at
least or four orders of magnitude that
can encompass the entire visible color
gamut and allow the values of outside of
a typical 021 range values of color in a
summary what it means that in high
dynamic range imaging we are no longer
limited to a specific color space we are
trying to encompass as I said all
visible colors but on the other hand we
need to remember that we no longer have
a convenient ready to display or ready
to print image high dynamic range data
requires some kind of manipulation
before it can be displayed but the big
advantage is that we can make this
decision at the moment when we need to
reproduce the image with our preference
instead of burning that to the image
this is a kind of simple explanation how
we can do that we can go back and select
the short exposure long exposure but
most importantly we can implement
something which was not needed before is
the tone knob rendering which will allow
us to achieve completely different
results for example here I can try to
combine in one image the details from
the highlights with the details from the
shade so now I'll try to answer the
question why use high dynamic range
images and the most important reason is
to preserve the scene referred
information that can be useful in
further image processing and this way we
want to avoid intermediate and coding
with
restrictive color gamut which was
happening in this previous approach
called output referred preferred and
also we can avoid irreversible
modifications that happen during the
image acquisition how to process high
dynamic range images the most and the
simple the simplest answer is that we
should not add any rounding or clipping
errors and for that we want to render
and capture the data in floating point
we want to store the entire image and
it's needed to process the color data in
extended color space which again will
not impose any clipping and at the end
we want to apply the tone mapping for a
specific image reproduction for example
that specific reproduction could be an
example I just showed you that I want to
see all the details in the image from
the highlights and shadows now let's
take a look at the file formats that we
are supporting in Tiger I think that the
most important citizen here is open exr
that comes from ILM first of all it has
the smallest quantization error and most
importantly as we will see later it
comes with the recommended way of tone
rendering which solves a lot of problems
in terms of presenting the image content
the other formats basically just defined
the way of encoding a decoding data with
preserving the image fidelity so now I
would like to show you if i can get my
this is demo2 machine my little
application in which i can open high
dynamic range images and what i would
like to show you is that we have to do
something with those values which are so
large and much bigger than what we can
represent by typical range of 0 to 1 and
one would think that
very simple approach would be to simply
mob the brightest point in the image to
the brightest point of the display but
if I do that with my little up demo
application you see that we don't see
too much in this image there is way too
much information beyond one and scaling
didn't produce any visible image another
very simple approach could be okay let's
say I would like to see whatever you
have in this image clip the values 201
typical range and show me that well as
you see the image quality somehow
improved but it's still very poor and
now if i use the open exr and this is
their default zero exposure value i'm
getting some reasonable result and i can
see many more details and not only this
I can do what I was talking about it I
can impose my preference at the moment
of reproducing the image for example
someone may like this kind of image or
and someone else may still want to focus
on this beautiful stained glass I want
to show you a couple of classic examples
like for example the same as memorial
church picture which comes from the
deviled egg website and the same thing
happens here if we just scale the image
the image the dates the image is
basically unreadable clipping will show
something that quality is really poor
openexr is doing a very good job here
another example is the picture I was
using in the previous slides of the our
garage at apple and this is how it looks
when scaled this is how it looks when
clipped once again typical huge shift
when clipping the data and open exr
producing quite reasonable result what
this leads us to the conclusion that
tone rendering is a very important issue
when processing the high dynamic range
images and maybe they
many different methods of doing that and
I think this gives me a very good segue
to introduce Gabriel Marcoux who will be
talking about high dynamic range tone
mapping developed at Apple thank you
very much for the introduction as you
have seen the main problem of rendering
high dynamic range images is how to map
the high dynamic range how to reduce the
high dynamic range into a low low
dynamic range or device and this problem
is not trivial so we have look into what
is available in the published literature
and here i put a list with few methods
that i select from this from what is
available and the site from openexr you
can see the histogram adjustment
proposed by one retina Calgary's this is
a class of methods and good review on
this method is proposed is published by
a Jamaican electronic imaging conference
in 2002 another interesting approach is
based on the color appearance models and
using one of these models which is icon
Fairchild and Johnson try to reduce this
high dynamic range gamut high dynamic
range imaging to the displayable won the
last three in this list are proposed at
siggraph 2002 and they are bilateral
fast filtering doujin burn method and
the gradient compression you can see in
this list that you can group the methods
in two classes one of the classes
general algorithm that are applied in
a bit images to get more information
from them and reduce the global contrast
to something that is visible on this on
the display and the other second class
of algorithm as P our algorithms that
are specifically designed to handle high
dynamic range images while doing this
research of this available methods we
came up with our own algorithm ention
the apple method which i will
demonstrate in a minute one of the
things that you face when you try to
design or even evaluate a method is what
is the intent of that method aside from
just compressing the high dynamic range
into the dynamic range of the display
and here is a flavor of to kind of
rendering intense one that is retaining
the look and feel of a high dynamic
range and you can see this on the image
on the left and another one that is
showing as much content from the image
and you can see that comparing these two
images around the highlight region in
the image that retain the look and feel
of of a high dynamic range image you see
the glowing around the window while in
the other you see more detail more color
of the stained glass of that sealed
window ceiling window and you can see
this the same thing happening on the
window on the wall and it depends on the
subjective preference of the user which
of these settings will prefer to to use
the good thing is that with upper method
we are allowing both intends to be shown
so I will do a demo on this and I will
show you how this
algorithm is working so we open the
viewer and with this we can choose
between different images let's start
with a memorial image with that is let's
say a classical one which is pick up as
an example by a lot of algorithms it is
a good thing because you can compare
algorithms on the same image this is the
rendering of glowing the intent of that
is showing the glowing around the
windows and around the highlight in the
image I also show here the high dynamic
range size which is a good metric for
that is the logarithm of the maximum
over the minimum values in the file and
this give us three stops or 13 stops and
you can see on the bottom the actual
range of floating values in the file I
said that we can support both kind of
intense and right now I switch to the
intent where you can see the color of
the glass in the side in the stained
glass of the window so we can open a
different kind of different kind of
files not only from one source and you
can you can evaluate this the power of
this method this is an image that is
from openexr user by the opening acts
out of the test image and it's quite
interesting it has 18 stops so it's a
quite large high dynamic range and you
can see details in the shadow you can
see details in the highlight also you
can look at the text that is on the book
and see the details under the table as
you have noted I haven't changed any of
the default parameters of this method so
our intent was to actually design a
method that where the user will
have us less intervention as possible
and will be allowed to take the default
parameter and get decent results with
this settings so i choose also other
image from another source nave one which
is the image with the highest dynamic
range that i was able to find publicly
available and you can see that this
image has 22 stops one stop being
meaning that doubling the range of the
values in the file so if the difficulty
in rendering this image is around this
window where you can you are required to
show colors of the window and also not
to show visible artifacts around like
ringing or or darkening the area around
the last image that i will show is the
image that an image that we create
ourselves with different algorithms
image and you can see details from
outside and inside and details in the
shadow and in the highlight eventually
with high resolution image you are able
to see when the fluorescent tubes on the
top so this is a demonstration of the
ATR viewer this brings me to the next
topic that i would like to touch which
is the creation of high dynamic range in
in summary about the viewer you can see
that with default parameters who are
able to open images from different
sources and this method is quite robust
in the sense of showing a very good
results from different high dynamic
range images how do we create high
dynamic range image images is the next
topic and this is quite interesting we
can start with the file format of these
images and we have to
code in RGB floats the radiance of the
scene so how do we capture this we turn
to a method that is was published by a
dybbuk and malik recovering high dynamic
range radiance maps from photographs and
essentially this method is requiring to
take multiple shots of different
exposures or of the same scene and then
combine these exposures into a high
dynamic range file we start with the
block diagram of the digital camera if
you look closely you can see that the
scene radiance it is transformed to the
digital output in the in the digital
file which is maybe jpg or other file by
a set of transformation first the image
is passing to the lenses then the
shutter then the image is captured by
the CCD and converted by a DC converter
and then some mapping is happening in
the camera for example gamma correction
or raw image to jpeg image
transformation and you finally get the
digital values in the file because the
scene radiance is in direct
correspondence with sensory Radian so
the first block can be skipped and we
can group the last four blocks in a
single entity which can be described by
a transfer function so we get the output
digital output in the file as a function
of sensory radians II and the exposure
time DT with a little math on this
applying the inverse transformation and
low transformation we can recover the
sensory radiance from the digital output
and the exposure time this is easily
possible the only thing that we need is
the middle term in there which is the
mark in blue and which is referred as a
camera exposure function one once we are
able to derive this function then we use
this equation to immediately find sin
radians so we will concentrate now on
deriving this function that we call in
here in the next slides with G so this
function described the exposure has the
exposure dependency on the output gray
level the idea is to take multiple
exposures and then to pick up a grade
level in one of the exposure which is
that white ring in there in the first
image and then look at how the gray
level is changing from one image to
another so we keep the same position
same x and y in the image and we look
and plot the variation of the gray level
from one image to another this will give
us a curve that is showing up like in
black in the diagram we can do this for
other grade levels in the image and we
can recover several curves and finally
we end up with a set of curves that
described this variation of the gray
level and what we need in the end is to
put these curves together and derive a
single curve that is the exposure
function of the camera so now once we
know this exposure function of the
camera we know the output gray level in
the digital file and the exposure time
we are able to recover the scene
radiance of the image that we capture
and with this I would like to do it them
on the application that is able to
create this high dynamic range image so
let's close this and open the Creator
the Creator is an application that first
will allow us to pick up several images
and I choose a number of already exposed
shots and we get a thumbnail view on the
left side and in here we can select any
of these images and see what is their
content you know immediately that no
matter how we take the images you can
see either details in the in the shadow
or either details in the highlight of
this and the high dynamic range file
will be able to capture all information
of the radiance of the scene and will
will encapsulate this in a high dynamic
range format the first thing is to
calculate the high dynamic range to
calculate the transfer function of the
camera and we did this in a single step
you have seen several curves put
together and you recover the transfer
function of the camera then the next
step is to use this this transfer
function and to compute the high dynamic
range image now you can see that even in
the paper it said that you need to do
this processing of the transfer function
of the comet over many in many images
until you get an average behavior of the
camera and use that transfer function to
create a high dynamic range we actually
add more robustness to the algorithm
that is computing the transfer function
of the camera such that we are able to
recover the transfer function only from
the same set of images that we used to
create the high dynamic range so we
recover the function from this set of
images and we apply this function to
these images and we create a high
dynamic range this bring us to a now
going that will be able to do this kind
of things by just specifying the set of
images so we choose a set of images then
the algorithm is computing the transfer
function of the camera and is computing
the high dynamic range
single shot and this gives independence
of this application from any camera
settings on any or any setup that you
may have been required to do for the
method that is published in the
literature so this let's say you are you
are switching to another set of images
for example from a different color and
you don't have to specify the camera and
you immediately get the high dynamic
range files direct from this from
specifying only the set of images the
interesting thing about this is we want
the user the Apple user to be able to
have less intervention in this algorithm
SES work and finally end up with an
application that will be able to provide
directly high dynamic range images
without taking care of anything this is
an advantage for the user finally I
would like to mention that as you have
seen here we select the images from a
folder but we have worked with the image
capture so I invite you on the Friday
afternoon from 5pm to see the an
integration of this algorithm each
capture modules and you will see an
interactive demonstration of how these
images are captured live with the camera
and then high dynamic range file is is
created and with this I thank you very
much and and I will turn back to look
[Applause]
thank you
averill so at the end of the
presentation on high dynamic range
images I like to touch on this subject
which is very close to our hearts of
color sync engineers we are really
interested in color managing high
dynamic range images and you must know
that this is the area which is under
very intensive investigation both in
academic society and in the industry
that Apple we also are developing our
own method of color managing high
dynamic range images and we are trying
to take a new approach which is based on
human adaptation to image viewing
environment we think that image contains
enough white point and adopting
luminance information that could be used
by color appearance model to predict
human perception of color in different
viewing environment which basically
means we can color manage our high
dynamic range images just to clarify
what kind of color appearance model
modeling we are dealing with let me say
that we are looking at this kind of
modeling which consists of two major is
based on two major concepts of on
chromatic adaptation which allows us to
predict the influence of adopted white
point on the color perception and on the
second concept is the degree of
adaptation which allows us to predict
simultaneous color contrast related to
the luminance of adopted white point so
in summary what we are trying to do is
to transform the Kurama tree of this
source using high dynamic range data to
our destination and then after doing
after color managing into and bringing
two new environment then at lying tone
mapping which will compress color to the
range of destination device and this is
what concludes our talk on
high dynamic range images and I'll turn
back microphone to David thank you so
thank you Luke and Gabriel free
discussion I just wanted to bring it
back just to do a quick summary slide
and then we'll have a few minutes of Q&A
at least just wanted to summarize once
again what's new and tiger we got a lot
of great stuff here first of all in
color sync we have floating point
support image I oh we have a brand new
modern API for reading and writing that
has optimized performance and support
for metadata and then we're doing a lot
with high dynamic range supporting
openexr file formats access to
compressed or unprocessed data and also
this is an area of all sorts of ongoing
and future research will have watched a
show for you so again we have a few
other places you might want to go
there's a graphics and media lab session
on Thursday where you can talk to us if
you have more questions and we can get
to today and then also there's going to
be some great demonstration at on the
last show last session on Friday talking
about image capture in high dynamic
range