WWDC2004 Session 204

Transcript

Kind: captions
Language: en
ladies and gentlemen please welcome vice
president Interactive Media Group Tim
chef good afternoon thanks for coming I
think the first thing I just wanted to
say is thank you all for coming we've
got a really busy week of activities
this week and it's real important that
you come and track us down and talk to
us while you're here we spend a lot of
people from Apple to come up here and
meet you and get your feedback I know
we're going to be dumping a lot of
information your way today and over the
next several days it'll be a whole lot
more useful for everyone if we can
actually have a dialogue about this so I
just want to encourage you to find the
Apple people and start talking to them
about what's going on so today's session
we're talking about graphics and media
overview here this morning you saw Steve
talk about some of the really exciting
features we're going to be providing for
for customers and for us developers in
the afternoon we heard Bert Ron talk
about some of the platform level things
we're doing to try and really drive the
innovation forward and in this session
what I want to do is focus on the audio
the graphics and the video pieces and
our perspective on this is that these
technologies are traditionally seen as
kind of the multimedia technologies for
multimedia types but we believe that the
situation really is changing and what's
happening is that if you want to build a
world-class application in this day and
age we think these technologies are
absolutely essential to being able to
get there so our presentation today is
certainly going to have some of those
qualities that you would expect in the
the media session but there's also going
to be a real heavy emphasis on
technologies that can be used in a wide
range of environments not just media
application and as you've been seeing
throughout the day there's a big
emphasis on how the CPU and the GPU can
work together to do some amazing things
and you'll see more of that here so
we're going to focus on three big issues
that affect all of you as
developers one of them is how can you
achieve the kind of gorgeous look and
feel that customers are coming to expect
using Mac os10 the second the second
element here is how can we achieve
radical performance we've all been
hearing the stories about how g5 is a
supercomputer in a box and we've been
hearing about the GPU and how it's
racing ahead of the GPU in terms of a
computational power how do you unlock
that it's not as easy as you might want
it to be and then last of all you're
making huge investments in these
software applications they have
lifetimes which are measured in several
years if not decades how do you protect
that software investment from the
changes that are undergoing taking place
in the industry at the hardware level
and lower levels in the operating system
I hope we're going to be able to help
answer some of those questions this
afternoon so the first thing I want to
do is step back a little bit and talk
about some larger industry issues this
is a original paper that Gordon Moore
wrote all the way back in nineteen
sixty-five this paper became the
outlines the ideas that ultimately
became known as Moore's Law and it was
basically as you know the the statement
about how the density the power the
integration of computer chips would
approximately double in complexity every
two years this chart over here on the on
the right is the original graph that he
showed and what's very interesting to
see is that in 1965 he made this
prediction when the typical IC consisted
of about 50 transistors keep that in
mind as we go forward so spin forward
about 20 years the original macintosh
had an eight megahertz 68000 processor
from motorola about 68,000 transistors
spin forward another ten years we're all
the way up to three million transistors
in the very first PowerPC chip that we
shipped in the very first power
macintosh bend forward to this year well
we've just introduced a 2.5 gigahertz g5
systems were all the way up to 58
million transistors so here we are
looking back at this paper that was
written in 1965 roughly speaking 40
years double every two years so that's a
20 doublings to the 20 it's about a
million so he sitting in a environment
where your typical IC had 50 transistors
we're now up to 58 million transistors
that's a pretty amazing prediction that
this guy did so as we've mentioned
earlier today there's a lot more
happening inside that box than just
what's going on in the microprocessor
the chips that we ship in our current
systems the Radeon systems are already
up at about a hundred and fifteen
million transistors this is about double
the complexity of the of this
microprocessor the the system that Steve
introduced this morning the Nvidia the
new nvidia geforce 6800 ultra processors
have 222 million transistors four times
the complexity of the g5 that's an
incredible opportunity for all of us be
able to enhance the functionality of our
applications but unfortunately it
doesn't happen automatically and the
question we're going to try to address
today is how to unlock this stuff there
are many many obstacles so in 1984 this
was the statement that Apple made about
the importance of rich graphics
interface WYSIWYG and interactive you I
this was really a dramatic statement
this brought into the mainstream
concepts which had not been seen in a
personal computer before spend forward
20 years to this year Apple's introduced
this product called motion I think it's
a very good example of the way the
expectations have evolved over the 20
years still rich graphics still
essentially a WYSIWYG kind of an
experience very much an interactive you
I but obviously the kind of sauce are
required to build this interface is
incredibly more complicated than the
software required to building the app of
1984 let's look at the operating systems
that you have to learn and understand
Apple has you know roughly speaking
every single year we're introducing
another thousand man years worth of
software every single year and you can
see this in a sort of very simplistic
view of the size of the disk that we
have to ship to you in order to
deliver the operating system we've moved
to a DVD this year because yes it's
growing again how are you going to
manage all of those toolboxes there's so
much stuff to learn as we introduce more
and more systems with more and more
diversity of hardware with fancier and
fancier architectures the question
becomes how the heck you get all this
performance out of the box it used to be
that you could sit down and you learn
the instruction set these rights and
programs and you'd learn what was fast
and then you'd write that in your code
and you have the fastest appt on the
planet it doesn't work like that anymore
it's really really complex and the
performance decisions that you would
make for a g3 will be different very
different than what you're going to do
on a g5 how can you do that
automatically this is not a puzzle you
probably want to spend all of your time
trying to solve when we sat down to
think about what would be most profound
in our contribution to tiger in the in
the graphics and media team that I run
we wanted to look at these issues and
say how can we unlock this power for you
protect your software investment and
give you tools to build beautiful UI and
this was our goal so we want to build
the technologies that are going to allow
you to build the next killer app because
we believe as cool as these demos that
you've seen today are we believe that
you folks are the ones who are going to
have the really really exciting ideas I
mean it's neat to see core image
processing these images and we're going
to show you some more of it today but we
are absolutely positive that your
inventions are going to blow us away
amaze and delight your customers and
just just baffle the world how do they
do that so here's the architecture of
the graphics and media layer as a
panther basically very familiar a block
diagram here we've got quartz 2d we've
got quick time we've got OpenGL all
layered on top of the graphics hardware
this has been a very very powerful
architecture for Apple there's only one
problem here you could look at this
another way as three stove-piped
technology staff and as many of you will
know if you start to try to combine
these technologies drawing the lines
from left to right right to left
horizontally you start to run in some
real challenges so in Tiger one of the
most important changes we are making is
we're kind of turning this diagram on
its side in order to give a much better
much more powerfully layered system for
you to work with now the api's aren't
like all different in Tiger it's your
applications are still going to be
compatible but we're working down in the
lower layers to reorient this
architecture in order to maximize the
kind of performance and the kind of data
interoperability that we'll be able to
provide through the API so that your
apps will be able to combine all these
technologies and still have a high
performance result broken this up a
little bit more detail you see the chord
stack you see the core audio stack
they've both got their their hardware
abstraction layer conceptually and so
what I want to do for the rest of the
talk is we're going to dive in and look
at what's going on in each of the
individual technology areas and tell you
some of the highlights of what's new and
hopefully give you some plants and seeds
for how you might be able to take
advantage of these things our first
areas core audio as we as we've
expressed earlier today this is a very
profound technology for processing of
audio this is the most advanced audio
sub system built into any operating
system anywhere else when you look at
other operating systems if you want to
achieve the kind of performance that
we're able to achieve inside of Mac OS
10 you always end up having to build
special things into the OS you have to
add special extensions in order to keep
the kind of performance that you get
with mac OS 10 out of the box what does
that look like well to the ultra-low
latency that allows you to do demos like
the guitar demo you saw earlier this
morning playing live in to the system
it's processing the audio in real time
sending it back out out the audio
outputs and it feels like he's playing
it live as if it were a piece of
hardware in a in a music studio rack
we've got high-def this was the first
media processing stack to build in a
full floating point processing pipeline
and it's worked tremendously well for us
not only does it allow us to support
current state of the art processing
24-bit type samples we can go way beyond
as the industry evolved we support all a
very very wide range of sample rates and
there's no constraints on the channel we
have beautiful robust plug-and-play
compatibility across a wide range of
connectivity protocol we've got built-in
natives built-in MIDI support very low
latency just like the audio support and
then we have this very important
architecture for extending the system
one of the most important parts of an
audio subsystem is the set of customized
audio plugins that you can bring to bear
to provide your audio a unique kind of a
sound or unique kind of processing now
we've had a tremendous amount of success
with our audio units and this is a small
sampling of the developers who have
gotten on board with audio units now
force the universe of developers who are
using Core Audio is much much larger but
here's a really important collection of
developers who are focused on audio unit
development this success has brought
some very interesting new challenges and
some new opportunities as the community
of developers who are centered around
core audio continues to grow one of the
things that we've been very concerned
too to be able to provide is a very high
level of compatibility between all these
different kinds of plugins and so
earlier this year we introduced a
program to a program and a piece of
software that would help Audio Units
developers ensure that their audio unit
plugins were going to be robust
compatible as they went from one apps to
the next and to the next we're
introducing today a new product called
au lab which is a part of the developer
the developer toolset that's going to
take this one step further and I'd like
to give you a chance to take a look at
that in just a second and what this is
going to allow you to do is it's going
to help help the audience developer
develop an even more robust audio unit
that can be plugged into the ever
increasingly complex and sophisticated
range of audio applications so why don't
we come over and take a quick look at
this
long walk so we have this little
application called au lab and basically
the model of au lab is it's a mixer and
the first thing we're going to do is
we're going to come in here and we're
going to add a generator and we've built
into this tool a generator that's
capable of bringing an audio from a file
now of course it can also process audio
in real time coming in through the
different kinds of inputs that you can
support so the first thing we're going
to do is we're going to open up the
little configuration panel for this
bring in an audio file and we can play
it just like a audio playback system
great now what you can do is you can
bring in your audio unit plugins and
load it into this little mixer and it's
now it's now been inserted into the
processing chain and what we've got here
is we've got a new audio unit that we're
providing and tiger called time pitch
and what it has the ability to it's a
very sophisticated DSP algorithm that
allows you to alter the the timing of a
piece of audio without altering the
pitch or you can alter the pitch without
altering the timing and I'll do a very
give you a very quick example of what
this might be useful for so this is a
piece of music by a couple of guitar
players John McLaughlin aldi meal two of
the fastest guitar players I've ever
seen in the universe and what you
encounter with this kind of guitar
playing is that they start playing
really really fast my son is a guitar
player and he's always asking how can I
learn how to learn this music it's going
too fast this little plugin basically
allows you to take these files
and you can dramatically slowed them
down without altering the pitch you can
go faster and again it sounds just like
the just like the original just like the
original audio so there's a very simple
demo of how a you lab can be used to
validate and test out your audio units
and we think it's going to be a very
important part of our overall strategy
for ensuring broad adoption of audio
units and being able to help facilitate
a very powerful community in the audio
world relying on core audio so here's
another situation that's coming up more
and more as these core audio
applications are being enhanced typical
kind of a studio setup you've got
multiple audio devices today in core
audio we can handle these devices just
fine but when it comes to managing those
devices all the burden of managing which
channels are going where and the inter
device synchronization is the burden is
on the application we're going to be
introducing a new technology that allows
you to aggregate these devices together
into one logical unit and we think it's
going to make the process of developing
applications for sophisticated studio
setups much much easier last of all I
want to talk about open al open al is an
industry standard API for managing
spatialized audio it's used all over the
place in games so technology developed
by creative labs and we have been
working with them and other developers
to build a highly optimized version of a
open al and it's capable of supporting
not only super high quality facialized
audio processing but it's also got a
variety of modes to be able to do lower
complexity mode so you can get lots and
lots of channels and we're going to
build this into Tiger and we think it's
going to be a great addition to the
OpenGL technology for a complete
solution for game development so this is
the story about what's happening with
core audio what I want to do next is I
want to ask Peter graphic nino our
courts extremists to come over and
you about everything that's going on in
the courts world and it's a very large
world
I guess I can go to the podium other
rear okay so I'm going to walk you
through the technologies in the graphic
stack looking through starting angle
basically walk you up the stack from the
hardware talk a little bit more about
that OpenGL and then talking about
courts and then Tim's going to come back
and talk about good time so on the
hardware side you've seen this graph
many times this is pixels process per
second and if you look at also memory
fill rate the current generation of
graphics chips are about 35 gigabytes
per second of memory bandwidth which is
pretty incredible and as you've heard a
couple of times already this is kind of
an inflection point in computer graphics
we now have programmability at the pixel
level it's floating point it's accurate
enough for for high-end work and high
dynamic range it enables a lot of
applications beyond just the traditional
gaming stuff so we're seeing a
convergence of graphics and media
processing together in the GPU and
there's some new programming paradigms
you heard me in Bertram's talk talk
about the stream computing model that
we're using for core image and tiger
head will advance the state-of-the-art
here in bring a lot of ways to bring the
power of the GPU into your applications
so there's a bunch of things you can do
in Tiger as far as a GPU programming
there's low level access via OpenGL
there's accelerated 2d graphics we have
with core graphics which we'll talk
about in a second we have accelerated
image processing with core image we we
have core video which we're also talking
about basically we've got a bunch of
ways to leverage the power of the
graphics processor and the trick for
your apps is to try to use the highest
level of abstraction you can which
allows us to do more of the heavy
lifting there's nothing wrong with going
down to be OpenGL layer if you need to
do that or want to do that but we've got
these higher level services as well
which you could take advantage of and
let the platform do the rest so the the
tiger quartz layer is optimized to take
advantage of programmable GPUs now it's
not necessarily required to run tiger
beyond a program will part but it's
optimized in that way and by
programmable GPU what do I mean I mean
on the ATI product line any part that
the ati radeon 9600 or higher or on the
nvidia side the geforce fx or higher or
the new card that we just announced
today by technology i mean ARB fragment
program if you're an opengl programmer
or on the windows side you sometimes
hear called directx 9 capable hardware
that's the basic class of hardware now
interestingly enough if you look at the
GPU versus the CPU the the GPU is not
necessarily 0 is faster so something to
keep in mind that you really need to
treat it as a kind of a co processing
environment the high-end CPU if you put
tool Duke to dual gigahertz to 2.5
gigahertz g5 in a computer with a
low-end but programmable graphics part
the CPUs will be able to easily easily
beat the low-end GPUs so you always have
to be aware of the trade up and really
view the GPU as kind of a coprocessor
and also understand if you're flexible
about using GPU or cpu that your apple
scale to a much greater level with the
GPU scaling factors that we're seeing
that are exceeding moore's law that's
all I'm going to say about hardware just
motivate you guys to learn more about
what's going on there talk a little bit
about OpenGL OpenGL is the the
foundation and sort of the hardware
abstraction layer for our graphics
hardware as Tim mentioned and there's a
bunch of things going on in cut and
tiger there's the OpenGL shading
language which will be supporting
there's floating point pixels support
some of this made its way into Panther
there is major enhancement to the OpenGL
profiler to tool which is a really
popular developer tool for analyzing
your OpenGL performance there's new
resource management improvements we're
probably on our third or fourth
generation of kind of treating the GPU
as a full-fledged resource to be managed
within the colonel there's a lot of new
stuff to support many of the things
15 today and I'll see throughout the
conference and I did want to mention one
session that was not in the show guide
here which is the introduction to OpenGL
shading language which is going to be on
Friday so be sure to check that out
that's OpenGL let's move up to the
courts player this year we're sort of
dividing the courts layer into three
things there's core graphics of course
which is our 2d graphics and windowing
system and then there's core image and
core video which are new this year core
image is our image processing engine and
core video is our video processing
engine on the core graphics I'd we're
not staying still here either so let me
talk about that the big news in core
graphics land is that course 2d goes
extreme so if we're talked a little bit
last year about putting having quartz 2d
on OpenGL and this year it's going to be
the default in tiger so we have quality
with the 2d rendering through OpenGL
that's virtually I did identical the
software quality and any of you guys who
who know about how GPUs make different
trade-offs about 2d graphics it's
actually pretty tricky to do this in all
cases to get really high quality text
with OpenGL is kind of a pain but we've
taken care of all of that we catch even
LCD quality sub-pixel position glyphs
and can render them and flip them onto
the screen with all the proper blending
you need to do to do that so this
acceleration of course duty however does
require programmable hardware to be able
do the LCD bleeding and all the
programmable blend mode the low level
benchmarks when you put quartz duty on
top of GL increased by two to a hundred
times so there's some real impressive
performance gains to be had but the key
to getting those are to reuse your
resources so if you have CG image reps
or CG pattern ref things like that and
you're going to draw them more than once
just be sure to hold on to that and it
will get cached up in video memory the
other thing since we can't accelerate
quick-draw if you use quick-draw within
a window and you're using courts to the
extreme we fall everything back to
software
so the core primitive vents benchmarks
with quartz 2d you can see the increases
here pretty striking obviously GPUs are
great at filling memory so in 800 x 800
rectangles 236 times faster it could
probably even go faster for you to
figure out how to feed it line drawing
for example 8 x text strings our
software text path is actually pretty
tuned at at one and a half two million
blitz per second but we're getting
almost 5 million glyphs per second with
the with the hardware which is really
good so we didn't turn it on by default
in the wwc build for those of you know
about the courts debug tool you can go
in and turn it on it works pretty well
it works a little bit better on a TI
hardware than nvidia hardware right now
but you can turn it on we know that
there are a few bugs in there but it's
going to try out your app and see if
things get faster if not go catch your
pattern reps and image rest and you'll
probably see quite a performance
performance difference the other thing
to look at in quartz debug is you'll
notice a show user interface or
resolution menu item and what we're
talking about this year at WWDC is
getting ready for for a resolution
independence in the user interface and
the school kids and i think this is
thanks
this is going to have a profound impact
on our ability to build products
together and just on the on the way that
users can interact with the machine in
your application and there's a lot of
work to do on this and it's both our job
and your job in the applications there's
no silver bullet that we can just
magically scale everything we've got
scalable graphics api's with quartz 2d
and OpenGL but there's still pixel crack
assumptions and things like that in
people's code so actually what I'm going
to do now first thing I'm going to talk
a little bit more about that and then
give you a demo so the the motivation
for resolution independence there's a
number of things there's the real estate
versus detail trade off on those big
monitors you might not want to use it
all for detail you might want to
actually you might not use it want to
use it all for real estate rather you
might want to use it for detail and just
have finer text or higher res ered more
high resolution displays that sort of
thing but we really need to break the 72
DPI assumption that's kind of wired into
a lot of code out there and it's really
the once you have a scaling matrix in
your graphics API we found the next
problem is you're going to have pixel
boundary assumptions in code where you
know what scroller lines up on an
integer pixel boundary well that may
actually be in between two pixels if you
if you put a scaling matrix in so
there's three modes your application can
run in and just cover them briefly here
and then you can learn more in like in
the cocoa and carbon sessions because
there's a little bit different work to
do in both cases first mode is called
framework scaling for carbon and cocoa
this is where the framework takes care
of all of those all of the work for you
and unless you're doing your own drawing
if you have a custom sub view or a
custom a chive you on the carbon side
everything should just work there's
nothing you have to do the math
coordinates will get scaled the drawing
will get scaled if you're a carbon app
and you don't want the system to do it
for you you can also have what we have
application scaling here the carbon app
app cos for the scale factor enforces
that itself it can ask for you I to be
drawn bigger but it might have its own
graphics engine that it knows how to
scale already and then finally we have
magnification which is just sort of a
accessibility style zoom where the
windowing system
courts compositor we'll just scale your
app for you and that's for unmodified
carbon apps although there are some
cases where if you do have coordinate
assumptions even that may not work
exactly the way you want but find out
more in the sessions like I said and let
me do a real quick demo of that on the
demo machine here just to show you
courts debug if you don't know courts
debug you can find it in the developer
tool that's under the performance tool
you can see I've got courts to the
extreme enabled here I'm going to go up
to that Tools menu and bring up the user
interface resolution and let's crank up
things a little bit and give us kind of
a virtual scaling factor of about that
let's do something something big like
1.75 and now any app I launch is going
to get a 1.75 scale factor so let's
launch Safari and see I get a huge menu
bar and I'm not connected to the
Internet well that's too bad but you can
see how the the app is drawn much larger
the menus or high resolution rendered
you can see the rest of the if I switch
to another app for example you'll make
it clear here's courts debug menus are
still small whereas so far as menus are
big and so you can see that there's
going to be some drawing bugs like over
here by the Google search menu which
were you know we're working through with
the apps and the framework but you
should use this tool on the developer CD
and test out your app with with the
resolution independence so back to
slides please
the next thing we're doing in in court
studi is floating point pixel support so
yes thanks what comes after millions its
Jillian's banana but it's full
floating-point pixel processing pipeline
floating-point both on the source and
destination so CG image reps can be
floating point as well as PG bitmap
contexts as destination for drawing we
use unclamped colors process all
throughout the pipeline and we're going
to have a floating-point CMM based on
the color sink in the tiger GM that's
pretty exciting and to go with that we
have a new framework and core graphics
called image i/o which is a thread-safe
image handling library for file i/o and
it includes new HDR high dynamic range
floating-point format such as openexr
from ILM and various flavors of
floating-point tips that are out there
next up is PDF get PDF kit is a
high-level framework for dealing with
PDF as an objective-c framework up at
the app kit layer it's kind of you can
think of it as a preview on a pallet and
much in the way that WebKit is sort of
Safari on a pallet PDF kit is the PDF
sort of half of preview running on a
pallet complete with linked reversal and
printing and and you know fit the page
and all that sort of stuff so right
there's going to be a session on that as
well and I'll give you a point of that
in the sec but I thought first I would
talk a little bit about quick-draw and
all the things we're doing quick draw in
tiger which is nothing except not break
it so this brings me to the point where
I have to basically tell you that quick
draw is not evolving anymore we're not
going to give it floating point
coordinates or alpha support it's really
been limited in hardware acceleration
floating point pixel support resolution
independent core image integration all
of these things would be very difficult
to add to quick draw and frankly just
not really worth our time or your time
changing all your quick draw code so you
know as is a little tribute to quick
draw though quick-draw has served as
well I mean it's been a brilliant
invention
you know far more profound perhaps than
anything we're showing you this year it
really rusher din you know a whole
generation of graphical user interface
and had a number of really cool
innovations in its day and it's been the
cornerstone of Mac OS development for 20
years but it's time to move on so we're
going to be deprecating quick draw we're
not going to break your apps immediately
of course 2d entire will fill in the
last of the big holes that we've
definitely gotten feedback on the
headers are still available but they'll
be marked deprecated if you compile
against the tiger target binary
compatibility of course the league's it
will continue to exist at least for now
and you should definitely budget time in
your next revision of your app to move
to core graphics and we really want to
make it work for you so we're definitely
open to feedback and we have a whole
session about transitioning the courts
2d that specifically targeted for
quick-draw developers and got a lot of
new stuff that should make it easier and
we want to hear about more things you
might need so to help motivate that a
little bit let me do a quick demo of
line-drawing performance so this is one
thing we got feedback on as being much
faster in quick-draw than it was in
court 2d course d D to D would lovingly
anti-alias all of your intersecting
polygons and sum N squared algorithm and
would just be really slow so people
would use quick draw for stuff like this
well we've improved courts to D such
that it's now 1.5 million and this is
actually the software renderer and once
we go on top of GL we're actually up at
11 million lines per second great and
this is still only immediate mode in
open jail if you know anything about
OpenGL there might be another 5 10 X on
top of this as well but clearly GL is
the way to draw a line and so just to
drive at home we're looking at you know
41 we didn't draw the initial courts 2d
software which is somewhat less than one
point out at some of you might know but
getting onto course 2d is clearly what
you want to do here so back to slides
so some bonus sessions in the 2d
graphics package 2d graphics pass the
high dynamic range is going to be talked
about together with the image io
framework that system is going to be on
Wednesday and then we have the session
on PDF kit tomorrow so please attend
those talk a little bit about core image
you've heard about it in the keynotes
and other talks again it's a framework
for GPU focused image processing with a
rich set of built-in filters and our
plug-in architecture the great thing is
that you don't have to know about OpenGL
to use core image which is a nice a nice
feature for those of you coming from the
the 2d world core image is actually as
an objective C API it's based on a
stream based processing model the number
of abstractions is very few there's
images and there's kernels which are the
processing units and then there's
samplers which basically kernels use to
access data from within images kernels
are described in a runtime compiled
language as a high-level see like
language the parallel ISM is implicit as
i said before on the other talk there's
no explicit loops or threads you just
use the image unit or program the image
unit to do what you want and the
evaluation model since its implicit is
mappable two GPUs symmetric
multi-threaded CPUs or other parallel
architectures you might imagine we have
a software fallback that's actually
quite good as well uses the velocity
engine and a just-in-time compiler
that's custom for core image and it
generates Auto vectorized code optimized
either 4G 4G five or even dual g5 with
multi-threading if if that's available
and it's the evaluation engine does a
number of like sort of compiler
techniques at the at the graph level as
you as you construct image units
together to minimize the number of
passes and handle the temporary buffers
p buffers and caching and things like
that so it does a lot of heavy lifting
for you and let me give you a demo of
that
so when we were preparing for the
keynote we did a lot of sort of
experimentation with with core image and
some image some imagery that we didn't
use in the keynote I thought it'd be fun
to show you guys here so here's here's a
couple of interesting interesting
filters here's a circular halftone
screen on this image I can change the
width it's all getting computed on GPU
this is a radeon 9800 xt guard some
other ones that are kind of fun the
surfer wave reset back and let's do a
zoom blur on him oops not white pointed
Jeff
so that's kind of cool and change the
amount what else that I have let's do a
layer effect so I'm going to bring a
water droplet background image let's
turn off zoom blur let me add another
filter which is a piece of water artwork
and let me add a rounded kind of 3d
effect on that and you can kind of see
see what that looks like that's pretty
neat the other thing I wanted to show
was kind of a build of one of the
stacking effects builded the the
electric zebra this morning and I'm
going to show you a different one which
is kind of also in the in the 60s theme
so let's we're going to try to go for
some kind of silk screen look so we're
going to color poster eyes the image a
little bit then I'm going to over
saturate it using a over saturate filter
I can use a false color false color just
does it converts the image to grayscale
and then maps it through a ramp and I've
got a couple of colors here that I'm
going to use so yeah and that's what I'm
looking pretty psychedelic then the next
thing I add in is every 60s poster needs
a lenticular halo there we go which this
is doing a the generator that generates
kind of a Sun striation sort of a thing
let's do that okay let's bring in some
line art well that's okay the 60s album
which is the type and will insert a bump
map in there to do our bump distortion
and then lastly i think i have a crop in
here app so you can crop out the edges
so but the great thing is the whole
stack is live i can totally
non-destructive it's just sort of
remembering that
recipe to create the image and you can
kind of turn off the layers and see see
how each one is doing or go back and
change the posterization level on the
fly I can even go back and change the
image on the fly I could do it too you
know the water image if I you know my
band changed or I can even run video
through the background so you know you
get your your album cover in your your
video at the same time and it's all
still alive I can find the bump let me
add the bump back in i love forever
saints of posterization from the layers
off on real time that's pretty cool
so you definitely want to go to the core
image session hopefully we've teased you
enough by showing it in every single
keynote session so and it's tomorrow
afternoon at five o'clock so you'll
definitely want to see that core video
core video is one way to think about
core video is it takes the output of
QuickTime and maps it into the GPU and
so it's basically new for QuickTime is
separating the decoding logics in the
presentation logic so it solves the
video on a texture problem which we know
a lot of our internal developers and I
know a lot of you guys as well have had
we've had like four or five different
examples on the website of how to do
video on a texture none of them were
actually optimal and we're just solving
that in Tiger in a great way where you
get total asynchronous a synchronous
behavior and data flow between the GPU
and the CPU and that's really what
allows us to do some of these demos like
60 p high definition h.264 it allows
also once you get the video frames up in
the GPU you can use core image on them
for video effects so bonus session on
that of course the core image effect the
core image taught as well as new
directions for QuickTime performance
where you're going to feel out of the
stuff in action so that's core video so
we have the whole stack of technologies
we call courts at the core level the
operating system and we think it's a
great substrate for you guys to build
apps on and I'm going to show you a
quartz composer one more time I had a
quick demo of it before I spend a little
bit more time on it here quartz composer
is an application that harnesses the
power of the courts layer using core
graphics opengl core video and all of
the technologies compositions can be
saved in files kind of like you can
think of immense metafiles and they can
have opengl info core graphics etc and
there's a simple playback API it's a
framework the engines built in VOS so
you can load a composition and play it
and it creates a procedural animation
you can actually expose the variables
outside of your animation to the key
value coding system in cocoa and set
parameters like images or text strings
into the animation so let me give you a
demo of that real quick here
so let me navigate back again to the
example of directory so it's in
developer example quartz composer and
I'm going to show you a couple of things
this time that I didn't show last time
like masks slideshows kind of neat
basically take some of the images from
the screen saver and puts them through a
mask it generates a mask with with
OpenGL and then does a blit using the
the graphics hardware I think this one
pretty much is straight it straight up
OpenGL it doesn't use any core image so
if used as blending modes and the
hardware so that one's that's pretty
interesting when we switch back to
finder and show you distortion FX
distortion effects is an interesting one
so here's something you have you hadn't
seen before with the other composer demo
says drag and drop an input file into
the input parameters and so what this
composition does is it's all this wiring
in the details of the plumbing but it
exposes two variables and image in
adoration and so it's waiting for input
and so let's drag Copenhagen just an
image of Copenhagen Harbor there and
let's look at that that's full screen
it's using some core image effects the
OP tile and the bump distortion to do
that so you can see how if you had an
application and you wanted to do that to
an image you could just actually create
a quartz composer composition load it up
and then just keyvaluepair iya Buhl's
and you've got it you've got it going so
to drive that home a little bit more let
me show you another one we have which is
a we have engraved your ipod's well we
have engraved your g5
let's look at the input also to you down
here I can say Peter's g5 and you can
see it announced as Peters g5 you can't
really see it too well so let's let's
actually build an app based out of this
out of this composition so I actually
have a nib file there's nib file in the
example directory there you can see we
have a palette the quartz composer
palette which is one of these I think
there it is which has a controller which
is the green thing and a view which is
the quartz composer view and what's
happening is I have a patch controller
that it brought into my project down
here which is controlling the patch the
g5 engraving patch and I have a view and
if we look at the parameters on the view
by bringing up the inspector I'll look
at the bindings you can see that the the
the view is bound to the patch
controller which is this guy here who's
controlling the patch now if I look at
this text field its value the mouse's
little twitchy sorry about that the the
value of the of the text field is bound
to the patch into a variable in the
patch called text value which is that
thing was that was exported in the in
the composer and so now if i run this I
have a simple application and if I can
just type in the text field and say
Peters g5 see I can label my own g5 so
it's pretty easy to go into quartz
composer and create a procedural
definition of either an animation or a
still thing like this and then just wire
it up and drive data into it with cocoa
binding so we think that's kind of a
pretty powerful integration technology
so back to slides so we're going to have
a session on the court composer called
discovering the quartz composer it's not
a 9am on friday but if you want to stop
by the hands-on lab and get going with
this stuff tomorrow morning a bunch of
us are going to be in there to help you
with that if you want to just look at
the examples and get up and running on
the tool so anyway so that's our
platform of all the new stuff we're
doing in courts that I just highlighted
a lot of the new stuff there's tons of
sessions i think there's about 15
sessions just on the graphics side of
stuff that you want to talk in a bunch
more on the quicktime and audio stuff so
have a great conference and we will see
you with the sessions in back to Tim
thanks a lot here ok so we've covered
audio we've covered graphics now it's
time for the video stuff and of course
that means quick time this year I'm not
going to spend any time at all talking
about marketing stuff we've got a
tremendous amount of good news to share
with you but tomorrow morning Frank
Casanova and his team are going to put
on a big extravaganza here talking about
what's going on with the marketing side
of QuickTime of the business side how
we're doing in them out in the
marketplace fantastic story going to
tell you an awful lot about our 3G
strategy which is a tremendous success
in marketplace we have by far the best
system for supporting mobile media in
the world we supports all the all the
global standards at this point and we've
got just a fantastic and dance Torian
and I think you'll like it we're going
to focus today on the technical side of
QuickTime this is the architecture of
quicktime circa nineteen ninety-five you
know obviously it's very simplified but
the basic point here is that quicktime
is a very modular very componentized
architecture but the components tend to
be oriented towards supporting quicktime
so they're kind of inwardly focused we
had video processing components they
have audio processing components and all
of that was layered on top of quick draw
on top of the sound manager if we spin
forward to look at what we're doing in
in Tiger this year you're going to see
some dramatic changes taking place down
in the core of quicktime and we're going
to be layering quicktime on top of
quartz we're going to be putting it on
top of core audio and there's going to
be some tremendous features here we're
going to take advantage of all these
technologies and I think you're going to
really learn really enjoy what starts to
happen with QuickTime today I'm going to
talk about for details that were for
very very important happenings in the
world of QuickTime that I think we're
going to have profound impact on your
applications and on the ways that quick
time starts to proliferate around the
world so the first thing I want to talk
about is h.264 you saw some demos of
this earlier let's look at what this is
really about so you all know what MPEG
is these are the guys who make dvds
digital TV it's actually this this
compression technology is actually a
collaborative effort between MPEG and a
lesser-known organization called itu the
International Telecommunications Union
these are the guys who back in the 1860s
the 1860s first set up interoperability
and interoperability organization to
ensure that people would be able to
Telegraph across national boundaries
they then applied that kind of mindset
to radio and more recently they do it
with telecommunications and they are
really one of the premier organizations
for advanced technology in the area of
compression as it applies to
communication the goal is very simple
build the best codec that anybody has
ever seen it supports a lot of different
modes we'll talk about that some more in
a second what you may not know is that
Apple's been involved in this standard
for a long time and we've actually got a
whole bunch of patented technology built
into the core of the standard and the
format is built on the QuickTime file
format of course so that's a great thing
here's a little bit of a time line chart
showing you kind of how the the video
compression efficiency story has evolved
the upper line shows you the story for
mpeg-2 and the lower lines are showing
you what's been happening with some of
the more modern codecs and the orange
piece obviously is the
26 for a couple things to notice we
learned an awful lot about video
compression over the last 10 years and a
lot of that was able to be played out in
the mpeg-2 standard so even though the
standard was established in 1994 for the
next ten years we saw dramatic
improvements in the efficiencies of
mpeg-2 so you sometimes hear a story
which says well your codec isn't good
enough you don't have the right standard
the thing you got to separate out here
is the politics from the reality of the
technologies and what we know is that
with each successive generation these
technologies they're an incredible a
number of tools that are built into
these little algorithms that we all have
to learn how to use and this shows you
in a very graphic way what happened with
MPEG over the years I mean this is
dramatic they went from almost 6
megabits down to just under 2 megabits
to achieve the same level of quality and
what we're going to see with mpegs with
mpeg-4 part 10 h.264 AVC goes by a
billion different names we're going to
see again a continuing evolution as we
learn how to use this codec let me give
an example of the kinds of stuff that
we're working through so this is mpeg-4
part 2 this is the version of mpeg-4
that we ship in QuickTime six today that
we've distributed literally hundreds and
hundreds of millions of copies of there
this is a sort of a summary of the
various algorithms that to get are
combined together to create the overall
compression effect h.264 offers this
vocabulary of tools so you might imagine
that a software developer trying to
build these kinds of technologies is
going to take quite a while to learn how
to use them optimally we've been working
on this for a long time we think we have
a fantastic implementation already but
it's going to get a whole lot better as
the years go on and we're really really
excited about it what I want to show you
now it would never go show you a little
demo of something we've never been able
to do in quick time before I'm going to
do it in h.264 which you would think
would be the hardest place to do it
we've been talking about high def
content a lot today and as you can tell
Apple's very very interested in high def
we have we can support hi def it's a
standard frame rate
30 frames 24 frames per second we can do
amazing things with large frame sizes
but there's another flavor of high def
that that's often call it referred to a
60 p and this is actually 60 frames per
second I'm going to show you a little
demo of quicktime playing this content
at 60 frames per second first thing you
notice I pull down the menu and it
didn't stutter pretty good pretty good
now so what's interesting here this is
you know the video itself is not
terribly dramatic you saw a different
clip of this this morning but it's
playing at 60 full frames per second now
you would think well this is yeah this
is a computational problem you've got a
really fast computer actually it's not
that heavy duty of a compute of a
compute problem we're not using that
much of the CPU but what's the what the
problem has always been has been that
you can't get the frames to the cards
fast enough to be able to keep up
because of the fact that we're now
layering QuickTime on top of course
video we're able to do this perfectly so
core video combined with QuickTime is
not only providing us access to these
amazing image processing capabilities
that are built into the GPU it's also
providing some absolutely outrageous
abilities to transfer data over to the
card which is obviously the other half
of the problem if you can't get the
stuff to the card fast and if you're not
going to be able to do anything with it
we're working with the core video guys
as well to come up with very highly
optimal methods for pulling the data
back if you can't pull it back out and
you're not really going to be able to
get it out firewire to your cameras or
wherever else you want to go very
effectively and so we're working on a
lot of problems in this area and it's
very exciting that we're real airing
this stuff the way we are so that's
quick time playing 60p content something
we've never seen before
Peter referred to the session earlier
there's a whole bunch of information
available about what we're doing in the
area of 264 in particular the underlying
format is outrageously complex and we're
introducing some new capabilities and
quick times you be able to manipulate
these formats in the ways you're
accustomed to being able to manipulate
videos today in quick time even though
when you learn the facts you would think
it would be impossible to edit this
stuff ever and we're going to make it
very very easy so that your applications
have no changes in the vast majority of
cases and if you do extremely
sophisticated stuff you'll have make it
you'll have to make a few changes and
it's going to be very exciting okay so
the next thing I want to talk about is
some of the specifics about what happens
when you put quick time on top of these
courts technologies I showed you some
data transfer optimizations that are
pretty exciting for a bunch of people
who are focused on video let me show you
some other things that we can do now
that are very cool now I'm not going to
I don't have slides with words i think
the demo will will say it all here we go
this is a little test app that we
developed in the last couple of days
that's another very important point this
stuff has gotten so easy to program
because we're really working all in
cocoa now and it's just it's amazing how
fast these things come together and
you're not really making you're not
making performance compromises by going
down the easy path then credibly
important so here we have a video iRobot
trailer and the first thing you'll see
and turn the bond down a little bit
first thing that you can see is it just
like a lot of the apps that we're making
now live resize very smooth moves like
butter but there's a lot more going on
here than meets the eye so I'm going to
zoom out zoom back from the video a
little bit and what you can see is I can
just grab this video and move it around
again I'm not losing frames or anything
like that that's kind of neat what you
can also see is that actually what we're
doing is we're playing video on to a
surface okay well that's that's pretty
cool this is sort of the infamous G if I
can only play to a GL surface than I
could do all these wild and crazy things
but that's always been virtually
impossible we've made that really really
simple now so the next thing we're going
to do is we'll just do a little fade
back here to normal and i'm going to
show you we built some presets into the
application to allow me to talk and make
things change at the same time and let
me show you a couple of them so this
first one is just going to take the
video and we're going to just slide it
down into the corner now we have the we
have all these frames flying through for
video and want to be able to kind of
visualize what that might look like we
built this groovy groovy little feature
so let me fit the frames out now let me
show you this is not this is not like
rendered it's not faked up I mean this
is all lies this is live i can do all
the same kind of 3d stuff that we were
doing when it was just on the single
surface let me go back to normal so we
do an animated sip it we can do a little
slow mo thing because it's the way these
things have to work
we've got another one here this one will
take it and put it over in the other
corner so now the frames are coming back
out this way you know that's kind of fun
to go back and forth we've got another
little effect we can do here where we
just start taking the frames and
twirling them that's kind of fun this is
one I always wanted to see it's sort of
our matrix our matrix effect here we'll
take the video I'll put it back in the
straight-ahead mode so now we're looking
into the frames that can bring back it
out kind of cool and then we can go back
go back to normal so this is an
obviously a very small example of the
kinds of things that you're going to be
able to do with quicktime layered on top
of the course technology it's really fun
it's really fun so that's quick time on
court what's going wrong what in the
world there we go what
thank you okay quicktime in core audio
so we've been talking a lot about
quicktime on the image stacks i'd talk a
little bit more about what's happening
with quicktime on the on the audio side
so we've basically re layered quicktime
entirely on top of core audio has some
very very obvious benefits we're going
to be able to support all the same high
def audio formats that you can support
in core audio today will support
including surround audio formats and
when you go to some of the other
quicktime sessions where they'll be
talking about this in detail you'll see
some amazing demos this also has an
incredibly important but perhaps lesser
lesser known effect which is that the
quality of the synchronization across
tracks is going to be rock solid we're
going to be down to sample level
accuracy in the synchronization on part
of the system which is fantastically
important and you'll be able to leverage
the DSD capabilities that are present in
audio units so i think i have a little
demo of that as well yeah okay so i
showed you a very simple time scaling
demo in the au lab application i want to
show you what you can do with this
applied to more of a real world
application so we have a little movie
here harry potter trailer this is what
happens when you normally go fast
forward know if you get to get the kind
of tip monk thing okay well that's cute
but not a lot of fun so change the mode
that quick times running in when you
start start the movie up again this time
let's get it going now listen to what
happened double edged 390 school
you almost can't tell that it's been
sped up because you can actually
understand what's happening now so we've
taken this one step further and we
actually added a little menu in here to
be able to access the controls a little
bit more directly since the fast forward
is sort of more constrained and I can
play it along so now I've got the
ability to speed it up okay Congressman
Mike took pole to be able to pound you
can try that again let's try that again
and it's not going to work 10 page 309
got my pitch okay I think it's working
now
Sirius Black is the reason supporters
head back and now you want to finish
what you started anyway you get the idea
you can modify the pitch in the tempo
and we're going to come and try and come
up with some clever ways to be able to
integrate distance and more of the
products that we build but we think it's
a very interesting technology I just
want you to quick glimpse of what would
be possible as we start to leverage
these inside of the whole stack so the
last thing I want to talk about is
something we call cutie kit so we have
gotten a lot of feedback about what it's
like to program quick time I'm Mac os10
and I've heard loud and clear creek
times very hard to learn there's 2400
api's how am I ever supposed to figure
this out you use data types that I don't
like I don't understand what they are I
don't want to have to create them you're
messing up my life why do I have to do
this so I heard it I've heard it so I'm
very very happy to report that we're
introducing a whole new framework to be
able to work with quicktime and it's
going to give you a very rich set of
services but like a lot of the other
services that you find in the system
these days going to be really really
straightforward to use fantastic
impedance match with Coco of course and
it's going to be able to take advantage
of all the other advancements we've been
talking about today we'll be able to be
combined with all the other tool kits
and you're going to be able to do
amazing things to help me illustrate
some of the very basic elements of this
I want to ask oh yes sir we asked Tim
Monroe to come up and show you a very
simple demo Tim's one of the designers
of QT kits and it's very cool I done hi
thank you
so one of the first things people ask
when they get a new framework in cocoa
is what can I do for free show me the
zero lines of code demo and that's what
i want to show you here now so what i'm
going to do is to show to emphasize with
their zero lines of code actually first
i'm going to clean up here okay i'm
going to do everything inside of
interface builder so you can't write any
code in interface builder so this will
prove it so here's a new palette for the
quicktime movie view the QT movie view
which is a central view aspect of the
new kit what i'm going to do is ask for
a new Koko application so now I've just
got a window here and I will drag the
palette item into the window and there
you can see it already looks a little
bit like a movie let me resize this to
fit the window and then I want to set a
few attributes of this first I want to
set this size so that when I resize the
movie window the QT movie view will
resize automatically and then let's look
back at the attributes pain you can see
I've got sort of a handful of things i
can control here i can show or hide the
controller bar i can make the movie
editable or not edible and let's just
stick it with that the last thing I want
to I can do here is attach a movie to
this cutie movie view and I'm just going
to grab the harry potter and now i can
go ahead and run this inside of
interface builder I'll test interface
and I've got a QuickTime movie that I
can run with zero lines of code this is
using the accelerated visual context
I've got all the goodies in cocoa let me
just illustrate that briefly one thing
you like to get with a view is if I hold
down the ctrl key and click I get a
contextual menu so we've got a nice
contextual menu here that allows me to
control the movie another thing I can do
here totally with zero lines of code
is make some selections I can come up
here and cut perhaps go over here and
paste that that was such a good section
i'll paste it over here maybe i'll come
in here and cut this out and because
we're built on top of cocoa i can undo
all of this up to the very top i can
undo the cut undo the second paste undo
the first paste under the original cut
and now I'm back to where I can't undo
any more because I'm back to the movie
let me show you one more thing I'll
start it playing and again Tim sort of
talked a little bit about this but I can
resize this and you'll notice that I'm
getting sold frame playback while I'm
resizing it so that's your basic live
resizing and automatically for you by
our new cutie cat framework and that's
what I've got to show
thank you so that's really exciting you
know this stuff is very powerful we're
actually building quicktime player on
top of QT kit now so the quicktime
player that's been in your seed in the
in the in the developer seed that we've
handed out today is actually built on
top of QT kit it's a very powerful tool
kit there's a session that will cover
this in great detail coming up on
Thursday and you should check it out
very cool so here are four really big
developments that we're making
advancements we're making in QuickTime
this is probably the most profound set
of changes we've made the quick time
since it was introduced back in 1991
it's very very exciting we've got a
whole new architecture we're working
very closely across all the different
layers within the team to be able to
build highly optimized implementations
to be able to share data across the
different tool kits I think it's gonna
be extremely exciting and so that's
quick time so we're just about at the
end here I wanted to tell you one
reminder there is this graphics and
media lab it's a hands-on lab to
happening running all week long I'm not
sure if you know where it is but it's
downstairs in the back and there's you
know a lot of engineers hanging out down
there waiting to help you with your
applications to help you two and them
help you figure out how to do things
tell you about the new stuff and I hope
you have a chance to go down and see
this so told you to getting of the talk
we wanted to focus on some very valuable
technologies that would help you improve
the quality of your user interface would
help you unlock some amazing performance
and would help you build applications
and tools that we're going to stand the
test of time and I hope you can see that
with this staff quicktime courts opengl
on top of the great apple hardware we
think we've got a great platform for
innovation here and i just want to say
thanks i hope you have a great time with
it thank you
you