WWDC2004 Session 204

Transcript

Kind: captions Language: en ladies and gentlemen please welcome vice president Interactive Media Group Tim chef good afternoon thanks for coming I think the first thing I just wanted to say is thank you all for coming we've got a really busy week of activities this week and it's real important that you come and track us down and talk to us while you're here we spend a lot of people from Apple to come up here and meet you and get your feedback I know we're going to be dumping a lot of information your way today and over the next several days it'll be a whole lot more useful for everyone if we can actually have a dialogue about this so I just want to encourage you to find the Apple people and start talking to them about what's going on so today's session we're talking about graphics and media overview here this morning you saw Steve talk about some of the really exciting features we're going to be providing for for customers and for us developers in the afternoon we heard Bert Ron talk about some of the platform level things we're doing to try and really drive the innovation forward and in this session what I want to do is focus on the audio the graphics and the video pieces and our perspective on this is that these technologies are traditionally seen as kind of the multimedia technologies for multimedia types but we believe that the situation really is changing and what's happening is that if you want to build a world-class application in this day and age we think these technologies are absolutely essential to being able to get there so our presentation today is certainly going to have some of those qualities that you would expect in the the media session but there's also going to be a real heavy emphasis on technologies that can be used in a wide range of environments not just media application and as you've been seeing throughout the day there's a big emphasis on how the CPU and the GPU can work together to do some amazing things and you'll see more of that here so we're going to focus on three big issues that affect all of you as developers one of them is how can you achieve the kind of gorgeous look and feel that customers are coming to expect using Mac os10 the second the second element here is how can we achieve radical performance we've all been hearing the stories about how g5 is a supercomputer in a box and we've been hearing about the GPU and how it's racing ahead of the GPU in terms of a computational power how do you unlock that it's not as easy as you might want it to be and then last of all you're making huge investments in these software applications they have lifetimes which are measured in several years if not decades how do you protect that software investment from the changes that are undergoing taking place in the industry at the hardware level and lower levels in the operating system I hope we're going to be able to help answer some of those questions this afternoon so the first thing I want to do is step back a little bit and talk about some larger industry issues this is a original paper that Gordon Moore wrote all the way back in nineteen sixty-five this paper became the outlines the ideas that ultimately became known as Moore's Law and it was basically as you know the the statement about how the density the power the integration of computer chips would approximately double in complexity every two years this chart over here on the on the right is the original graph that he showed and what's very interesting to see is that in 1965 he made this prediction when the typical IC consisted of about 50 transistors keep that in mind as we go forward so spin forward about 20 years the original macintosh had an eight megahertz 68000 processor from motorola about 68,000 transistors spin forward another ten years we're all the way up to three million transistors in the very first PowerPC chip that we shipped in the very first power macintosh bend forward to this year well we've just introduced a 2.5 gigahertz g5 systems were all the way up to 58 million transistors so here we are looking back at this paper that was written in 1965 roughly speaking 40 years double every two years so that's a 20 doublings to the 20 it's about a million so he sitting in a environment where your typical IC had 50 transistors we're now up to 58 million transistors that's a pretty amazing prediction that this guy did so as we've mentioned earlier today there's a lot more happening inside that box than just what's going on in the microprocessor the chips that we ship in our current systems the Radeon systems are already up at about a hundred and fifteen million transistors this is about double the complexity of the of this microprocessor the the system that Steve introduced this morning the Nvidia the new nvidia geforce 6800 ultra processors have 222 million transistors four times the complexity of the g5 that's an incredible opportunity for all of us be able to enhance the functionality of our applications but unfortunately it doesn't happen automatically and the question we're going to try to address today is how to unlock this stuff there are many many obstacles so in 1984 this was the statement that Apple made about the importance of rich graphics interface WYSIWYG and interactive you I this was really a dramatic statement this brought into the mainstream concepts which had not been seen in a personal computer before spend forward 20 years to this year Apple's introduced this product called motion I think it's a very good example of the way the expectations have evolved over the 20 years still rich graphics still essentially a WYSIWYG kind of an experience very much an interactive you I but obviously the kind of sauce are required to build this interface is incredibly more complicated than the software required to building the app of 1984 let's look at the operating systems that you have to learn and understand Apple has you know roughly speaking every single year we're introducing another thousand man years worth of software every single year and you can see this in a sort of very simplistic view of the size of the disk that we have to ship to you in order to deliver the operating system we've moved to a DVD this year because yes it's growing again how are you going to manage all of those toolboxes there's so much stuff to learn as we introduce more and more systems with more and more diversity of hardware with fancier and fancier architectures the question becomes how the heck you get all this performance out of the box it used to be that you could sit down and you learn the instruction set these rights and programs and you'd learn what was fast and then you'd write that in your code and you have the fastest appt on the planet it doesn't work like that anymore it's really really complex and the performance decisions that you would make for a g3 will be different very different than what you're going to do on a g5 how can you do that automatically this is not a puzzle you probably want to spend all of your time trying to solve when we sat down to think about what would be most profound in our contribution to tiger in the in the graphics and media team that I run we wanted to look at these issues and say how can we unlock this power for you protect your software investment and give you tools to build beautiful UI and this was our goal so we want to build the technologies that are going to allow you to build the next killer app because we believe as cool as these demos that you've seen today are we believe that you folks are the ones who are going to have the really really exciting ideas I mean it's neat to see core image processing these images and we're going to show you some more of it today but we are absolutely positive that your inventions are going to blow us away amaze and delight your customers and just just baffle the world how do they do that so here's the architecture of the graphics and media layer as a panther basically very familiar a block diagram here we've got quartz 2d we've got quick time we've got OpenGL all layered on top of the graphics hardware this has been a very very powerful architecture for Apple there's only one problem here you could look at this another way as three stove-piped technology staff and as many of you will know if you start to try to combine these technologies drawing the lines from left to right right to left horizontally you start to run in some real challenges so in Tiger one of the most important changes we are making is we're kind of turning this diagram on its side in order to give a much better much more powerfully layered system for you to work with now the api's aren't like all different in Tiger it's your applications are still going to be compatible but we're working down in the lower layers to reorient this architecture in order to maximize the kind of performance and the kind of data interoperability that we'll be able to provide through the API so that your apps will be able to combine all these technologies and still have a high performance result broken this up a little bit more detail you see the chord stack you see the core audio stack they've both got their their hardware abstraction layer conceptually and so what I want to do for the rest of the talk is we're going to dive in and look at what's going on in each of the individual technology areas and tell you some of the highlights of what's new and hopefully give you some plants and seeds for how you might be able to take advantage of these things our first areas core audio as we as we've expressed earlier today this is a very profound technology for processing of audio this is the most advanced audio sub system built into any operating system anywhere else when you look at other operating systems if you want to achieve the kind of performance that we're able to achieve inside of Mac OS 10 you always end up having to build special things into the OS you have to add special extensions in order to keep the kind of performance that you get with mac OS 10 out of the box what does that look like well to the ultra-low latency that allows you to do demos like the guitar demo you saw earlier this morning playing live in to the system it's processing the audio in real time sending it back out out the audio outputs and it feels like he's playing it live as if it were a piece of hardware in a in a music studio rack we've got high-def this was the first media processing stack to build in a full floating point processing pipeline and it's worked tremendously well for us not only does it allow us to support current state of the art processing 24-bit type samples we can go way beyond as the industry evolved we support all a very very wide range of sample rates and there's no constraints on the channel we have beautiful robust plug-and-play compatibility across a wide range of connectivity protocol we've got built-in natives built-in MIDI support very low latency just like the audio support and then we have this very important architecture for extending the system one of the most important parts of an audio subsystem is the set of customized audio plugins that you can bring to bear to provide your audio a unique kind of a sound or unique kind of processing now we've had a tremendous amount of success with our audio units and this is a small sampling of the developers who have gotten on board with audio units now force the universe of developers who are using Core Audio is much much larger but here's a really important collection of developers who are focused on audio unit development this success has brought some very interesting new challenges and some new opportunities as the community of developers who are centered around core audio continues to grow one of the things that we've been very concerned too to be able to provide is a very high level of compatibility between all these different kinds of plugins and so earlier this year we introduced a program to a program and a piece of software that would help Audio Units developers ensure that their audio unit plugins were going to be robust compatible as they went from one apps to the next and to the next we're introducing today a new product called au lab which is a part of the developer the developer toolset that's going to take this one step further and I'd like to give you a chance to take a look at that in just a second and what this is going to allow you to do is it's going to help help the audience developer develop an even more robust audio unit that can be plugged into the ever increasingly complex and sophisticated range of audio applications so why don't we come over and take a quick look at this long walk so we have this little application called au lab and basically the model of au lab is it's a mixer and the first thing we're going to do is we're going to come in here and we're going to add a generator and we've built into this tool a generator that's capable of bringing an audio from a file now of course it can also process audio in real time coming in through the different kinds of inputs that you can support so the first thing we're going to do is we're going to open up the little configuration panel for this bring in an audio file and we can play it just like a audio playback system great now what you can do is you can bring in your audio unit plugins and load it into this little mixer and it's now it's now been inserted into the processing chain and what we've got here is we've got a new audio unit that we're providing and tiger called time pitch and what it has the ability to it's a very sophisticated DSP algorithm that allows you to alter the the timing of a piece of audio without altering the pitch or you can alter the pitch without altering the timing and I'll do a very give you a very quick example of what this might be useful for so this is a piece of music by a couple of guitar players John McLaughlin aldi meal two of the fastest guitar players I've ever seen in the universe and what you encounter with this kind of guitar playing is that they start playing really really fast my son is a guitar player and he's always asking how can I learn how to learn this music it's going too fast this little plugin basically allows you to take these files and you can dramatically slowed them down without altering the pitch you can go faster and again it sounds just like the just like the original just like the original audio so there's a very simple demo of how a you lab can be used to validate and test out your audio units and we think it's going to be a very important part of our overall strategy for ensuring broad adoption of audio units and being able to help facilitate a very powerful community in the audio world relying on core audio so here's another situation that's coming up more and more as these core audio applications are being enhanced typical kind of a studio setup you've got multiple audio devices today in core audio we can handle these devices just fine but when it comes to managing those devices all the burden of managing which channels are going where and the inter device synchronization is the burden is on the application we're going to be introducing a new technology that allows you to aggregate these devices together into one logical unit and we think it's going to make the process of developing applications for sophisticated studio setups much much easier last of all I want to talk about open al open al is an industry standard API for managing spatialized audio it's used all over the place in games so technology developed by creative labs and we have been working with them and other developers to build a highly optimized version of a open al and it's capable of supporting not only super high quality facialized audio processing but it's also got a variety of modes to be able to do lower complexity mode so you can get lots and lots of channels and we're going to build this into Tiger and we think it's going to be a great addition to the OpenGL technology for a complete solution for game development so this is the story about what's happening with core audio what I want to do next is I want to ask Peter graphic nino our courts extremists to come over and you about everything that's going on in the courts world and it's a very large world I guess I can go to the podium other rear okay so I'm going to walk you through the technologies in the graphic stack looking through starting angle basically walk you up the stack from the hardware talk a little bit more about that OpenGL and then talking about courts and then Tim's going to come back and talk about good time so on the hardware side you've seen this graph many times this is pixels process per second and if you look at also memory fill rate the current generation of graphics chips are about 35 gigabytes per second of memory bandwidth which is pretty incredible and as you've heard a couple of times already this is kind of an inflection point in computer graphics we now have programmability at the pixel level it's floating point it's accurate enough for for high-end work and high dynamic range it enables a lot of applications beyond just the traditional gaming stuff so we're seeing a convergence of graphics and media processing together in the GPU and there's some new programming paradigms you heard me in Bertram's talk talk about the stream computing model that we're using for core image and tiger head will advance the state-of-the-art here in bring a lot of ways to bring the power of the GPU into your applications so there's a bunch of things you can do in Tiger as far as a GPU programming there's low level access via OpenGL there's accelerated 2d graphics we have with core graphics which we'll talk about in a second we have accelerated image processing with core image we we have core video which we're also talking about basically we've got a bunch of ways to leverage the power of the graphics processor and the trick for your apps is to try to use the highest level of abstraction you can which allows us to do more of the heavy lifting there's nothing wrong with going down to be OpenGL layer if you need to do that or want to do that but we've got these higher level services as well which you could take advantage of and let the platform do the rest so the the tiger quartz layer is optimized to take advantage of programmable GPUs now it's not necessarily required to run tiger beyond a program will part but it's optimized in that way and by programmable GPU what do I mean I mean on the ATI product line any part that the ati radeon 9600 or higher or on the nvidia side the geforce fx or higher or the new card that we just announced today by technology i mean ARB fragment program if you're an opengl programmer or on the windows side you sometimes hear called directx 9 capable hardware that's the basic class of hardware now interestingly enough if you look at the GPU versus the CPU the the GPU is not necessarily 0 is faster so something to keep in mind that you really need to treat it as a kind of a co processing environment the high-end CPU if you put tool Duke to dual gigahertz to 2.5 gigahertz g5 in a computer with a low-end but programmable graphics part the CPUs will be able to easily easily beat the low-end GPUs so you always have to be aware of the trade up and really view the GPU as kind of a coprocessor and also understand if you're flexible about using GPU or cpu that your apple scale to a much greater level with the GPU scaling factors that we're seeing that are exceeding moore's law that's all I'm going to say about hardware just motivate you guys to learn more about what's going on there talk a little bit about OpenGL OpenGL is the the foundation and sort of the hardware abstraction layer for our graphics hardware as Tim mentioned and there's a bunch of things going on in cut and tiger there's the OpenGL shading language which will be supporting there's floating point pixels support some of this made its way into Panther there is major enhancement to the OpenGL profiler to tool which is a really popular developer tool for analyzing your OpenGL performance there's new resource management improvements we're probably on our third or fourth generation of kind of treating the GPU as a full-fledged resource to be managed within the colonel there's a lot of new stuff to support many of the things 15 today and I'll see throughout the conference and I did want to mention one session that was not in the show guide here which is the introduction to OpenGL shading language which is going to be on Friday so be sure to check that out that's OpenGL let's move up to the courts player this year we're sort of dividing the courts layer into three things there's core graphics of course which is our 2d graphics and windowing system and then there's core image and core video which are new this year core image is our image processing engine and core video is our video processing engine on the core graphics I'd we're not staying still here either so let me talk about that the big news in core graphics land is that course 2d goes extreme so if we're talked a little bit last year about putting having quartz 2d on OpenGL and this year it's going to be the default in tiger so we have quality with the 2d rendering through OpenGL that's virtually I did identical the software quality and any of you guys who who know about how GPUs make different trade-offs about 2d graphics it's actually pretty tricky to do this in all cases to get really high quality text with OpenGL is kind of a pain but we've taken care of all of that we catch even LCD quality sub-pixel position glyphs and can render them and flip them onto the screen with all the proper blending you need to do to do that so this acceleration of course duty however does require programmable hardware to be able do the LCD bleeding and all the programmable blend mode the low level benchmarks when you put quartz duty on top of GL increased by two to a hundred times so there's some real impressive performance gains to be had but the key to getting those are to reuse your resources so if you have CG image reps or CG pattern ref things like that and you're going to draw them more than once just be sure to hold on to that and it will get cached up in video memory the other thing since we can't accelerate quick-draw if you use quick-draw within a window and you're using courts to the extreme we fall everything back to software so the core primitive vents benchmarks with quartz 2d you can see the increases here pretty striking obviously GPUs are great at filling memory so in 800 x 800 rectangles 236 times faster it could probably even go faster for you to figure out how to feed it line drawing for example 8 x text strings our software text path is actually pretty tuned at at one and a half two million blitz per second but we're getting almost 5 million glyphs per second with the with the hardware which is really good so we didn't turn it on by default in the wwc build for those of you know about the courts debug tool you can go in and turn it on it works pretty well it works a little bit better on a TI hardware than nvidia hardware right now but you can turn it on we know that there are a few bugs in there but it's going to try out your app and see if things get faster if not go catch your pattern reps and image rest and you'll probably see quite a performance performance difference the other thing to look at in quartz debug is you'll notice a show user interface or resolution menu item and what we're talking about this year at WWDC is getting ready for for a resolution independence in the user interface and the school kids and i think this is thanks this is going to have a profound impact on our ability to build products together and just on the on the way that users can interact with the machine in your application and there's a lot of work to do on this and it's both our job and your job in the applications there's no silver bullet that we can just magically scale everything we've got scalable graphics api's with quartz 2d and OpenGL but there's still pixel crack assumptions and things like that in people's code so actually what I'm going to do now first thing I'm going to talk a little bit more about that and then give you a demo so the the motivation for resolution independence there's a number of things there's the real estate versus detail trade off on those big monitors you might not want to use it all for detail you might want to actually you might not use it want to use it all for real estate rather you might want to use it for detail and just have finer text or higher res ered more high resolution displays that sort of thing but we really need to break the 72 DPI assumption that's kind of wired into a lot of code out there and it's really the once you have a scaling matrix in your graphics API we found the next problem is you're going to have pixel boundary assumptions in code where you know what scroller lines up on an integer pixel boundary well that may actually be in between two pixels if you if you put a scaling matrix in so there's three modes your application can run in and just cover them briefly here and then you can learn more in like in the cocoa and carbon sessions because there's a little bit different work to do in both cases first mode is called framework scaling for carbon and cocoa this is where the framework takes care of all of those all of the work for you and unless you're doing your own drawing if you have a custom sub view or a custom a chive you on the carbon side everything should just work there's nothing you have to do the math coordinates will get scaled the drawing will get scaled if you're a carbon app and you don't want the system to do it for you you can also have what we have application scaling here the carbon app app cos for the scale factor enforces that itself it can ask for you I to be drawn bigger but it might have its own graphics engine that it knows how to scale already and then finally we have magnification which is just sort of a accessibility style zoom where the windowing system courts compositor we'll just scale your app for you and that's for unmodified carbon apps although there are some cases where if you do have coordinate assumptions even that may not work exactly the way you want but find out more in the sessions like I said and let me do a real quick demo of that on the demo machine here just to show you courts debug if you don't know courts debug you can find it in the developer tool that's under the performance tool you can see I've got courts to the extreme enabled here I'm going to go up to that Tools menu and bring up the user interface resolution and let's crank up things a little bit and give us kind of a virtual scaling factor of about that let's do something something big like 1.75 and now any app I launch is going to get a 1.75 scale factor so let's launch Safari and see I get a huge menu bar and I'm not connected to the Internet well that's too bad but you can see how the the app is drawn much larger the menus or high resolution rendered you can see the rest of the if I switch to another app for example you'll make it clear here's courts debug menus are still small whereas so far as menus are big and so you can see that there's going to be some drawing bugs like over here by the Google search menu which were you know we're working through with the apps and the framework but you should use this tool on the developer CD and test out your app with with the resolution independence so back to slides please the next thing we're doing in in court studi is floating point pixel support so yes thanks what comes after millions its Jillian's banana but it's full floating-point pixel processing pipeline floating-point both on the source and destination so CG image reps can be floating point as well as PG bitmap contexts as destination for drawing we use unclamped colors process all throughout the pipeline and we're going to have a floating-point CMM based on the color sink in the tiger GM that's pretty exciting and to go with that we have a new framework and core graphics called image i/o which is a thread-safe image handling library for file i/o and it includes new HDR high dynamic range floating-point format such as openexr from ILM and various flavors of floating-point tips that are out there next up is PDF get PDF kit is a high-level framework for dealing with PDF as an objective-c framework up at the app kit layer it's kind of you can think of it as a preview on a pallet and much in the way that WebKit is sort of Safari on a pallet PDF kit is the PDF sort of half of preview running on a pallet complete with linked reversal and printing and and you know fit the page and all that sort of stuff so right there's going to be a session on that as well and I'll give you a point of that in the sec but I thought first I would talk a little bit about quick-draw and all the things we're doing quick draw in tiger which is nothing except not break it so this brings me to the point where I have to basically tell you that quick draw is not evolving anymore we're not going to give it floating point coordinates or alpha support it's really been limited in hardware acceleration floating point pixel support resolution independent core image integration all of these things would be very difficult to add to quick draw and frankly just not really worth our time or your time changing all your quick draw code so you know as is a little tribute to quick draw though quick-draw has served as well I mean it's been a brilliant invention you know far more profound perhaps than anything we're showing you this year it really rusher din you know a whole generation of graphical user interface and had a number of really cool innovations in its day and it's been the cornerstone of Mac OS development for 20 years but it's time to move on so we're going to be deprecating quick draw we're not going to break your apps immediately of course 2d entire will fill in the last of the big holes that we've definitely gotten feedback on the headers are still available but they'll be marked deprecated if you compile against the tiger target binary compatibility of course the league's it will continue to exist at least for now and you should definitely budget time in your next revision of your app to move to core graphics and we really want to make it work for you so we're definitely open to feedback and we have a whole session about transitioning the courts 2d that specifically targeted for quick-draw developers and got a lot of new stuff that should make it easier and we want to hear about more things you might need so to help motivate that a little bit let me do a quick demo of line-drawing performance so this is one thing we got feedback on as being much faster in quick-draw than it was in court 2d course d D to D would lovingly anti-alias all of your intersecting polygons and sum N squared algorithm and would just be really slow so people would use quick draw for stuff like this well we've improved courts to D such that it's now 1.5 million and this is actually the software renderer and once we go on top of GL we're actually up at 11 million lines per second great and this is still only immediate mode in open jail if you know anything about OpenGL there might be another 5 10 X on top of this as well but clearly GL is the way to draw a line and so just to drive at home we're looking at you know 41 we didn't draw the initial courts 2d software which is somewhat less than one point out at some of you might know but getting onto course 2d is clearly what you want to do here so back to slides so some bonus sessions in the 2d graphics package 2d graphics pass the high dynamic range is going to be talked about together with the image io framework that system is going to be on Wednesday and then we have the session on PDF kit tomorrow so please attend those talk a little bit about core image you've heard about it in the keynotes and other talks again it's a framework for GPU focused image processing with a rich set of built-in filters and our plug-in architecture the great thing is that you don't have to know about OpenGL to use core image which is a nice a nice feature for those of you coming from the the 2d world core image is actually as an objective C API it's based on a stream based processing model the number of abstractions is very few there's images and there's kernels which are the processing units and then there's samplers which basically kernels use to access data from within images kernels are described in a runtime compiled language as a high-level see like language the parallel ISM is implicit as i said before on the other talk there's no explicit loops or threads you just use the image unit or program the image unit to do what you want and the evaluation model since its implicit is mappable two GPUs symmetric multi-threaded CPUs or other parallel architectures you might imagine we have a software fallback that's actually quite good as well uses the velocity engine and a just-in-time compiler that's custom for core image and it generates Auto vectorized code optimized either 4G 4G five or even dual g5 with multi-threading if if that's available and it's the evaluation engine does a number of like sort of compiler techniques at the at the graph level as you as you construct image units together to minimize the number of passes and handle the temporary buffers p buffers and caching and things like that so it does a lot of heavy lifting for you and let me give you a demo of that so when we were preparing for the keynote we did a lot of sort of experimentation with with core image and some image some imagery that we didn't use in the keynote I thought it'd be fun to show you guys here so here's here's a couple of interesting interesting filters here's a circular halftone screen on this image I can change the width it's all getting computed on GPU this is a radeon 9800 xt guard some other ones that are kind of fun the surfer wave reset back and let's do a zoom blur on him oops not white pointed Jeff so that's kind of cool and change the amount what else that I have let's do a layer effect so I'm going to bring a water droplet background image let's turn off zoom blur let me add another filter which is a piece of water artwork and let me add a rounded kind of 3d effect on that and you can kind of see see what that looks like that's pretty neat the other thing I wanted to show was kind of a build of one of the stacking effects builded the the electric zebra this morning and I'm going to show you a different one which is kind of also in the in the 60s theme so let's we're going to try to go for some kind of silk screen look so we're going to color poster eyes the image a little bit then I'm going to over saturate it using a over saturate filter I can use a false color false color just does it converts the image to grayscale and then maps it through a ramp and I've got a couple of colors here that I'm going to use so yeah and that's what I'm looking pretty psychedelic then the next thing I add in is every 60s poster needs a lenticular halo there we go which this is doing a the generator that generates kind of a Sun striation sort of a thing let's do that okay let's bring in some line art well that's okay the 60s album which is the type and will insert a bump map in there to do our bump distortion and then lastly i think i have a crop in here app so you can crop out the edges so but the great thing is the whole stack is live i can totally non-destructive it's just sort of remembering that recipe to create the image and you can kind of turn off the layers and see see how each one is doing or go back and change the posterization level on the fly I can even go back and change the image on the fly I could do it too you know the water image if I you know my band changed or I can even run video through the background so you know you get your your album cover in your your video at the same time and it's all still alive I can find the bump let me add the bump back in i love forever saints of posterization from the layers off on real time that's pretty cool so you definitely want to go to the core image session hopefully we've teased you enough by showing it in every single keynote session so and it's tomorrow afternoon at five o'clock so you'll definitely want to see that core video core video is one way to think about core video is it takes the output of QuickTime and maps it into the GPU and so it's basically new for QuickTime is separating the decoding logics in the presentation logic so it solves the video on a texture problem which we know a lot of our internal developers and I know a lot of you guys as well have had we've had like four or five different examples on the website of how to do video on a texture none of them were actually optimal and we're just solving that in Tiger in a great way where you get total asynchronous a synchronous behavior and data flow between the GPU and the CPU and that's really what allows us to do some of these demos like 60 p high definition h.264 it allows also once you get the video frames up in the GPU you can use core image on them for video effects so bonus session on that of course the core image effect the core image taught as well as new directions for QuickTime performance where you're going to feel out of the stuff in action so that's core video so we have the whole stack of technologies we call courts at the core level the operating system and we think it's a great substrate for you guys to build apps on and I'm going to show you a quartz composer one more time I had a quick demo of it before I spend a little bit more time on it here quartz composer is an application that harnesses the power of the courts layer using core graphics opengl core video and all of the technologies compositions can be saved in files kind of like you can think of immense metafiles and they can have opengl info core graphics etc and there's a simple playback API it's a framework the engines built in VOS so you can load a composition and play it and it creates a procedural animation you can actually expose the variables outside of your animation to the key value coding system in cocoa and set parameters like images or text strings into the animation so let me give you a demo of that real quick here so let me navigate back again to the example of directory so it's in developer example quartz composer and I'm going to show you a couple of things this time that I didn't show last time like masks slideshows kind of neat basically take some of the images from the screen saver and puts them through a mask it generates a mask with with OpenGL and then does a blit using the the graphics hardware I think this one pretty much is straight it straight up OpenGL it doesn't use any core image so if used as blending modes and the hardware so that one's that's pretty interesting when we switch back to finder and show you distortion FX distortion effects is an interesting one so here's something you have you hadn't seen before with the other composer demo says drag and drop an input file into the input parameters and so what this composition does is it's all this wiring in the details of the plumbing but it exposes two variables and image in adoration and so it's waiting for input and so let's drag Copenhagen just an image of Copenhagen Harbor there and let's look at that that's full screen it's using some core image effects the OP tile and the bump distortion to do that so you can see how if you had an application and you wanted to do that to an image you could just actually create a quartz composer composition load it up and then just keyvaluepair iya Buhl's and you've got it you've got it going so to drive that home a little bit more let me show you another one we have which is a we have engraved your ipod's well we have engraved your g5 let's look at the input also to you down here I can say Peter's g5 and you can see it announced as Peters g5 you can't really see it too well so let's let's actually build an app based out of this out of this composition so I actually have a nib file there's nib file in the example directory there you can see we have a palette the quartz composer palette which is one of these I think there it is which has a controller which is the green thing and a view which is the quartz composer view and what's happening is I have a patch controller that it brought into my project down here which is controlling the patch the g5 engraving patch and I have a view and if we look at the parameters on the view by bringing up the inspector I'll look at the bindings you can see that the the the view is bound to the patch controller which is this guy here who's controlling the patch now if I look at this text field its value the mouse's little twitchy sorry about that the the value of the of the text field is bound to the patch into a variable in the patch called text value which is that thing was that was exported in the in the composer and so now if i run this I have a simple application and if I can just type in the text field and say Peters g5 see I can label my own g5 so it's pretty easy to go into quartz composer and create a procedural definition of either an animation or a still thing like this and then just wire it up and drive data into it with cocoa binding so we think that's kind of a pretty powerful integration technology so back to slides so we're going to have a session on the court composer called discovering the quartz composer it's not a 9am on friday but if you want to stop by the hands-on lab and get going with this stuff tomorrow morning a bunch of us are going to be in there to help you with that if you want to just look at the examples and get up and running on the tool so anyway so that's our platform of all the new stuff we're doing in courts that I just highlighted a lot of the new stuff there's tons of sessions i think there's about 15 sessions just on the graphics side of stuff that you want to talk in a bunch more on the quicktime and audio stuff so have a great conference and we will see you with the sessions in back to Tim thanks a lot here ok so we've covered audio we've covered graphics now it's time for the video stuff and of course that means quick time this year I'm not going to spend any time at all talking about marketing stuff we've got a tremendous amount of good news to share with you but tomorrow morning Frank Casanova and his team are going to put on a big extravaganza here talking about what's going on with the marketing side of QuickTime of the business side how we're doing in them out in the marketplace fantastic story going to tell you an awful lot about our 3G strategy which is a tremendous success in marketplace we have by far the best system for supporting mobile media in the world we supports all the all the global standards at this point and we've got just a fantastic and dance Torian and I think you'll like it we're going to focus today on the technical side of QuickTime this is the architecture of quicktime circa nineteen ninety-five you know obviously it's very simplified but the basic point here is that quicktime is a very modular very componentized architecture but the components tend to be oriented towards supporting quicktime so they're kind of inwardly focused we had video processing components they have audio processing components and all of that was layered on top of quick draw on top of the sound manager if we spin forward to look at what we're doing in in Tiger this year you're going to see some dramatic changes taking place down in the core of quicktime and we're going to be layering quicktime on top of quartz we're going to be putting it on top of core audio and there's going to be some tremendous features here we're going to take advantage of all these technologies and I think you're going to really learn really enjoy what starts to happen with QuickTime today I'm going to talk about for details that were for very very important happenings in the world of QuickTime that I think we're going to have profound impact on your applications and on the ways that quick time starts to proliferate around the world so the first thing I want to talk about is h.264 you saw some demos of this earlier let's look at what this is really about so you all know what MPEG is these are the guys who make dvds digital TV it's actually this this compression technology is actually a collaborative effort between MPEG and a lesser-known organization called itu the International Telecommunications Union these are the guys who back in the 1860s the 1860s first set up interoperability and interoperability organization to ensure that people would be able to Telegraph across national boundaries they then applied that kind of mindset to radio and more recently they do it with telecommunications and they are really one of the premier organizations for advanced technology in the area of compression as it applies to communication the goal is very simple build the best codec that anybody has ever seen it supports a lot of different modes we'll talk about that some more in a second what you may not know is that Apple's been involved in this standard for a long time and we've actually got a whole bunch of patented technology built into the core of the standard and the format is built on the QuickTime file format of course so that's a great thing here's a little bit of a time line chart showing you kind of how the the video compression efficiency story has evolved the upper line shows you the story for mpeg-2 and the lower lines are showing you what's been happening with some of the more modern codecs and the orange piece obviously is the 26 for a couple things to notice we learned an awful lot about video compression over the last 10 years and a lot of that was able to be played out in the mpeg-2 standard so even though the standard was established in 1994 for the next ten years we saw dramatic improvements in the efficiencies of mpeg-2 so you sometimes hear a story which says well your codec isn't good enough you don't have the right standard the thing you got to separate out here is the politics from the reality of the technologies and what we know is that with each successive generation these technologies they're an incredible a number of tools that are built into these little algorithms that we all have to learn how to use and this shows you in a very graphic way what happened with MPEG over the years I mean this is dramatic they went from almost 6 megabits down to just under 2 megabits to achieve the same level of quality and what we're going to see with mpegs with mpeg-4 part 10 h.264 AVC goes by a billion different names we're going to see again a continuing evolution as we learn how to use this codec let me give an example of the kinds of stuff that we're working through so this is mpeg-4 part 2 this is the version of mpeg-4 that we ship in QuickTime six today that we've distributed literally hundreds and hundreds of millions of copies of there this is a sort of a summary of the various algorithms that to get are combined together to create the overall compression effect h.264 offers this vocabulary of tools so you might imagine that a software developer trying to build these kinds of technologies is going to take quite a while to learn how to use them optimally we've been working on this for a long time we think we have a fantastic implementation already but it's going to get a whole lot better as the years go on and we're really really excited about it what I want to show you now it would never go show you a little demo of something we've never been able to do in quick time before I'm going to do it in h.264 which you would think would be the hardest place to do it we've been talking about high def content a lot today and as you can tell Apple's very very interested in high def we have we can support hi def it's a standard frame rate 30 frames 24 frames per second we can do amazing things with large frame sizes but there's another flavor of high def that that's often call it referred to a 60 p and this is actually 60 frames per second I'm going to show you a little demo of quicktime playing this content at 60 frames per second first thing you notice I pull down the menu and it didn't stutter pretty good pretty good now so what's interesting here this is you know the video itself is not terribly dramatic you saw a different clip of this this morning but it's playing at 60 full frames per second now you would think well this is yeah this is a computational problem you've got a really fast computer actually it's not that heavy duty of a compute of a compute problem we're not using that much of the CPU but what's the what the problem has always been has been that you can't get the frames to the cards fast enough to be able to keep up because of the fact that we're now layering QuickTime on top of course video we're able to do this perfectly so core video combined with QuickTime is not only providing us access to these amazing image processing capabilities that are built into the GPU it's also providing some absolutely outrageous abilities to transfer data over to the card which is obviously the other half of the problem if you can't get the stuff to the card fast and if you're not going to be able to do anything with it we're working with the core video guys as well to come up with very highly optimal methods for pulling the data back if you can't pull it back out and you're not really going to be able to get it out firewire to your cameras or wherever else you want to go very effectively and so we're working on a lot of problems in this area and it's very exciting that we're real airing this stuff the way we are so that's quick time playing 60p content something we've never seen before Peter referred to the session earlier there's a whole bunch of information available about what we're doing in the area of 264 in particular the underlying format is outrageously complex and we're introducing some new capabilities and quick times you be able to manipulate these formats in the ways you're accustomed to being able to manipulate videos today in quick time even though when you learn the facts you would think it would be impossible to edit this stuff ever and we're going to make it very very easy so that your applications have no changes in the vast majority of cases and if you do extremely sophisticated stuff you'll have make it you'll have to make a few changes and it's going to be very exciting okay so the next thing I want to talk about is some of the specifics about what happens when you put quick time on top of these courts technologies I showed you some data transfer optimizations that are pretty exciting for a bunch of people who are focused on video let me show you some other things that we can do now that are very cool now I'm not going to I don't have slides with words i think the demo will will say it all here we go this is a little test app that we developed in the last couple of days that's another very important point this stuff has gotten so easy to program because we're really working all in cocoa now and it's just it's amazing how fast these things come together and you're not really making you're not making performance compromises by going down the easy path then credibly important so here we have a video iRobot trailer and the first thing you'll see and turn the bond down a little bit first thing that you can see is it just like a lot of the apps that we're making now live resize very smooth moves like butter but there's a lot more going on here than meets the eye so I'm going to zoom out zoom back from the video a little bit and what you can see is I can just grab this video and move it around again I'm not losing frames or anything like that that's kind of neat what you can also see is that actually what we're doing is we're playing video on to a surface okay well that's that's pretty cool this is sort of the infamous G if I can only play to a GL surface than I could do all these wild and crazy things but that's always been virtually impossible we've made that really really simple now so the next thing we're going to do is we'll just do a little fade back here to normal and i'm going to show you we built some presets into the application to allow me to talk and make things change at the same time and let me show you a couple of them so this first one is just going to take the video and we're going to just slide it down into the corner now we have the we have all these frames flying through for video and want to be able to kind of visualize what that might look like we built this groovy groovy little feature so let me fit the frames out now let me show you this is not this is not like rendered it's not faked up I mean this is all lies this is live i can do all the same kind of 3d stuff that we were doing when it was just on the single surface let me go back to normal so we do an animated sip it we can do a little slow mo thing because it's the way these things have to work we've got another one here this one will take it and put it over in the other corner so now the frames are coming back out this way you know that's kind of fun to go back and forth we've got another little effect we can do here where we just start taking the frames and twirling them that's kind of fun this is one I always wanted to see it's sort of our matrix our matrix effect here we'll take the video I'll put it back in the straight-ahead mode so now we're looking into the frames that can bring back it out kind of cool and then we can go back go back to normal so this is an obviously a very small example of the kinds of things that you're going to be able to do with quicktime layered on top of the course technology it's really fun it's really fun so that's quick time on court what's going wrong what in the world there we go what thank you okay quicktime in core audio so we've been talking a lot about quicktime on the image stacks i'd talk a little bit more about what's happening with quicktime on the on the audio side so we've basically re layered quicktime entirely on top of core audio has some very very obvious benefits we're going to be able to support all the same high def audio formats that you can support in core audio today will support including surround audio formats and when you go to some of the other quicktime sessions where they'll be talking about this in detail you'll see some amazing demos this also has an incredibly important but perhaps lesser lesser known effect which is that the quality of the synchronization across tracks is going to be rock solid we're going to be down to sample level accuracy in the synchronization on part of the system which is fantastically important and you'll be able to leverage the DSD capabilities that are present in audio units so i think i have a little demo of that as well yeah okay so i showed you a very simple time scaling demo in the au lab application i want to show you what you can do with this applied to more of a real world application so we have a little movie here harry potter trailer this is what happens when you normally go fast forward know if you get to get the kind of tip monk thing okay well that's cute but not a lot of fun so change the mode that quick times running in when you start start the movie up again this time let's get it going now listen to what happened double edged 390 school you almost can't tell that it's been sped up because you can actually understand what's happening now so we've taken this one step further and we actually added a little menu in here to be able to access the controls a little bit more directly since the fast forward is sort of more constrained and I can play it along so now I've got the ability to speed it up okay Congressman Mike took pole to be able to pound you can try that again let's try that again and it's not going to work 10 page 309 got my pitch okay I think it's working now Sirius Black is the reason supporters head back and now you want to finish what you started anyway you get the idea you can modify the pitch in the tempo and we're going to come and try and come up with some clever ways to be able to integrate distance and more of the products that we build but we think it's a very interesting technology I just want you to quick glimpse of what would be possible as we start to leverage these inside of the whole stack so the last thing I want to talk about is something we call cutie kit so we have gotten a lot of feedback about what it's like to program quick time I'm Mac os10 and I've heard loud and clear creek times very hard to learn there's 2400 api's how am I ever supposed to figure this out you use data types that I don't like I don't understand what they are I don't want to have to create them you're messing up my life why do I have to do this so I heard it I've heard it so I'm very very happy to report that we're introducing a whole new framework to be able to work with quicktime and it's going to give you a very rich set of services but like a lot of the other services that you find in the system these days going to be really really straightforward to use fantastic impedance match with Coco of course and it's going to be able to take advantage of all the other advancements we've been talking about today we'll be able to be combined with all the other tool kits and you're going to be able to do amazing things to help me illustrate some of the very basic elements of this I want to ask oh yes sir we asked Tim Monroe to come up and show you a very simple demo Tim's one of the designers of QT kits and it's very cool I done hi thank you so one of the first things people ask when they get a new framework in cocoa is what can I do for free show me the zero lines of code demo and that's what i want to show you here now so what i'm going to do is to show to emphasize with their zero lines of code actually first i'm going to clean up here okay i'm going to do everything inside of interface builder so you can't write any code in interface builder so this will prove it so here's a new palette for the quicktime movie view the QT movie view which is a central view aspect of the new kit what i'm going to do is ask for a new Koko application so now I've just got a window here and I will drag the palette item into the window and there you can see it already looks a little bit like a movie let me resize this to fit the window and then I want to set a few attributes of this first I want to set this size so that when I resize the movie window the QT movie view will resize automatically and then let's look back at the attributes pain you can see I've got sort of a handful of things i can control here i can show or hide the controller bar i can make the movie editable or not edible and let's just stick it with that the last thing I want to I can do here is attach a movie to this cutie movie view and I'm just going to grab the harry potter and now i can go ahead and run this inside of interface builder I'll test interface and I've got a QuickTime movie that I can run with zero lines of code this is using the accelerated visual context I've got all the goodies in cocoa let me just illustrate that briefly one thing you like to get with a view is if I hold down the ctrl key and click I get a contextual menu so we've got a nice contextual menu here that allows me to control the movie another thing I can do here totally with zero lines of code is make some selections I can come up here and cut perhaps go over here and paste that that was such a good section i'll paste it over here maybe i'll come in here and cut this out and because we're built on top of cocoa i can undo all of this up to the very top i can undo the cut undo the second paste undo the first paste under the original cut and now I'm back to where I can't undo any more because I'm back to the movie let me show you one more thing I'll start it playing and again Tim sort of talked a little bit about this but I can resize this and you'll notice that I'm getting sold frame playback while I'm resizing it so that's your basic live resizing and automatically for you by our new cutie cat framework and that's what I've got to show thank you so that's really exciting you know this stuff is very powerful we're actually building quicktime player on top of QT kit now so the quicktime player that's been in your seed in the in the in the developer seed that we've handed out today is actually built on top of QT kit it's a very powerful tool kit there's a session that will cover this in great detail coming up on Thursday and you should check it out very cool so here are four really big developments that we're making advancements we're making in QuickTime this is probably the most profound set of changes we've made the quick time since it was introduced back in 1991 it's very very exciting we've got a whole new architecture we're working very closely across all the different layers within the team to be able to build highly optimized implementations to be able to share data across the different tool kits I think it's gonna be extremely exciting and so that's quick time so we're just about at the end here I wanted to tell you one reminder there is this graphics and media lab it's a hands-on lab to happening running all week long I'm not sure if you know where it is but it's downstairs in the back and there's you know a lot of engineers hanging out down there waiting to help you with your applications to help you two and them help you figure out how to do things tell you about the new stuff and I hope you have a chance to go down and see this so told you to getting of the talk we wanted to focus on some very valuable technologies that would help you improve the quality of your user interface would help you unlock some amazing performance and would help you build applications and tools that we're going to stand the test of time and I hope you can see that with this staff quicktime courts opengl on top of the great apple hardware we think we've got a great platform for innovation here and i just want to say thanks i hope you have a great time with it thank you you