---
title: WWDC2001 Session 404
framework: wwdc
role: article
path: wwdc/wwdc2001-404
---

# WWDC2001 Session 404

## Transcript

Kind: captions Language: en this is a session 404 OpenGL high performance 2d and the first of five specific OpenGL sessions running today and tomorrow here at WWDC the way we structured structure the demos is to have each session addressing specific OpenGL development aspects from 2d 3d rendering and optimization so you'll be able to pick the information you need from the experts here presenting to you for your application development what we are not going to talk about is Hardware specific acceleration features we're gonna be general and so to prove to you that our OpenGL implementation is consistent across our graph platform regardless of your the hardware acceleration we have some really cool stuff we're gonna see specially on this demo here that OpenGL is incredibly stable and scalable and versatile which to us it's a great feature for again across our product line and the I must say that the OpenGL team has done a great job including the DTS folks and bringing about an employment a shin that runs both Mac OS 9 and my quest and consistently as off today so your application obviously can run on both our asses without glitches but this is really only the beginning what we really want to do is bring OpenGL to a new height and what I mean by that is to finally really prove to the industry that an open source open standard vendor-neutral 3d API is the best way to go so we're going to see some really cool stuff and what we really intend here to do is to give the best of our insight to you and to inspire you to come up and design the best 2 and 3d applications based on OpenGL and you know give us your feedback well we're going to be here for an hour providing you with a lecture we're really looking forward to hearing back from you hopefully in them in a very candid manner how do how do you we can help you bring your applications to to the platform and to leverage your investment to really easier development and and performance of your application using OpenGL so to present section 404 OpenGL high performance 2d I'm pleased to welcome DTS engineer Jeff Stowell thanks Sergio today I'm going to talk about how to use the OpenGL pipeline to get real impressive 2d performance out of the platform this includes both 2d or traditional viewing pictures those kind of things and two and a half gee maybe some work with sprites work with the movies and those kind of things and we'll see how you can really use the OpenGL pipeline to display those 2d kind of raster images so first we'll talk about why are we talking about OpenGL a 3d API for 2d work then we'll move on to implementing 2d with OpenGL things I'll talk about first is how to display images we'll talk about how to load images into textures and use textures to image from you can use this extrapolate this to using anything you generate from your program we've seen demos before in the past with using PDFs as a source of a texture and texturing that onto an object you need to do the same techniques there we'll move on and talk about sprites and side-scrolling now use side-scrolling kind of loosely with OpenGL obviously there's no limitation on whether you're going to use it for vertical and horizontal scrolling know where you're going to create the next great game the next to the next route creation of defender and then we'll talk about liubao QuickTime integration and beyond using just the QuickTime API for loading textures how we can use the QuickTime API and movie to integrate with OpenGL to display them first why why are we talking about OpenGL it really is the fastest pipeline on our system we've done a tremendous amount of work the engineering team has done some great work to get great throughput to the video card I would be hard-pressed and I think that most people in this room would be very hard-pressed to try and duplicate that effort for your own application unless you're an expert at HP graphics you have access to all the resources of the engineering team has and understand the PowerPC new processor inside and outside it'd be very difficult to get the kind of performance like six hundred and fifty megabytes per second through that across that AGP bus to the graphics card so I recommend as a way to get that high performance graphics to use a OpenGL it's also a flexible API the API is not specifically designed to work with 3d you can use it for two and two and a half D by just going to an orthographic context and kind of forgetting about that Z value and using that to display your 2d images also hardware acceleration we have folks from ATI and NVIDIA wandering around and that they've basically bet their companies on the fact that they can provide world-class hardware acceleration to the graphics pipeline and I think they've done a hell of a job so use that leverage that work that they've done and the board today Apple engineering is done for your applications and use the OpenGL pipeline for your apps and also there's lots of sample code out there if you get stuck there's tons of resources on the web OpenGL is a it's a very stable API with lots of code out there to teach you what to do and help you when you get stuck but I would be remiss if I didn't talk about why not OpenGL what things about OpenGL may turn you away what where where may you not go down the right path my pipeline here so to speak by thinking about it using OpenGL for your application first it works very well in 16 and 32-bit modes but it does have some problems if you're trying to get high performance in an 8-bit mode the 8-bit mode is gonna be you're gonna fall into the software case and in a lot of special cases where you have special law you're doing a pallet animation for example here you're doing pal animation OpenGL doesn't have a real equivalent of doing 8-bit pallet animation so I suspect the your application could be specially coated to handle that in a case of Oh better than OpenGL so that would be a case that you may want to not think about using OpenGL and use a special case of glitter of your own also let's say you're doing fonts you do you want to have very scalable fonts you want to do specific kerning with fonts or handle Unicode fonts OpenGL doesn't have any built-in properties to do this you could combine quartz and OpenGL to do this but really if you're trying to do like the next InDesign or page maker or those those kind of applications if I don't think about the using the courts 2d API because that really has a robust feature set for handling images handling fonts and those kind of things while OpenGL really is where the high-performance bit images can be handled so what we're going to talk about first went to hop in the image display talk about texture loading and talk about orthographic display and scaling then we're talking about sprites with alpha blending and alpha test it's talked about using some matrix manipulations to move them around the screen then we'll work that right into the side scroller showing it's showing a large scaling background with sprites flying around than that and lastly we'll talk about QuickTime integration first let's talk about image display one of the key things here is how to load textures how to get any text or any image you want into your system we're going to use a quicktime image compression API it's a great API for loading images you can bring up a nav services dialog and select any any image you want use QuickTime to load it you didn't have to worry about where there was a BMP a tiff I'll pick a JPEG whatever it'll load it for you it'll put it in a buffer you can use some of you have looked at this may say well hold on if I use new G world for that buffer I have some padding problems I'll show you how to get around that we'll show you how to use any buffer that you can take you're going to take a non padded buffer and you can use that directly for texturing to OpenGL with that you want to use pack pixels the pack pixels extension or the pack pixels that is in OpenGL 1.2 drivers will allow you to use a texture format directly from the image to an off-screen gee world to a texture without having to do any pixel Swizzle in whatsoever so you don't have to touch those pixels unless you want you to do some special effects the 3 meses methods we'll talk about is first a simple way OpenGL must have a power of two textures currently so what we're gonna do the first way we'll talk about scaling with Quicktime to a power of two texture onload and then fixing it as we display it next we'll talk about segmenting to power of two textures a little more complicated when you take a large image that is of abstract size any size you want and you segment it down so you get the pixel perfect representation of that image in multiple textures and lastly we'll talk about we're not gonna really talk about I'll talk about it here we'll talk about texture rectangle texture rectangles and extension coming down the road that will allow you to take any shape or size now any shape but any rectangular texture that is not necessarily a power of two and texture with that all the vendors are working hard to support that we're working hard to support that so look for that in the future when you're talking about image loading an image handling with OpenGL it'll allow you to handle textures of arbitrary size directly into the OpenGL hardware accelerator I think I better take a drink of water for this one the QuickTime image compression API this is let me actually back up a little bit with with the code here the code I'm putting up here will all be available on sample code there's no code you'll see on the screen that you'll not be able to get a hold of so don't try and take specific notes exactly how to do it on the on this all most of this stuff will be up before the end of the week some of its up already and the rest of the stuff will probably be up next week so you'll definitely get up the code and look at it yourself and look through it in decipher and figure out what we're doing with that so if you listen listen hard I think I can go through and explain exactly what's happening here first thing to do is get graphics importer for a file you get you have an FS spec from your your nav services dialog or something that you create yourself and you get graphics importer for football perfect for file it returns the GI comp variable there what thing I do is get the natural bounds of that basically tells me how big the image is so I know what I'm working with second thing I do after that is decide on what the stride of that image is too crude to create a pointer for to hold that image the reason I do this is that's a step you may not normally see that step creating the straw I tells you exactly bit for a bit what's the width of that image without padding and then the new pointer I'm going to create a buffer that's exactly the size of that image without any padding without any pad it's through a four byte or anything that a gee world may normally do I then call QT new Geo world from pointer it's not a function very often but it works very well because it will use that pointer I've created for that new G world and you'll be able your supplying the memory you're supplying the padding it'll say I know the information there I have the stride at the end so I'm gonna use that packed image buffer for the G world and I'm not going to allocate any memory myself that is key to using PAC pixels and using direct texture into OpenGL because that allows you not to have to skip rows not to have to do strange things with the image before using it with OpenGL and you see there's two different versions here the reason there's two different versions and you probably could could make it one with assigning that constant to a variable but in this case there's I wanted to show you that there is 2k 32 AAA RGB pixel format and in K 16 16 B e55 pixel format those are open I'm sorry there's a QuickTime constants then indicate the QuickTime what kind of pixel format you're gonna have in that G world there so now what are we talking about remember I said the first technique uses a scaled image into the texture for Oprah texturing make it easy and we're gonna scale it to whatever power of two is closest in this case I'm gonna use QuickTime to do that QuickTime this scale image matrix works really well the reason why it works well the first time it has to set up the MIT has to set up the scaling but after that for example if you're using the same kind of scaling with a movie it's set up a lookup table and it does a lookup type of scaling so it has accurate and quick scaling and really I haven't seen any performance penalty from using QuickTime to scale images on load so you set the identity matrix and then you're set your scale matrix it's pretty simple basically you're taking the difference between the check texture whit your desire and the image width you have and you're scaling it down it's gonna fit into that matrix remember to note that you are at this point losing information you've actually throwing away pixels so then if that's important to you if an important to get every pixel you may not want to use this exact format you want me want to use the next one and then you set graphics graphics import set matrix remember that GI comp we got before we're gonna set the matrix there and that'll set the scaling of anything we draw into that so lastly we're gonna do a graphics import set G world graphics import set the quality for lossless we're gonna get gee world pixmap of course remember to lock that and check the value of your lock on the return because you don't want to draw into a non lockpicks map it's not good enough just to call that lock pixels and then use graphics import draw unlock pixels and close your component now in that pics back that you've created is the image off the disk no problems so one of some other things we want to talk about let's talk about packed pixels how do we get that imaged into OpenGL first real critical if you're using a GL make sure you use a GL no recovery a GL no recovery says I don't want to create a software backup renderer to backup if the system runs out of resources for this what that does in this case it says I don't want I want to respect the hardware's pixel formats only so if you're sending to a hardware all you're going to do is create the pixel format OpenGL is only going to use a pixel format for that one specific piece of hardware it's not gonna try and use a software backup which happens to not support pack pixels so may convert everything to 32-bit so if you have a 16-bit format here you want to make sure you do that to keep it 16-bit you need OpenGL version 1 2 or the GL Apple pack pixel extension 4 to have this to support pack pixels and if you notice one thing you'll notice it actually is the extension string is packed pixel not packed pixels which I think it's listed as and this is just a little caveat so if you search for pack pixel no matter which way it which way we go with that if we if it changes in the future you'll always make sure you get it physical if you search for packed pixels you may not get it in the future or another way around any case that this is the key here is yield text in each 2d with a B GRA extension in the unsigned in 8 8 8 8 reversed and then B GRA unsigned short 1 5 5 reverse the way you read that is the reverse of the pixel format applies to the B GRA so if a RGB 155 or a RGB 888 which we recognize as a standard Mac pixel format thus you can use these to directly texture out of your image buffer so scaling to power of 2 what do we do with this if you notice I have the example images here the first one is like a 256 by 256 compressed down to the exactly what the text is going to look like what we're gonna do with this is we're going to restore the aspect ratio by storing the image took ratio that we read in and the setting the polygons aspect ratio to the same image texturing to fill the polygon which would just stretch it back out again we're losing information here but the image appears correct on our screen so scale the power of to this piece of code here especially pretty simple and maybe easier to look through it on your own but really what I'm doing here is fairly simple what I do is if the image size is I step through I take my Marge's largest power of true that I want I step through and remove that from I'm sorry that's incorrect basically what I'm gonna do is I'm gonna take I'm gonna use the fact that I can shift things down a power of two at a time by using the shift registers and I'll shift it down until there's nothing left when there's nothing left I'll put a one in the first bit and shift it back up and what that does it tells me the largest power of two image that's nearest and then at the end I basically say if it's closer to one than the other I checked above and below so basically I got to the point where where I have a power of two that's that it's gonna be smaller and are larger than the image so if I add a 249 pixel image I shift it all the way down and I shift it back up I got 256 because I have that one bit in one of the one of the I have a one in one bit of that of the image size and then I compare that to that shifted to 128 I see which is closer so I'm getting the nearest one so with this little piece of code does it defines the nearest power of two there's probably different ways to slice it I thought this was probably the easiest way to do it and it's not real easy to explain but I think if you step through it you can see how it's working and that's all I must bring up with the demo machine and I'll show you this in action here what we're gonna do is we're gonna choose a file to open here and this is going to open an image and it's just gonna display it in OpenGL window if you notice is stand a standard window standard image remember this image does not have the complete image quality that we had originally if you read if you look at the text which may be hard to read up there but I'll tell you what it says it's a 645 480 by 32-bit image that obviously isn't a power of two we've put it into a 512 by 512 texture and then we've rescaled it out so it looks like the exactly at the right image if you want to see the actual polygon I can zoom it out for you that's actually a single polygon with a single texture 512 by 512 stretched back to the recurrent aspect ratio for the image one of the nice things you see here is high-performance 2d right that's what we're talking about this is high performance 2d using the OpenGL transforms no problem rotating the image and if you want to zoom in we can do that too very easy we'll go to full screen still get really nice transforms real easy to manipulate images using OpenGL using OpenGL as your 2d transform but you say you know really I want all that information that uh we can go back to the presentation now I want all the information that I had in the original image I want to have all that I don't want to lose information oh let me back up if you notice feel carefully at the screensaver naka was 10 OS 10 uses the same technique it's it translates all the images to 512 by 1024 textures and uses that but you said man that screensaver looks great great detail you're losing detail that images is 77-68 tall it's square it's squished down the 524 if it's 900 wide it's stretched out to 1024 so it uses that to get consistent images and have an easy texture loading so that's an actual use of this in a 2d environment and you see then they can get the fades using the OpenGL hardware acceleration instead of having to do some kind of fade themselves which is going to be a real hard and real real taxing for the system so the other option we have is segmenting to power of 2 you don't want to lose that and this is this is a an interesting technique a little bit more complicated but it uses the same basic basis as the other ones in this case what we have is you can see the image then the green lines indicate the actual polygons that produc comprise that image I've segmented this image to about 4 by 4 as far as using it to display and so I've kept all the pixels directly in one texture and then I'm then I'm sliced out of that texture for each image the way I do that is when you call Jill text image 2d you provide it with a base address that base address can be manipulated it doesn't have to be the beginning of your buffer so what you can do is provide it with the beginning of every single segment of that buffer very simply the other thing you can provide is OpenGL has a constant called unpacked Road length and normally the width you pass into GL text image to D is the width that it uses obviously if you were taking a piece out of this that weight wouldn't make any sense and because you would get to be all skewed it would try and take the next pixel over and stick it on the end of your image so you get this kind of skewed image that came across your screen well set unpack Road lengths to actually the width of the entire image vise so with that little piece of texture we want so what it'll do is for every row it'll skip that much that's like it's image stride it'll skip that much and come down to the next one grab a piece of it skips the entire rest of the row and grab the next piece so it'll have that nice lined up pieces of the polygons to win an Ashley tries to draw it so we use the image width in the unpack row length and well then the texture width will be some smaller subset of that ok Sigma into power of 2 what we're gonna do is this first first black box statement up here basically tells me how many textures I have in the image dimension same technique we kind of used before basically what I do is I say okay I know that if I look at the the bitwise representation of that dimension I can just kind of count the bits that are actually turned on so if I look at the ones and zeros I can go how many ones are there and that's the number of textures I actually need to use it's real simple works very well when you realize that that that's a nice little thing to have I then do a new pointer clear for texture names and then that's text your ex by text or Y so I'm gonna create an array that I can store my texture names in I'm gonna use the pixel store I as we talked about the set for the image width for the entire texture within this case the entire texture width and then I'm gonna do a Gen textures to generate texture names for each texture I need for that image so in that case you saw it was a four by four I'm going to generate 16 texture names then when I draw the image I'll use GL blind textures I'll bind each texture in turn and I'll draw the image that way so let's look at actually getting that getting it into memory another little black box function we have here I'll show you in a few minutes is get to exercise it takes where you are currently in the image and says what's the next size what it does is I say I have a max text as well as talking about earlier like I confused I have a maxed exercise and want to use 256 512 maybe I don't want to go up to 2,000 for an image mean I want to keep it down to 128 so I have 128 segments whatever you prefer it steps through 128 removing that from the length of the image and then once it gets past that once there's less than 128 left it takes the next biggest power of 2 off the image and it keeps doing that kind of down so it gives you 128 64 okay there's no 32 so I'll give you a 16 and then whatever else you got left in this case we get the next exercise in the loop we do go for x and y for each of our textures we off to our offset is offset is zero for the y offset that means we're we're in going in two columns we're going to start the top of the image forever every row we're also going to above this we're gonna have the X offset is zero at the very beginning and what we're gonna do is we're gonna take the the pointer to the buffer is the normal image buffer we got before from quick-draw when we when we use the the 2d get your new jeer wheel from pointer we're gonna use the offset Y by the texture width times the image depth divided by eight in this case it's shifted three so what basically we're doing is saying that this is going to offset into the image for that corner of every texture so we offset across four X offset down four Y for every row we offset in based on what the the width of each texture we need to use is get the next exercise for the height we've got it for the width so we put in the previous exercise we call bind texture we do that bind texture we pick the next texture name in our list of 16 and then we basically do the same in the same GL text image 2d if you notice I'm using the P buffer and I'll go back a slide the P buffer is that P buffer not the P image buffer the pH buffer was the original pointer and the P buffer is the pointer to whatever specific texture and we've bound we've told it the the unsigned row length unpack row length is the width of the whole image and so we use the GL text image 2d at that point to grab the right texture out of the image so we're kind of doing the cookie cutter take a piece take a piece take a piece move down take some more pieces we got we slice our image up that way this is a little snippet of code from the git texture numb from texture dimension same kind of thing first thing I'm going to do in this is I'm going to remove what are my max size of textures I have max text size I'm gonna divide by max text and count how many max text or images fit in there then I'm gonna take whatever is left over the texture dim reset there and I'm gonna is there anything of 8,000 is there anything of 4,000 is there anything of 2,000 is there anything of 1,000 I just go down each one if you notice when you add all that up it'll hang up back to back to the complete size of the image so this tells me how many textures across I have or down and lastly here get the good text and get next exercise this says hey I got the size of my current one is it one of the first max sized ones or am I gonna step down into my list and basically do the exact same thing checking to see if I have an eight thousand four thousand a two thousand and 1000 like that so how do we display these things first we're going to use an orthographic display set up to the scale of the image the scale of the screen so the images you saw I scale your display to be the pixel scale of the window so when you're manipulating them if I offset three pixels I'm often sending three pixels in the window so this actually is a good way to display it so you're thinking like a quick like quick drawl or normal what you used to thinking as far as you invert the X and you invert the y coordinate your X is your origin is in the upper left corner and at that point you're moving things in a pixel centric manner I may ignore Z I don't need Z so I don't need to establish a depth buffer and I ignore the Z for this one I use poly I'll use a polygon to scale the image when you saw me scaling and out that image before I use the polygon to scale I use manipulate the polygon positions I didn't use any in texture coordinates I didn't scale my my my window at all I did was scale my polygon size my texture stayed the same in OpenGL takes care of the hardware accelerated transform to get onto the polygon and you can use the modelview matrix very simply in the in this case to offset and rotate so as you saw me dragged I used to a drag on the mouse I use a real real nice carbon events told me when the mouse drags I figure out what the star point in the end point for the mouse drag is and I just drag the image but that many pixels my offset since already in pixel space is exactly that when I apply it into the modelview matrix rotation I get the rotation in degrees for the for the turn of the mouse and I apply that again to the rotation matrix of the around the z-axis for the modelview matrix to manipulate the rotation of it and now we're going to go back to the demo and one thing about the the multi-textured demo here is that I can load the same image I did before if you look notice this one it's it styled in multiple textures let's let's expand it out and I'll zoom in a little bit to actually see what it looks like what you've seen there is actual pixel grid of the actual pixels on the screen there are actual pixels of the image and you see how the text is actually lie right on the pixel grid so you see that these come together right on the pixel grid of the image and OpenGL does a great job of stretching that image I mean there's only a certain amount of data there but it was a great job of scaling the image and you still got great performance on this 640 by 480 image but you say you know you know maybe I want something bigger so I had to dig around to find a bigger image but I did find a bigger image and I haven't tried this one for a while so I'll hope that hopefully this one will be okay this one takes a while to load and kind of how big this image is I think it's 2800 by 2100 so it's a pretty big image look at the polygons will zoom out a little bit those are 5 the max texture size 9 by 6 I think the max texture size is 512 in this case so 2800 by 2100 pretty nice oh you want to rotate it we can do the rotation too that's easy to do you want to zoom in to see the actual imprint nice detail there for the image so you can see opengl works very well for handling 2d images one thing it might be interesting is see if I can find one of these well we'll help ourselves people have been working with opengl sail but there's a little problem with my technique and there actually is they find the the corner I'm gonna fail show it'll show up right here probably Oh you kind of can see it here not very well I salute you better than I thought there is a discontinuity along that line and there's probably better examples of it let me see you actually might find a better one because it's kind of a good thing the good thing to understand probably on the edge here we'll be able to see it better back to the middle you say there's it I can see it a bit down here okay you ought to think oh there it is actually you can see right there see across there you can see right across here there's a discontinuity discontinuity is OpenGL uses filtering OpenGL filters across the texture to when it when it tries to man you mine five minute five bytes by a texture by a texture by texture I don't know every best way but it it looks at each texture separately remember you've moved this image from image space that you had to the textures and it looks at each texture separately wants to filter it when it when it tries to do the scaling for form innovation and magnification what that means is it looks at the next pixel over and says what is the color of the next pixel well in this case the it says there's no next pixel at the edge of the texture so there's a filtering discontinuity between two edges so if you really want the perfect representation of this something I didn't do in this demo what you can do is take a texture border you can take a border of one row of pixels from the formula from your image so you make your textures like one pixel smaller and take a border around them so that minification magnification filtering is more accurate not something I did here but you can do it if you if you want the perfect example of this so again this is a this is scaled up like two-and-a-half times and you can see that it does a pretty good job on this stuff so go back to the slides okay let's move on talking about sprites sprites are actually pretty simple sprites are just gonna be textures and handle by OpenGL so we'll load the sprite frames and Stormin textures well what I'm going to do for sprites that have multiple multiple images and most people who want to use sprites want to handle multiple images in one sprite even if you want to animated cursor for example how would you do that what you can simply do is have one texture that fits all of your sprites are some of your sprites in there and you can set it up so that your sprite frames are offsetting you use the texture coordinates to offset into the sprite frame so you load the whole thing in one texture and instead of like we did before where you're loading a segment that one image into a segmented texture you're loading one texture and then using it as a segment as it's in segments by using texture coordinates every time you go to a polygon for example if you had a polygon if you had ten sprites you would use a tenth of the texture coordinates for that one sprite and you can move maneuver across depending on what sprite frame you're talking about one thing you can also most will want to do with sprites is they want to use some kind of masking I want to have just that cursor just my animated cool animated cursor spinning around or whatever they want to use that some kind of masking so how do you do that well you can easily do it by passing in an alpha Channel or if you read an image from QuickTime you may not be sure of the alpha chain you may not be sure that's the exact alpha channel you want so you want to go in there blast the Alpha bits of something you know in this case I used a background color of pure green and I looked for it basically the green screen kind of thing and I just remove the green screen by setting alpha that's a zero I did it a couple times and the first time I did it I came out with what's on on the year on your right hand side here with that little green line around that it was pretty cool effect but again texture filtering that is it looks at the next pixel and says huh I know this is green there you want me to filter from that green onto your image and that's what you're gonna give you so what you can do is the other image what I did was I said I'll put a kind of a neutral gray in there and the neutral gray allowed me to filter it to a reasonable color so that's something just keep in mind when you're doing it something the OpenGL will want to filter that and it will use both filtering on alpha and in filtering in the image domain so you want to make sure your filtering set up correctly you're using an alpha mask so let's talk about how to do alpha mask basically you want to set the alpha alpha Channel onload you want to look at the Alpha Channel once you load it with QuickTime and you want to blast the exact bits you want into it in this case we're talking about the top eight bits in 32-bit format or the top 1 minute a 16-bit format if you wanted to you can use other formats by the way I'm not limiting you to these they're just something that I'm using for the talk but you could use a four four four format OpenGL would understand that which would give me the four bits of alpha in a 16-bit representation of colors basically background color pictures with pixels we're going to set the Alpha to zero and we're gonna set the pixel value of those of those pixels to some neutral back background color for all other pixels we're gonna set that make sure the pixel is set to alpha so in this case in the 16-bit case well or that with 8,000 hex and in the 32-bit case we'll set the two high bits to FF to basically make sure they're all ones in the two high bits you want to ensure the alpha set because sometimes depending on where the image comes from and I've seen images that I thought out of Photoshop that I had set the Alpha correctly and it comes to a format and when I look at it coming through into the machine the Alpha is not set like I would expect it to so just make sure you you keep that in mind that you have a known source of your alpha or your setting into what you want to set them to so how do we do this it's actually pretty simple if it's 32-bit image what we're gonna do is we're gonna get the pixel from them from the image buffer we saw before we're in a group we're gonna step through for texture with and texture I through the whole thing we're all going to look at the the bits that are our color masks in this case I'm saying green FF and 32-bit format is our color mask so I'm what I'm doing is I want to ignore the Alpha in these bits I don't want to compare to 0 0 0 0 FF I don't want to compare to all 32 bits I only want to compare to the 24 bits of color so I'm not sure about that alpha so I may have a difference in alpha and they won't quite match my own mic my source color key so I'm going to look at the color bits so I'm gonna mash those color bits out and if they equal that pure green I'm gonna set the pixel to a reasonable gray in this case with a 0 0 and the high bits for alpha so it'll be blank and the filtering war will look look good if not I'm gonna or that with FF and the reason I want to or it with that bath is to make sure that those two alphabets are set at the top or eight alphabets actually there's two - and hex two digits are set for four alpha and I'm going to loop through the next pixel four sig the 16-bit case very similar in this case I'm going to compare it to the seven ffff to mask out that one alphabet and make sure that three zero which is the same five bits of set for the green I set it to a neutral gray or set the name to the top alphabet so let's talk about alpha blending there's another way you say now you have the sprite mask you have your cursor that you move around you've cut out the background in we're talking about alpha blending another thing you may want to have some kind of transparent text or some kind of feature some kind of you know you have the heads-up display in a game you want to just use that with alpha mask real simple to set with alpha mask in this case what I'm going to do is I'm going to use a luminance value of this texture to set an alpha mask that's what this is an example of here what I'm going to do is I'm going to take the average which is actually I'll correct myself before anyone else does this is not actually the luminance value this is this is my poor man's luminance value if you want really did you the y UV conversion and just take the Y component you should do a correct y UV conversion but in this case I'm going to assume this is grayscale all the bits of the same and so that really the divided by three is not necessary but it looked good for the presentation so we take our poor man's a poor man's luminance value we we divide by three to get the average for the 255 we're going to shift it up into the high bits and we're gonna or it into the pixel so in this case what we're gonna say is whatever the the kind of grayscale value of that pixel is is gonna be the Alpha so we'll get transparency that varies with the luminance this is good if you if you get a some artist it has some kind of background or something like that you just want the luminance value we want to set an alpha that equals it you can use this kind of technique so how do you draw with these things what are the key OpenGL the unlock the OpenGL Kingdom here we get the jail alpha func what I do here this is the masking which talked about that alpha we talked about the mouth mask or the first thing we talked about in this case we're gonna say greater than 0.5 or less than 0.5 so if it's greater than 0.5 we're gonna put draw if it's less than 0.5 or not and so in this case when we set everything to 0 we won't draw anything was set the one we will draw and we said we our polygon needs alpha test or if our our sprite needs alpha test will turn off with test on for blending for transparency we use a blend func in this case we'll use source alpha + 1 - source alpha so but basically this say the contribution of the front pixel is source alpha it's a sources alpha value the contribution of whatever is there already is 1 minus that so if your alpha is 0.8 you can see the source is 80% contributor the background is about 20% contributor and we enable blending if we have transparency lastly depth test and then when I do a depth test so if you have multiple multiple pixels that are or multiple sprites that are over top of each other you can actually use Z even in an orthographic context as a delimiter of what stress were drawn in front of what else otherwise we're going to do doing the painters algorithm what's ever drawn in the front last is gonna be in the front so you may want to use the blend function for that so let's talk a little bit about how to do a side-scroller the most people when we when we think of side scrollers who think of like you know 1400 by 480 kind of backgrounds that if you play defender on or run around and your your 2d version of quake or whatever you're doing and most people think of using a 2d raster engine to move these things around to the screen because we can we can compact it down we have a lot of black spaces we can use the parallax scrolling I'm saying you can use OpenGL to get a much faster version of this with a lot less imitations on you what we're gonna do is we we use a tiled images to form a large textured background pretty unlimited you can even load the tiles you know I mean if you think of something like Diablo 2 it actually was a similar method where it loaded terrain loaded sprites as it needed to it was a forced perspective kind of two and a half D thing but a reality it was still kind of the same idea of using an accelerated engine to drive something that could have been done years ago or would be to attempt to be done years ago with a 2d engine so we used a grid of polygons for the background we're gonna and there's a lot of different ways to draw all the grid of polygons we toyed around with him in talking about it and there's some different kind of techniques you can use as far as deciding what's on the screen and off the screen depending on your limitation so I'm not gonna try and tell you exactly how to do that because I found out that depending on what your limitations are of your application that you can find some ways are really easy and some ways they're very difficult the idea here though is that you keep a grid of polygons you're gonna draw something slightly bigger than the screen or make sure you draw the polygons that are actually present on the screen and you're gonna draw the ones that you need to as the person moves whether whether it's north south east or west when they move that way you're gonna draw some additional polygons basically when you draw them gridded polygons you're gonna find out what your world center is in this case we just attach it to a sprite and we pick that sprites world center and we offset everything else by that world center so everything else just moves off the screen out as it would I mean so you don't have to worry about some kind of strange transfer when you're writing yourself just use OpenGL modelview matrix and transform it as you would with with the world center attached to attach to one of your sprites we're gonna rotate it by world rotation so if your sprite rotates you can rotate the entire world or you can rotate your sprite whichever way you'd like to do it so there's no limitation that everything has to be kind of orthographically displayed across your screen which i mean if you're thinking about it from a 2-d standpoint I'm not thinking that it's gonna be really easy to take a 1400 by 480 image and turn it slightly or OpenGL can do that for you as we've seen and your scale by world scale most of time you can zoom in and zoom out this case you can zoom in and zoom out or you can draw your little radar map up in the corner of your game or application or showing the how you're manipulating a large image by just showing it a scaled version using MIT maps of that larger texture you've already drawn so how do we handle textures you load the background as one texture or a set of set of textures but basically you load a large image as a single large texture you're going to segment it with the techniques we've shown before and you're let OpenGL handle the text and do the texture healing for you use generate a gen textures to generate texture list as we seen before and will let OpenGL handle the memory memory management in and out of VRAM or in a manner to a DP space because the OpenGL implementation on ten is a very good job of handling a gpn it's gonna get better in the future and that is a really should take advantage of that fact instead of trying to have your own what's in what's in memory what's not in memory let's load something let's remove something just let it handle it for you it'll push things out you're not using it'll bring things in the that you need and I think you'll find it it'll be very effective in managing those textures for you and we this is again the same as your segmented images this is exactly the same technique we've talked about before so how do you draw how you draw this in my case I set up an orthographic context I thought about it later I thought they were really cool that you could use a perspective context actually for your side scroller and when you zoomed in and zoomed out it actually would have that kind of parallax weird zooming but I didn't have time to do that but that's something to think about so you mean if you really don't really have to set up an orthographic context you can say you think about it as your artwork bench where you're looking straight down on one of those you know camera mounted above a table you actually can can change or use the geo you look at and you can change the field of view in the aspect you want to zoom in you just change your field of view to be wider you want to zoom out you have changes to be narrower you change your your-your-your near and far planes and that I mean that kind of thing you can do and gives you a lot of the features that you really would work hard for doing it yourself in with your own 2d engine you can give it to you with with hardware acceleration in OpenGL in this case we're gonna pin the world to a sprite of interest here's another another good thing you can do let's say you're you have some kind of a strategy game or some some something that you have multiple places of interest you have a very large document you're handling you can have the the user can jump to an area of interest very easily by just manipulating the center of that image by it'll do a lookup of your stripe sprite information or do a lookup information into your document jump to this object on my document I want to see where how it's positioned in my preview and they can do that basically by you to read out where that position is and offset to that position without having to do some kind of calculation off I mean to move a lot of pixels around yourself and what I did was to draw the sprites I basically do it you kind of grossly call the sprites to it to your to your visible list you don't want to send OpenGL every single sprite that's off the screen in your your massive area but you just grossly call the ones you think are pretty or close to visible not OpenGL do the fine work you may spend you may find if you spend way too much time trying to replicate the transformations trying to figure out exactly what's on the screen and off the screen you're spending a lot of time doing that one one cat one thing i'll i'll tout here is the open g optimization session tomorrow with John Stauffer will look very closely at what could open GL can do for you what I can't do as far as optimizations so come to that session look at his example of optimization apply it to your same application make sure you know in real time what percentage of time you're in your application and what percentage of time you're in OpenGL null out those drawing routines and find out if you run your application open-loop what actually is usually if possible frame rate a lot of times you'll find that you're not really spending much time in OpenGL you're spending a lot of time of your application it doesn't matter how fast the graphics engine is you're gonna have to improve your application speed and lastly we use the depth buffer for layering I think we've seen that before so we're gonna do a little demo here I'll open my spread file it's a couple sprites here I think the background image is 2,000 by 1600 I have another texture in there there's a thousand by thousand both 32-bit and then I have my sprite images which are also there's actually 256 by 256 and I just use a scaling of OpenGL to do it so we're going to run around and see what this thing can do so as you can see I can move move the rocket back and forth I can fly around very easily and we can zoom in or we can zoom out real nice that you get that you get a nice feedback and if I can do this right I probably can find Apple on here this is that months to be a photograph of Cupertino let me see you and do this where am i oh I think we're up there that's no can't find out but I'll find it later in any case you can see what I did here for the poor man's version of tiling I flipped the image and I guess use OpenGL to do the same image you can zoom in you can see that and we got the cloud layer there so we have a nice layer of clouds that's good flying over it one thing you can do is this is your standard side scroller you kind of fly around at scroll side-to-side one thing you can do also is you can do this very easily in OpenGL which works the same it's doing the same amount of work so I can fly around in this case and my standard method of you know up up you know the top is up and have a pretty pretty convincing game and make you all sick at the same time so we'll we'll go back to the presentation now and the last thing we're to talk about it's QuickTime integration how do you integrate QuickTime and I have about 10 minutes I think I can get through this a lot of this we've talked about already the same techniques this whole thing is using the same technique it's all based on the same stuff it's not it's not rocket science it's not separate stuff you learn some of this stuff once and it apply to all these techniques QT new G welcome pointer we've seen that at least three times today that is how you can allocate the texture to get a packed texture format without having any any extra space to do help to not allow your texture we're gonna use pack pixels for direct texture handling we're gonna synchronize to QuickTime to minimize our update what that means is we're gonna sink we're going to give a callback procedure to QuickTime to tell us when it updated the frame if we haven't updated the frame we're not going to update the texture what this means is the only textures uploaded to the card are the ones that actually got updated you can see obviously if you upload the frame every single time it's like playing an uncompressed movie you can see how that can compare to a compressed movie where you're only really updating the frame every time it changes if you have a frame if you have a OpenGL that can do 100 frames per second in QuickTime something in the movie at 24 frames per second there's no reason to update that texture let the texture sit in free Ram and then only update it when you need to drawling we're going to use sub in the subtext image to that's wrong may be wrong in any case we're going to use a do a sub texture upload so roll in fits the movies usually are not 256 by 256 or 512 by 512 they're usually a four by three which is usually not the texture size so we're gonna take the sub texture out of that and we're going to only update the part that's actually updated again saving bandwidth to the card so what we're gonna do here new pointer clear we've seen that QT New Year award from pointer we've seen that set gee world set movie gee world we haven't seen that so all you're gonna do here is set the movie gee world same as we did we think with the graphics compressor and that'll tell the movie to play into that G world texture formats pretty simple gilt xmh 2d RGB that that would be an uncompressed format and then they have the reversed format showing showing the actual pack pixel format quick time synchronization here's something new you look up there you see draw movie draw upp that's gonna give you a universal procedure pointer to a movie to the movie completion procedure would be new movie completion procedure so you're actually creating it with my movie movie draw proc and then you're set the procedure very simply for the movie G movie is actually the pointer to your movie so that's something you get from QuickTime and then in your completion proc oh I'm sorry if you look at the call it says movie drawing call wind changed so it's going to call you every time it updates that movie into the G world so QuickTime will tell you when it has something changed this is the only code you have to do to like QuickTime know when you've actually updated it you can see this could actually be very useful and some other things that use QuickTime so you know when there's actually something changed in the movie you're not trying to update something all the time and then a movie draw proc really simply all I'm gonna do is in the refcon I have a global variable GF movie text update and I'm gonna set that to true every time I get to that procedure so when I go down to draw my polygon I'll get alright all ready to draw and I'll say do I need to update the texture if I do I'll react the texture from the movie G world if not I'll just draw the polygon with the texture you can use the the idea of current textures because you can bind it to any texture you want you can have multiple movies plane you can also just in this case among the example you'll see I just have one movie one texture so I just leave it leave a current and how do we draw GL tech sub image 2d I was right it was wrong it's not subtext images text sub image 2d geotextile image 2d you'll see here and this basically is the exact same thing you've seen before as far as using the same kind of cord same kind of call can you use the 32-bit version or the 16-bit version then I'm going to move the frame what that does is that's just my call to move the frame around the screen however I'm manipulating that quick time frame that's the only the frame of the polygon I'm using the modelview matrix to do this I'm not doing this this is not manipulating the movie pixels itself just the frame and then I'm gonna do it GL begin on coop with quads and then you set the texture coordinates for the corners I'm gonna set the color to white for the whole for the whole thing you can use cut you could colorize your movie if you wanted to by changing the polygon color and then I'm going to very simple verdant vertices use the user basically a square vertices and the movies going to be inside that square so let's go back to this demo okay I'm gonna show you a couple different things here first in this case we're gonna bring our display size at 640 by 480 this demo does not use a segmented texture but you could easily integrate that into this demo this uses a single texture that the movie is compressed into 256 by 256 movie I'll use a 16-bit off screen full screen I will use pack pixels and I will sink to the VBL which we'll see how that works not on this actually let me turn off sink to vbl water then we'll turn it back on for the next run I'll pick a movie we've seen this before and mr. Schafer would you prefer window or aisle so middle 620 friends per second playing that movie through up with GL let's play in the movie fullscreen and texturing as fast as I can a single frame some real real simple there hi now we'll stay with this demo and we will in this case well you would just throw some effects in when use fog and we'll sink the VBL and we just leave it like that and we'll play it in a window this time and mr. Schaefer would you prefer window or aisle middle about 70 frames per second I'm not sure about the sink on this because it seems to be very more than I normally see because of the the video simple fog turned on linear fog for this real simple to show an effect with the movie you get the full complement of opengl cool stuff you can do with it without having to do any extra works it's a little movie I'm working let's go back to the presentation so where do you get resources for this it's pretty simple the red book and blue book if you're working with OpenGL make sure you got the red book in the blue book the other thing that I think that you should everyone should have is in our OpenGL SDK you'll see the spec look at the spec I use that more than I use the red book and blue book if you notice if you look at the spec the red book and blue book look very much like the spec but it has a lot of it goes into more math and many of us want to know sometimes but it actually has the real what's behind the engine how does it actually work which is real important if you want if you're trying to debug something like fog you want to know exactly how the fog is gonna work or you're trying to look at how the texture that the unpacked Road length you want a good explanation we'll unpack roll length it goes through exactly how it handles pixels so I think the OpenGL one-to-one spec is a very good thing to have and all programmers working with OpenGL should have it it's in our SDK it's in a PDF format you can print it out and or look at it online good places to know if you're working with OpenGL wwl opengl org a lot of resources tells you where the what the current standard of OpenGL is he has a complete list of the extensions for OpenGL gives you a lot of programming examples in starter code lists on Apple comm search for OpenGL we have an OpenGL Mac list I want to see you all on that list it's a great place to get in for information ask questions and we monitor the list and try and help out when we can and lastly developer.apple.com slash OpenGL so that's another good place to go all the developer.apple.com urls go to the opengl page it's a pointer to the sdk a pointer to the newest things we got for it so what did we talk about today we talked about using OpenGL mainly for 2d and two and a half T and we did some movie stuff too but how OpenGL can be used to complications that aren't interested in doing next the next quake or the next Tomb Raider or the next nanus or or a kromagg rally we're talking about games they want to use just to display images want to use the power of OpenGL to manipulate those images and it's a really great thing that you can do it easily you can set up your transforms to let the transforms handle the workload for you and this led OpenGL do the hard work and we will continue to improve OpenGL will continue to work with with the hardware hardware vendors hit the best hardware acceleration we can and that will give you they'll continue to improve your app even without you doing work on it next it does allow robust hardware acceleration you're not going to get that any other way you slice it you're not gonna be able get to the hardware acceleration unless you're gonna use OpenGL or use one of our provided API use don't have the resources and and the ability to get at some of the hardware acceleration unless you're writing the drivers unless you're writing writing the the OpenGL implementation and it really shows you a new paradigm to solve these real world problems you don't have to say well hey I use copy bits I'm gonna rotate that and I'm gonna gonna use that and that's gonna be really difficult maybe I'll go and Swizzle now so OpenGL rotate your modelview matrix throw it on the screen scale your modelview matrix throw it on the screen you can shift it over move it around it's real easy it's just a couple instructions sample code the samples will be up very shortly the movie samples up right now the image sample should be up by the end of the week and the the rocket samples should be up probably next week let's talk about the road map Sergio mekin mention there this is the first of five sessions the sessions you want that you want to go to all right after this is geometry modeling after that tomorrow morning open G optimization John Stauffer the head of our OpenGL team it's gonna be in here and soon to take you through optimizing an OpenGL app it's gonna be a really great session he has some really great tools to show you how to do how to optimize that app and I'll show you how to get the most performance out of an OpenGL application advanced rendering Troy Dawson one of our engineers in the opengl team is gonna come come and show some advanced rendering techniques and you'll definitely want to see that if she talks to talks about talks about Stenson's using the stencil buffer to do shadows talks about anisotropic filtering and some other topics that'll be really good talk and we have the feedback form for open OpenGL as the end of the day tomorrow please come talk to us about what you want to see an OpenGL where we're doing well where we're not doing well we want to get your feedback we want to take your your stuff in and we want to make sure that we improve it and make it the most world-class implementation of OpenGL and 3d graphics API anywhere and lastly there's a feedback form for QuickTime for quick the QuickTime API which I think you've seen is a very official to your work and OpenGL for texture handling and for movie playback this feedback for him for a quick time also Thursday at 3:30 Sergio Mello is the technology manager for 3d technologies at Apple it's a great contact if you have questions about implementations questions about where we're going questions about about what we have today for to solve your problems please contact him Sergio to Apple comm with those kind of questions you