---
title: WWDC2003 Session 208
framework: wwdc
role: article
path: wwdc/wwdc2003-208
---

# WWDC2003 Session 208

## Transcript

Kind: captions Language: en good morning everyone my name is it's brown and the graphics and imaging evangelist and I want to welcome you to session 208 fragment fragment programming with OpenGL and now this session is our second session and programmability in terms of some of our focus on exposing the capabilities of GPU to do interesting things like vertex operations and in this case doing per pixel operations very advanced per pixel operations at incredible speeds a lot of you saw some demonstrations using fragment programs and earlier this week in the graphics imaging overview and what we're gonna do in this session is really sort of drill down and focus on fragment programming and we're really looking forward to seeing what you the developers going to be able to do with this incredibly new technology so it's my pleasure invite james macomb to the stage to take you to the presentation [Applause] morning everyone so thanks Travis so today yeah I'm going to introduce you to this pretty new technology that we have and we have a really great implementation of it at Apple combined with some I think really pretty cool tools that should make this easier to do than it would be perhaps on some of the other platforms before I start I just want to point out because a lot of people have noticed this if you think my spelling is wrong I don't think it is it's just that i'm using the UK sewing for a lot of this stuff and uh i was criticized a lot of with this so i'm going to try and just warn you now anyway so first thing why would you want to do fragment programming well has so many applications which is one of those exciting things about it not only can it be used for 3d lighting calculations and the like but you can also use it to do sort of even like think of the Photoshop filters well you can implement 2d effect even color correction style things using these programs you can do 2d displacements all sorts of interesting things things that you maybe wouldn't even think of such as using the GPU for just general purpose computation even thinking of as a second processor on your computer the other great thing about it is you can load this program up in your card and you can open up your cpu monitor on the surprise surprise you'll notice it's taking only no cpu time that's pretty compelling reason to offload some of your intensive per pixel calculations after the GPU and then the my third point Apple I think this definitely provides the best tools in the industry for sheeter but for building shaders specifically last year I was here and showed you a vertex programming showed you how to use shader builder to do that well shooter builder has had a lot of work done to it and not have support for new languages and a lot of new features which I'll cover later on in my presentation so to run through and seven points here what you're going to learn where does fragment programming fit in the OpenGL pipeline will also take a look at you know does your current graphics hardware your have you have allow you to do this if so great if not what do you need to buy next the next thing we'll look at is a ARB fragment program this is the language that the architecture review board approved for fragment programming the great language I'm going to talk more about it later will if you don't have the absolute cutting edge and you you weren't willing to do that we'll look at a what sort of a condolence prizes we have for you and then we'll again we'll run over the shader builder and we will then I move on open up shader builder and show you a basic fragment program and we'll work our way forward without to something to show you just how the language works and move forward then I'm going to show you something kind of interesting with fragging program that is using the GPU for general-purpose computation and then I'll move on and show you how to get multipass working in the most optimal way with our OpenGL implementation at Apple then we'll look at how optimization applies to fragment programming as programmers we optimize to where we shoot off to my stuff often we'll see how that applies to this slightly different way of thinking about software and then finally I will move on to something a little more math intensive and look at implementing per pixel lighting in a fragment program I'm going to pull that apart and hopefully remove a lot of the mystery that may surround that so starting off of my first point where does it fit in the OpenGL pipeline well this has been covered pretty extensively in some of the prior session so I'm on you most of you I think will be familiar with the traditional OpenGL pipeline so I'm going to try and move quickly here on this slide you have your vertex data and you have your pixel data vertex data can enter through immediate mode calls or vertex to raise pixel data usually going through the text image calls or those could come from displaylist that you have already prepared at your program initialization pixel data gets through all the pixel store stuff gets unpacked and makes its way towards the graphics card now I've vertex data traditionally it is a very fixed function that's done to the vertex the vertices that you submit the they're simply x the model view on the projection matrix inside of OpenGL then they're clipped into window into the window coordinate and then OpenGL will make use of the GL lights that you've enabled it will calculate the vertex colors based on on a fairly primitive lighting model on a per vertex basis so the colors are interpolated across the polygon surfaces which we all know those look so good after that stuff calculated it the rasterizer goes along on the colors in the frame buffer basically excels and then this last part that the blue dotted line that's just come up that indicates that this is the process of feedback and I was taking your frame buffer and eventually bringing it back through as it can be fed into a texture units again for multipath operation so I just pulled out the fixed function transform clipping lighting and I'm also going to pull out the fragment operation the fixed function ones and then vertex program which has been around for longer it replaces that part with a programmable language today I'm going to talk about replacing this part with our pregnant program so support it hardware this slide I'm going to amend slightly at this point the scene for our fragment programs the ati radeon 9700 but the good news now is that all of our new high-end systems the new g5 boxes all of them will support art fragment programs so if you've been thinking about buying one of those maybe this is another reason you might want to look into that come and then the ATI text fragment shader this is a slightly significantly less capable language but something that you might want to look out and I will give you a little bit of information on it later that's supported on some of the lower end card and for instance our current 15 inch powerbook the latest ones do have a graphics card that's capable of vertex programming and support for this ad I vendor-specific extension so what exactly is a fragment program just out of curiosity how many people here have actually been writing fragment programs in the last while how many people have done it before okay that's good great so what exactly is it basically on the guy it's round this little program that you upload to the graphics card it's a it's round one for every color look up during the rasterization basically whenever the rasterizer is going across and filling in the pixels on the surface of the polygon normally it goes through a fixed function which can you know get pics attack souls from there's a different text you're going to unplanned them together well this allows that process to be totally programmable so yes you could go and get the data from the texture units or you could just return any color you want based on a mathematical function nice so again the output is a fragment program is good it's got a very well to find out put single RGB a color and optionally you can specify its position in the depth buffer so was it can be cold by later stages in the Geo pipeline regards to inputs to the fragment program the texture coordinate channels which you would normally set up in the vertex program you'll note that when you write an output vertex you can put into the texture coordinate channels and you have the number of channels is equal to the number of texture units on your graphics card the values you right in there on the vertex program those are interpolated across the surface that the fragment program is running on so you get that interpolation for free so that's one of the inputs you also get the interpolated vertex color and the position of that fragment in the window and went to coordinate also with our fragment program you can access all of opengl state which is certainly very convenient that means you get access to the matrices and also the program parameters as well which i'll discuss and then the other important input is you can you can sample texels from any of the texture units which is important also let's take a look at a specific superb fragment program great great news here if you've programmed our vertex program the language is is totally parallel to that it was intentionally designed that way too if we know our vertex program you know ARB fragment program as well the only st. difference is that instead of dealing with X X Y Z W components to specify a vertex position in 3d space you're now I defining red green blue and alpha components for a colorful and output color so it's same thing applies all these instructions there are a few exceptions ninety percent of these instructions are their vector instructions meaning if you do an ad it's obviously it does the ad for all four components there are some of these instructions talk about later which are considered scalar instructions and that is for instance the to the power instruction to the power of that's a scalar instruction meaning it only deals with one component so if you want to do a power for all four components you need for instructions that's one thing so group these together those are all your ass arithmetic instructions you've got some logic instructions like other languages unlike unlike programming a general-purpose CPU don't be expecting to be able to do bit shifts and things like that that isn't going to happen do you want people do shift left shift rights any of that but you do have some basic logic to determine if something is equal to zero or not or its greater less than zero the other thing then is these are the important ones that are pregnant program ads though the text your sampling instructions these allow you to get things from the texture units with text being the most common one and then use some miscellaneous instructions like vertex program you have a squizzle instruction so you can reorder your components in a vector and you have this kill instruction I haven't actually used it yet in the fragment program but what actually is supposed to do is stop a fragment from continuing down the OpenGL pipeline it actually literally just kills it off so it might be some possibilities for you without construction so inputs fragment program attributes I've already actually covered this one bit of duplication here but still needs to be in the slide color texture coordinates all for the foghorn I guess then again the OpenGL state matrices light materials these are these are all accessible in the fragment program then you've got these convenient little program parameters which mean that you can make a fragment program that is parametric for instance if your fragment program is is all is implementing a lighting model the position of the light in 3d space you want that to be sort of a parameter of the fragment program you can define a variable in your fragment program and then there is a new OpenGL entry point which you call and you can hand in a four components or for floating point values and that's just those four values get uploaded to the card and on when you when you flush your next seeing those get applied and then finally your output result color and depth advantages of our fragment program portability this is the standard now so if you're programming in this I think you'll find your urine you're in good shape because a lot of vendors are going to pick this up and our high-end machines support it and I think it's a good it's a good bet to go with on the arbor proved it which always helps and also the instruction set it's it's very rich in comparison to some others i'll show you which allows you better better be able to implement better better models for lighting or whatever and also you don't need to generate so many sort of frustrating look-up tables for instructions because they're just natively supported on the hardware and also flexible text your sampling there is a certain only hardware design challenges to certain aspects of this which i'll discuss maybe a little later but you can do with ARB fragment program it you can do many many samples you could have them depending on each other and stuff it's it's really cool this is this is what i want to cover for this is sort of to support some legacy hardware this is a vendor specific extension I've certainly written plenty of fragment programs in it it's you it's certainly quite usable but it's it's not it's not very pleasant to programming so it floats basic fragment programming there is a pheromone current hardware support shooter builder does fully support this language so you don't lose I'd it's not like we aren't going to provide a developer tool for you know you do provide all the tools we just discourage you from using unless you absolutely must so yeah again so now I'm going to introduce you again for those of you who saw shader builder last year you're going to i'm going to show you it again except this time it's a bit it's quite a bit different on the support for new stuff here's a here's a screenshot of it right here this is the new layout of the shader builder on the top left of the window here on the top left of the screen that looks pretty familiar to those of you have see before except you'll notice that now the the rendering window the OpenGL view is actually detached and on a separate poulos window which is convenient because the multi head users out there will be able to take that and drag it onto a separate display ahead and they'll be able to code on one screen see their graphic and full screen on the other which is convenient also also another major thing is that shooter builder adds is the ability to monitor the actual date of the graphics card in terms of how much expandable resources you're using so you'll notice on the top left most window up the sort of just above the debugger buttons you can see these little gauges which are showing you how much expandable resources you're using on the graphics card so as you put more instructions in that's going to that's going to keep growing until it hits the top and if your fragment program stops working well at least now you have a fighting chance of determining why that you filled up your graphics card and you buy a new one and then the the other great thing here is this like like of what it was before on the righteous above the resources and performance box you can see this list of up sort of identifiers when you're writing your program and you declare an identifier the actually will take that identifier and put it in this list for you meaning that when you're running the program if one of those identifiers is a program parameter you can pop open this great symbol editor which is on the bottom right most window the metallic window and you can click on that and you can slide those and that will actually change the parameters on the card you can have any number of those parameters up to the hardware limit that has some interesting side effects which I'll cover later which you're pretty exciting so just to summarize here add to support our fragment covered that texture / f you could there's more control of the texture units in this Tjader builder again program parameters can be changed in the fly you've got the resource monitor better solution for the multi head folks because the rendering window can be put on a separate head II underlying code has had some rework for better code sharing with the OpenGL profiler documentation has improved so this full documentation for all of the languages a pretty nice UI for that and it's assault it's with panther so your CD you got it it's all on there in terms of example code that that's available on the website so let's switch over to the demo machine and uh let me show you shader builder here here demo to actually great so what we have here is all I like the screenshot i showed you on the top right of the window we have the rent the rendering view which you know i can move it around at the moment is just showing a simple quad which is really what you want if you're just a writing fragment program you want to see the pixels you don't really care about the geometry so much for a lot of the things you'll be doing on the bottom right again you've got the tech the texture units inspector so here if I click here this shows me that in texture units 0 i have this rock texture loaded and text your unit 1 i have the water texture which is what's currently being shown now let's look at the code here we can see basically what's going on here this is a very simple pass through fragment program it's doing nothing special remember the programs running once I beg your pardon oh it looks ok here it must just be clear sorry with that not there ok sorry with that so what is this doing this is running once for every for every pixel let's just think of it as and what it's doing is the text instruction basically that acquires a color it gets an RGB a color from a text your unit at a specified texture coordinate so we can see that I have a temporary variable declared which I've just called t0 and I am sampling from texture unit one that's the second argument so text your unit 1 into t0 on this fragment text cord that you can see that is the interpreted a texture coordinate that it's using to look up at and then I'm simply in the last instruction here I'm moving t0 into result color result color is it's sort of a fixed thing in the language which defines the output color your writing let's look at something i could do so for instance this texture one that's it that's the texture you know that I'm sampling from so as I change that its you can see it's changing that's very straightforward let's look at something a little more interesting let's implement say we wanted a fragment program who would allow a program parameter to adjust the brightness of the image and do it all and hardware how would you do that ok let's create a new line of code right here so what we've done is we've declared a parameter P which is bind to a program parameter meaning that there's a there is a AGL entry point that i can call with the number with the index zero and I can pass in a floating point value which is sent to the card and run in this program so let's implement a brightness control pretty straightforward a simple multiply will do that so let's do a multiply remember that it's like assembly hero if you've written vertex rooms you'll be familiar it's the destination first and then the argument multiply takes takes a destination and to argument so we multiply 20 with the x component that's the first one last will change to make okay with the x component of P now right now the brightness is at the bottom so we're seeing it's black but what's this we move over here you notice p is in this identify or less i'll select stop i will open up an inspector right here it's the symbol editor right here this allows me to change those values so remember the brightness is stored in the X component so i'm editing p and you can see right here o of my mice is i can change the minimum a maximum value this slider will will go to and i want to set the range to go from 0 through to just to keep reasonable values and as i slide this right now you can see it's sending that value to the graphics card on the graphics card is that there's a brightness control implemented in hardware that's pretty straightforward stuff let's now show okay i want to implement a contrast control as well i'm going to do a pretty cheesy implementation of contrast here i'm going to use the i'm just going to basically raised the color to a power where that power is a contrast and as I if you may recall I mentioned that the power instruction is a scalar instruction note i only had one multiply instruction yet it multiplied all the three red green and blue components for me how well the power instruction will not work like that so i'm going to need three of them and i'm going to need to do it for each channel so i'm going to do that so I'm dealing with the red channel here red channel dealt with green and you notice that shader builder is keeping it like before it it as i type it's all real time it's updating on the fly and it's showing me all the sign tax information is highlighting the line that the error is on it's just great to work with ok we don't have a contrast control if I know take the y-component I set the range like I did the other 10 through five happens to work well here it's totally low contrast right now hence white but notice as I as I increase the contrast here it's becoming more and more and more contrasting and again the brightness control works as before so this is a simple example of writing a fragment program on showing how you get program parameters into it so let's let's ahead ahead almost a presentation here back to the slide so I'm going to try and keep the rest of this presentation pretty example centric and get get you thinking here so let's look at a 2d image displacement I think here of the the photoshop filters that allow you to do for instance to twirl effect that's what we want to think of be thinking about here so it's another example yeah if I to offload stuff to the GPU performance is excellent on the it could make an interesting inner scene transitions in the game or something it's certainly possibilities a little background and how this is going to work what we're going to do is I'm basically i'm going to show you how to do a twirl effect in a fragment program it's single path the way it's going to work is imagine a texture map I'm not texture map the colors of the pixels actually define the vectors of defined movement vectors like displacement vectors so what I'm showing here on the last of the slide pardon me is a very low resolution to buy to texture which we do a very coarse twirl effect now what i'm doing here is for the top left most ones that one that kind of looks red that's that's pointing to the right and it it's encoding the XY in the red and green component which you can see there and it you can see it's showing it as a twirl going on in my fragment program i'll be able to sample those those texels and i'll be able to interpret them as vectors not colors i'll be able to interpret them as just to think of texture units as just being woke up tables at this point think beyond them as being what you've used them for before important thing to note in four colors in texture units they they have to be between 0 and 1 and in positive space so I want these vectors to have you know full full full freedom of motion so what I'm going to need to do is they're packed into the texture units in a slightly different way whereby I want them to go from minus zero through to one so simple equation I just divide that by two and add point five to it pushing them into positive space keeping them between zero and one in the fragment program i will i will do the reverse of that to unpack them so that's describes describing what the displacement map is so back over to the the demo machine here where i'm going to just open up the other example which i have laid out here okay we just don't make it so you can see what's going on okay the same problem again huh okay there we go thank you so switching to the to the fragment program it's a pretty pretty straightforward fragment program it does much like what i described if you follow the comment let's like take a look at what's in the texture units in texture units 0 we have this rock texture which is the texture I'm going to displace going to warp it and then a text your unit one I wrote a little program that basically just generated a generator a spiral sort of effect and it encoded those vectors of colors and put them into positive space which is what you're seeing in the texture unit right there in touch unit 1 so my first instruction here is the tax instruction which samples the displacement map I just highlight that right here and then it does what I described which is stealing it from color space into into sort of into a vector which has negative and positive and then I have a program parameter a bit like what i showed in my last example where that allows me to control the magnitude of the effect so by multiplying that vector with a baghdad a normalized magnitude I can control how much of a spiral effect of going on I add it to the original so I'm displacing the texture coordinates here that's how i'm doing a look up i'm actually changing the texture coordinates for each fragment that's how you do the warping effect and then finally when I have a texture coordinate I sample from the rock texture map in the fruit and texture units 0 so let me show you what it looks like so here I select the magnitude parameter and the list I open the symbol editor again as I slide this you'll see there's actually a spiral effect going on you know it might not look as good as the Photoshop filter but the reason for that is simply because the quality of the displacement map that I put in was crudely generated it was a simple program you can imagine here how this theme code it totally generic you could create a displacement map that was a sine wave filter or any number of other displacements on this same fragment program you know all i'm doing is send for four values that for floating-point values across the bus this is all running on the graphics card I think that's pretty exciting so I'm let's see no I ok yeah that's a move on back to the back to the slides again so now we're going to move on to a general-purpose computation and a fragment program it given a new piece of hardware it seemed utterly correct to me to implement the game of life on it so we can say thank you to dr. John Connolly for coming up with this interesting interesting scheme so and it is quite a while ago too so yes it can be done using a fragment program it's correct to do in the fragment program because it's a totally parallelizable problem this is a great thing if you're if you have a mathematical problem which is kitty kappa be parallelized fragment programs a good place to do it as long as you have you don't have too many instructions will fit on the hardware it's also a great example of multipath because when you calculate one frame of life you need to you need to feed it back into the into it and run it again and you keep doing that for every time quantum and you keep running it to on until either it stabilizes or it keeps going and yes it runs extremely fast it's really cool and i'm going to show you that very shortly refresher course on the game of life for those who haven't looked at this in your daily job so this configuration that we see on the left of the of the display is sort of a sort of a famous configuration in life called our pentomino basically it's interesting because when run it runs for it doesn't a lot of life initial setups will die eyes and you'll just end up with a black screen well this one does not it runs and looks interesting for I think like 60 times quantum's or something and then eventually it stabilizes what life is is imagine a grid and for each cell it's either alive or it's dead there is no of no gray zones so either alive or dead so what I'm going to think about it here is think we're calculate we're running for every single cell on the grid we want to figure out is this current cell going to be alive or dead in the time quantum we're trying to calculate well the way you calculate that is very straightforward you first of all you sum up the the total of the Living neighbors in the surrounding eight cells so this basic equation on the left is what you do and number of neighbors and then you taught you some of them together when you have the total number of neighbors you apply a very simple consistent rule to that which I've outlined here on the honor on the right if the cell is currently dead if your current cell is dead and you have two or three surrounding neighbors you will become alive you turn yourself on your life if you're currently alive and you have actually died that they look wrong way around but i'm sure you can read the code oh but if you're currently alive and you have two or three neighbors you will stay alive otherwise you're the sort of fighting for resources an engine you'll die that's the basic principle of this run this for every pixel and do it per frame and you'll get this interesting sort of little colonies growing and stuff so let's take a look at I'm not going to show you how I did this in shader builder I figured it'd probably be more intuitive just to show you it on the slides here building in so let's take a look at the fragment program that implemented this first things first declaring our variables the important thing to note is that in a fragment program to address important thing in life is we need to be able to address each of the pixels in the fragment program and remember that an OpenGL the if you're not using if you're using 2d textures the texture coordinates are normalized so the last most is 0 right Ruth is one if we have a 256 the 200 six image to address like the eat of them you need to have a scale factor 2 x to be able to address each of the pixels accurately and that's very important in a in a calculation like this so the next thing we do is we need to know the coordinates of the surrounding eight cells remember this fragment program is currently determining the destiny of one cell is it going to be alive or dead so it needs to find out how the coordinates of the surrounding cells so those are the instructions to do it it's it's pretty straightforward what's going on there the next thing we do is we need to we've got the coordinates of the surrounding cells we need to sample them now so we were using it it text your sample instructions here to get the colors of those of those surrounding pixels the surrounding cells when we have that we add them together very straightforward here and then we implement the life logic and this is actually a little tough because of the limited number of logic instructions is know if there's nothing nice like that I had to use you know just is it is it greater than or equal to 0 or less than or equal to 0 and move them around and stuff and abdomen have the logic using that it it's not very clean but we'll we'll look at the better solution to that in a minute and then finally when we have the destiny of that cell is alive or dead we write it to the output color this runs in parallel for every pixel on there automatically so ok we've calculated what the life grid is going to look like for the first time quantum how do we get ready we need to feed that back in for the second calculation so how do we do that well opengl has provided the api for this for a very long time on our platform it's extremely fast it's really cool because the data does not have to travel across the bus the frame buffer can be fed back in to a texture unit without even leaving the graphics card to the process of feedback your CPU would you'll see nothing happening and your bus traffic will be zero because it's all on the graphics card if you follow these simple instructions the first thing you need to do a GL read buffer which defines where GL is going to read data from and you would set that to be GL front left which you could make it an auxiliary back buffer but for simplicity I just made it a single buffered application so I just got it from the front buffer and then when you have the read buffer set you do a GL copy text sub image to D which you basically get a rectangle from that and puts it into a texture target again it's covered pretty pretty much what i've said there enough a little code snippet it's only two lines to do it the it's very fast now the moment you've all been waiting for what does life look like when it runs on a fragment program so back to demo to hear let's not say goodbye the shooter builder for a moment and open that up okay watch it if it is quick yeah so basically this configuration it started off as three of those little art pentomino configurations I described you can see it's sort of it entered a stable state right now there's a little bit of movement but it's totally stable right now so that was life running on the on the graphics card it just kind of interesting can we cover it at the end Thanks okay now let's head back head over to the slides here and let's look at how does optimisation apply to this new environment that fragment program i showed you was hideously unoptimized extra sample instructions these are the very things that you really don't want to be doing in a fragment program because yes it's very quick it's very very quick but it could be quicker remember i was only i was only i had a very small window and I was only text I was only texture mapping the face of one polygon if I was you know in a complicated 3d game environment which I'm sure some of you work on on a daily basis performance really really matters and we're going to look at how you can make big big improvements in your fragment program performance suitable is going to help you here a lot I think let's first look at some of the tricks that we can do the first thing is it possible why instead of calculating things in the fragment program if it's something that isn't absolutely necessary for to be calculated on a per pixel basis remember you've still got your vertex program running that's running for every vertex now for that example there were only four vertices defining a quad but many many many fragments so I'm why not calculate it on a per vertex basis of its if it doesn't need to be as high resolution and B you can then pick up that data in an interpolated form in the fragment program as those values will be interpolated across the surface the other useful thing is using look up tables is at this time it can bring performance benefits and I'm going to show you how to do that in the you know instead of doing a calculation in the fragment program why not pre January it a doubt at your application launch time a table of numbers stored into a texture map and then be able to text sample that in your fragment program and there you have a you there you have a look up table I'm just like you're familiar with the other thing is the instruction set is very rich so I would encourage you and I know no one likes to read the manual but it's a good idea in this case to read there to read through the instructions shooter builder has a nice instruction browser go through them learn your toolkit learn the instructions you have because you could be implementing something in four or five instructions and you realize oh it's just a strange instruction I've never heard of actually does all of this in one instruction that's going to run faster and it's going to make your code short or too so let's look at a optimizing life specifically how did I go about that this part I'm a little concerned about because it's a little hard to explain but I'm going to do my best so I did it this is a little little clipping out of the light the grid life was running on its that our pentomino thing I was talking about one of the the thing that added a lot of instructions to that program that I showed you when I was running through it was the process remember there were it texture sample instructions not was it was sampling the state of all the surrounding cells well that's a lot of texture sample instructions if we could do that on the last it would be great not only do we have to sample those texture that those eight times we also needed to calculate the surrounding coordinates to do that we needed to calculate the surrounding extra coordinates nowhere eight of them well I'm going to show you a way that I went about it that actually has allowed me to drastically cut down the instructions here if we look carefully at the the life configuration we realize that it exists only in one color plan it doesn't have it doesn't require the three color channels so let's take advantage of that we realize that in fragment program the instructions ninety percent of them will deal will operate in one instruction it would like altivec on with it well it will operate on all four components at once so if we can pack more data into those channels and execute one instruction weaken in one instruction we can do the work of like three or four so what I did was outside of the fragment program just before I did the feedback where I was feeding it what I was doing that the feedback I actually took the the sort of one bit image and I took it and I offset it at one pixel to the left and I stamped it into the red channel then I offset it at one pixel to the right and I offset it and I stamp it into the blue channel and then in the center it within the green channel so i end up with I've magnified I but you can see what I've got here on the right hand side and that shows you how I packed the data in this is really good because now if I do one texture sample in the fragment program by accessing the red green and blue channels independently I have got the state of this of the left and right tick cells meaning to get all the surrounding neighbors all I need to do is read the is-3 texture samples instead of instead of the all those other ones I only need three and then by using the red green blue channels I've got all the pixels that's a useful thing the other important thing is remember that nasty logic that I implemented which was to choose the weather the cell would be on or off well that was like six or seven instructions let's get rid of those with like one instruction let's create a lookup table here is a texture map it's a two by eight texture map and basically what we do is on the y component of that on the Y text and the texture coordinate Y component we just put in a 1 or a 0 is this is the current cell on or off and on the X component we have the number of neighbors that we calculated and by simply doing a texture sample with with that lookup value it's a lookup table we get back on or off that's the color of the output cell that's another pretty useful thing to be able to do after doing that here was a benefit remember a builder had that nice thing would show you the number of instructions and a number of texture samples that were going on on the graphics card well I just created a chart to show you what I was able to get it down to and it runs exactly the same it looks no different except it's faster if you measure it the what we see here is the instructions went down from 36 instructions down to nine instructions that that's a pretty big deal texture samples those were expensive to run on the graphics hardware Gotham's down to nine down to four temporary variables well not 13 of them anymore got those down to five and then this is this last one is interesting added DTR that expand that for you it's a dependent text you read what a dependent text you read is so I've shown you the text instruction which allows you to kick a texture coordinate and look up and get a pixel back from a text from a texture unit well imagine that the coordinate that I use to look that up was actually derived from a previous text your sample you can imagine that you have dependencies between them well for every one of those dependencies that's known as a dependent texture read the graphics hardware has support for four of these four of these the pattern texture reads I am not using one of them it's still a huge performance improvement and dependent text you eat if you're going to use look-up tables in your program you going to inevitably use these you've got four of them I find that is more than enough for anything that I'm going to implement you can look into that after now I've this is my my final part that I'm going to look out in this part have had me most concerned because it seems that a lot of a lot of what I look at other demo programs of people who may be implemented for pixel lighting models using not necessarily ARB fragment program but other extensions I find it personally very hard to understand what they're doing I've looked at their example yeah they look cool but it's like I'm not really sure what they're doing so what I've done here is I just threw a lot of way and start from scratch implemented my interpretation of a lighting model which I think it's correct and it looks good I'm going to just explain step by step I'm going to break it apart for you sure it's how it's done and try and dispel any of the mystery there so this there's certainly some 3d math involved here but it's sure it should be quite understandable before explaining it let's look at what the contributing entities to this are what are the think what are the sort of inputs to this fragment program the first one is the position of the light source where the light source is relative to the fragment that we're currently calculating the color of well so we're going to call that LP the other thing is I support for variable brightness lights so that's another thing for it for the light source we have another just a scalar which is how bright it is the next important thing is where is the fragment in in in 3d space where is it located that's important to know as well because we're going to use it to calculate a vector from the light to the fragment important to know that the other important thing is the normal of the fragment this is the key part that makes per pixel lighting models look so cool and that is that the normals unlike the OpenGL traditional lighting model where normals are only our only exist on a per vertex basis under interpolated across the surface of the polygon here the normals are on a perfect cell via this meaning we can create these amazing detailed surfaces and the way it's done is if you recall back to when I showed you my displacement map where I had encoded a two-dimensional vector into your color this is exactly the same thing except we're encoding a three dimensional vector into each color so not only an XY but an XYZ so we're able to encode a three-dimensional normal into a texture map and that site by sampling that in the fragment program I can get the normal at that point which is the which is a large reason why this look so good and then finally we have the fragment the we're calculating the colors of the fragment on the surface of a polygon so we need to know the normal of that entire polygon so that we can transform the defragment normal I'll cover that in a minute another useful thing I did to make this program simpler was I wanted to calculate everything in model space so the in your 3d game your light sources you're probably going to want to define them in in world space or I space you're going to want to find them relative to your world and then what I did was in the vertex program I try x that that by the inverse modelview matrix to transform it into model space so the fragment program had something easier to work with so I've already actually covered the normal map kind of what it is but this is what it looks like that's the best texture map that I'm going to use and you can see the one below is the normal map well these psychedelic colors are the that are well defined the bumpiness of the surface again the same little trick applies the I have to do this sort of you know normals are obviously they range from their normalized they range from 4-1 through one I have to squash them and push them up into positive space using this super simple equation and I'll unpack them in the fragment program so quick quick run-through before I demonstrate it what's good what's going on here we calculate the vector from the light through the fragment we have a light source we were fragment nunavut vector we calculate that I'm a light vector incoming lvi then while we when we've done that it's pretty trivial to calculate the distance just a scalar the distance from the light to the fragment if we know the vector we can get the magnitude of the distance on that we use to attenuate the light later on to get a 10 get attenuation in here also then we acquire the fragment normal like I described we sample the normal map we get the normal FM then we transform that relative to the surface of the polygon I explained that I want to go to then the good part if we know the fragment normal and we know the incoming light vector we want to calculate it after its reflected so it comes in and we get it coming out like that we do that for every fragment that's important also and then when we have that we do a dot product between that with the incident light ray and then the direction of the light source so in this lighting model you could actually not just Nestle have the light pointing directly down on the surface but you could have it those sort of going along the surface you can change it I've actually made it fixed right now but there's no reason I couldn't move it around I just wanted to have them have spent so much time here on this then the next thing is to do the attenuation remember LD we calculate the distance from the light to the fragment we're going to use that to attenuate the the light intensity which will create through the rough fall off which is important also like for instance I hold a light here a little light here shining on this it's not going to have very much effect lighting up the far wall that's the attenuation and then we multiply at that with a light brightness control so quickly little bit of animation here this is the surface or a polygon FP that's the current fragment we're calculating here we have the FM the fragment normal which I acquired from the texture lookup encoded in the red green blue channels is our incoming light vector like so then we reflect we can't get it after reflection with the normal and then LD that was the distance I was talking about so I think it's probably very clear in your head now what I'm going to do and I'm going to move over to shader builder and I to illustrate this in action show you how really pretty cool it looks close our previous example okay just uh get these win over in shape here and the resolutions a little smaller than what one typically used to is that is that in range now can you see it is that good okay so if you can see this here's the fragment program I'm not going to go into it on a per instruction basis on this it simply would take too long this again this exact thing i'm editing right now on stage will be available as a demo afterwards so you can go home you can run it and onto the build up in your new g5 you buy and then you'll be able to see this you'll be able to see it play around with it what i've done here is i've actually commented out the pro signed the program in such a way that the exact order of the stages I described to you are the exact order of the instructions in this program and I put a comment that matches that exactly so when you look at it and you look at my corresponding slide you can write you can walk your way through this program after so let's take a look text your unit one there's our North there is our normal map in the bottom right corner of the screen in texture units 0 I have the best texture which is the gargoyle fifth in the rendering window right now what you're seeing is for every fragment you're seeing the color you're seeing the in the incoming light vectors in displayed for you in encoded and other decoded as a color so if i go here and I open up the symbol editor this will allow me to move the light source around so here I'm moving the light position around and you can see the vectors are updating let's pull the light back from the surface you can see it we're close to the surface there it's right at the light is right on the plan knife you can see a singularity there let's put it back no let's start on commenting some code and watch this effect build up let's calculate the vectors after they've been reflected off the surface and this also off depend this depends on the fragment nor it comes from the normal map slit on commenters line these are the light vectors after reflection again move the light around you can see them updating right there let's not do the dot product that i described with the light direction vector which is constant right now move it around again you can see it moving around and then let's do the attenuation and you'll see the things with big difference that looks better and then the next stage is we we multiply it by the color of the light the moment the lights just what white light what we want to be able to have that controllable so we do that a colorful light and then finally we modulate that with the best texture map which we know I have that's that's that now if we move the light source around you can see I see how good this looks I'm actually I'm going to make this window bigger so you can you can see it more clearly let's start moving the light around you can see it [Applause] because we cool the light away from the surface she is getting far away lights so far away now i leaked it is even showing up bring it in really really really close so I think it looks pretty good and then we can also do of course we can change the color of the light that's another program parameter by adjust that I can edit it as a color you see it's changing the color as well so multiple parameters as well as supported shooter builder so I've certainly uh um she should be fun for you to be able to look at so at the end of average the end of my presentation now I I hope you all find this enlightening um and I hope that you'll go home afterwards and we'll run the new shader builder if you look at these examples because I think there's a lot to learn here I'll a lot of scope for implementing these things so thanks a lot and I think I'm going to head back to come back to travel here [Applause] Thank You Jane um if you have any questions with information that you've seen today feel free to contact any one of these email addresses up there I'd like to actually add mine to it I'm Travis at apple com and again any questions about the graphics technologies or various things you've seen relating to 2d and 3d graphics at wwc feel free to email me with your questions you