WWDC2004 Session 220

Transcript

Kind: captions Language: en session 2 220 which is OpenGL shading language and we're pretty excited about being able to have for Tiger the OpenGL shading language available to our developers because apples extremely excited about the fact that we have programmable hardware these fantastic GPUs that we can do incredible things with many of which you've seen particularly this year and even starting last year with some of our demonstrations and also sessions on programmability with GPUs the interesting thing is that previously working with these gpus has been you know basically almost a step back to the past in terms of the way you program these gpus was often and sort of like the assembly language level which isn't approachable to a lot of developers so what's been happening in both GPU development that's also been driving how these functionalities are exposed to the developer is the evolution of higher level shading languages that's exactly what OpenGL shading language is and it's an approachable way for developers experience with languages such as C to be able to really look at leveraging the power of the GPU in their application and it's also important for you to realize that there's been some confusion that we've had at this wwc because we've talked about OpenGL shading language in two contexts one is obviously in its relationship to core image where core image has sort of a subset of OpenGL shading languages it supports enables you to do all the interesting effects that core image supports but I also want to be clear that the purpose of this session is to communicate about the full OpenGL shading language which will be part of OpenGL and tiger so that's going to give you all the capabilities of the language and it will expose all the capabilities of current and future hardware GPU hardware that supports programmability so on that note I'd like to actually might our speaker James McComb the stage to take you through the session thank you [Applause] morning and thanks for coming to this session I guess that's quite a lot of people make it back from cupertino last night but uh thanks for coming along yeah we're pretty excited about this the shading language stuff that we've been working on I guess I want to sort of start off with looking at where we were two years ago we came here and we hardware with coming out that had a programmable vertex unit and we talked about our vertex program which was a language to a life well reprogramming is the vertex unit and last year we talked about our fragment program which allows allowed you to reprogram the sort of fragment look up the fragment look up and some of the visual effects that you can do with this really are very powerful and increasingly more and more your games are showing up and various things are using this becoming clear that you can do some very powerful real-time effect with that however those languages were designed suddenly you know we were handed with this this fancy new hardware we needed to come up with a language to program it the languages that were created were very quite close to the hardware they were assembly like in nature well now we've had some time in the OpenGL community to sit down and discuss a way to expose that hardware and also future hardware as new capability appear we want a language that can expose that be as portable as possible in the sense of the same program can run on various pieces of hardware hardware supports new operations the language will expose that and we don't want to be revising the language every year we want something that we can kneeled on and there's been a language now for a very long time that's proven itself to be very good at this and that's the c programming language for the general purpose processor see got people away from programming and assembly all the time and i guess the fact that we're still programming in c today is a good sign that it is pretty well designed language so the Open GL ARB got together and pretty much took the C programming language and added some new some flooding you sign tax to it that that's required to I guess make it appropriate for pre shading so let's take a look at the support for this if you want to if you want to check in your application if it's there these are the four extensions that are going to be on an OpenGL implementation that support this as I go through the presentation you'll see why you really there are four of them but the most important one relief to check for is the one that's highlighted there are shading language 100 if you check for that and that's there you should have at least some level support for the shading language on your implementation hmm also a hardware support for this really any any hardware that has support for armed fragment program and our vertex program that hardware can support the OpenGL shading language or at least varying amounts of it the shading language being see it has looping branching function calls and also a lot of built-in functions like you know like an adult for generating noise and various things well it turns out that a lot of current hardware will simply doesn't have the actual machine code instructions that will do that so there could be programs that you'll submit that simply won't run on the hardware at this time however this is a forward-thinking language and you know as Hardware keeps coming out soon it will be fully supported but that leads me to my next point which is that you know current hardware resources are limited either in terms of the range of instructions that are available or or in terms of how many registers it has therefore I'm going to talk about software rendering now here we are at the cutting edge of technology that spawns a new hardware and well we're back to software rendering again well let's think about really why that that is in the past opengl really has sort of been used mainly for sort of non-photorealistic rendering it's being used for either previews in an application like a CAD application it's been use of the preview but for the actual final rendering typically applications have had to use their own software renderer or in the movies people have used render man and various things for that well OpenGL is developing and now with the shading languages photorealistic rendering is is becoming a possibility so a software renderer becomes necessary because we want something that is high precision has real 32 real 32-bit float and current graphics hardware if you take the same thing and render it on to vendors graphics cards and do a comparison the colors are slightly different the gentleman who looks ever so slightly different you don't get the exact same rendering so we want a software renderer that can provide consistent final renders and the other thing is massive frame buffers if this is for for the movies or even for print of some sort you might want to have a huge frame buffer so we want a software renderer that can really do that and also our new software render we wanted to support vertex and fragment program and it using using a runtime compiled technology we've been working on we're able to actually generate efficient altivec code that will run any of these fragment programs which is really pretty powerful and you know it's not as fast as hardware but this is the fallback if you submit a shading language programs that will not run on hardware it will run it'll fall back to the software renderer and it will run just not as fast so let's uh let's have a demo here of the software renderer in action just to show you sort of a what we've got here so for any of you who are here last year you might recall this this particular image here this is actually just an ARB vertex and fragment programming shader builder this isn't even the high-level sheet language I'm showing right now but this is really merely just to prove approve the point that our software renderer here you can hear you can see the scene being rendered this is running on a radeon 9800 in hardware so it's obviously you know it's pretty quick here and if i can move the light around like this now if we click this button here you can see now we're running in software this is a fragment program running in software not yet it's obviously slower but it's not bad so that's an example of our software renderer right there so let's just take another look at another example of this in action see here this is another one that if there's any medical imaging folks in the audience they might find this somewhat interesting it's a it's a shader that's basically I have a 2d texture which is just a slice of an MRI scan and one thing that I thought would be kind of fun to do with that is if we take a look at the texture units here in one of the text units I've got the slice of the MRI scan and in another one I have a one-dimensional texture it it pixels wide and one tall and it's just a color spectrum and I thought it'd be kind of interesting to write a shader that allows you to basically do a skill and bias on a minimum maximum clamp on the luminance values here and then basically quantize it and pick out bands from the from the spectrum so I just I guess I've commented out some lines of this shader code here but if I start to bring it in you can see there it's it's it's basically it's mapped the spectrum onto that and that's I I'm calm at these lines I've basically got control of these parameters so I can do this kind of thing and you know it's pretty quick kind of neat but if I switch the software renderer is just another example of it you can see that you know it's still it still runs pretty well so that's just an example of our software renderer that's in there right now so this will become increasingly more useful as people start to use the shading language and as we start to get the get the whole thing tied up so let's just really go through what you're going to learn today what is a shader we'll talk about specifically what the definition of that is we'll look at where they fit in the OpenGL pipeline and I'll go through some terminology which will help as I continue through the presentation we'll look at how shaders are currently expressed the current languages lots been too long on that and then discussed it again why a high level language is needed and then we'll go into the language itself get the specifics what you can expect when you start to program this language so first of all what is the shader well there's a couple of things really there's a program and this is in the terminology of the high-level shading language there's a program object and inside a program object you have a vertex shader and a fragment shader the the intent of a vertex shader is that it generates a transform vertex with an untransformed vertex and opengl state as inputs that's its goal takes vertices have been submitted from your application and perform some operation to put them into wind into winter coordinate a fragment shader eda it's kind of similar except it's dealing with with pixels it generates a pixel on the frame buffer with the vertex attribute interpolated across the scanline passed into it and also it has access to opengl state and obviously it can sample from the texture unit where do they fit in the pipeline so let's look at that you have your vertex data coming from your application and then you have pick em the pixel data handed in through your text image called or whatever and also this cano loves he comes from displaylist normally the vertex data gets transport get through a fixed function get x a model view matrix you get transformed and it gets clipped by the clipping plans on the sides of the screen or even user clipping plans if you have any and also the vertex color get step by a very basic lighting model that's built into opengl and then basically you end up with a bunch of triangles and then it goes down to the graphics driver or the software renderer which takes those triangles and breaks them up into an structure p zoids for the flat top and bottom and go through them scanline risk online and basically stamps these pixels into the frame buffer and it's a very fixed operation if you've got multi texturing turned on it will you know pick a text or from one text unit then another if they be module listen to get or something like that well the programmability part comes in whenever you start to pull those out and you're going to then have the ability to replace these two parts of the pipeline that's what you're doing essentially so take a look at some terminology here a primitive is is really a series of vertices with a connection rule so in a sense ever if you're an immediate mode everything between your GL begin and end in the begin call you specify the connection rule and everything in between that is your is your vertices and you can specify vertex attributes there too shaders are applied on a primitive basis you can't switch shader halfway through a primitive so it's important to note that that the shader remains sort of constant throughout that also opengl state it can only change on a primitive basis you know you can only enable or disable something outside of a begin aunt so that's going to be the state is going to be uniform throughout the execution of that shader and that up turn uniform is important because it's actually going to be a keyword in this in this shading language and I'm just trying to explain where they come from also looking at a vertex obviously it's a 3d point comprising primitive the vertex shaders are around once for every vertex now in hardware they could be ran in parallel but still is one execution for every one of those vertices they're independent from one another for that for the purpose of parallelism so therefore you cannot have communication between subsequent executions of a shader they're discrete from one another also AM vertices they have attributes associated with them you know mainly their position the color they can have a normal texture coordinate on a bunch of other user-defined attributes that you might want it weights or whatever else you might be using and then a fragment is really it's a pixel on your on your frame buffer the that comprises the scanline during the triangle rasterization the fragment shader determines its color and the fragment shader has access to data that's varying coming from the vertex shader and its varying because for every execution that fragment shader those attributes are there interpolated them across that scan line so let's take a look at the current languages that are available today the ARB approved languages are vertex and fragment program again there other said their assembly like except they have no bitwise operations they're all kind of high-level instructions somewhat high level meeting any sort of mathematical operation they all deal in floating point and it's a sim d instruction set and add is going to do an add-on for components in a register all the registers or four component float and also the instructions are a fairly close mapping of the hardware instructions which is actually the very thing that we are trying to get away from nye and the language has enforced resource limitations which can be annoying but the upside of that the good side of enforced resource limitations is that you have a clear you know if it's going to run on hardware or not you know that you can query it and as long as your program is within those well-defined bonds it will run on the graphics hardware this is just a look at the the instruction set that was available I broke it into four sections this is a the ARB fragment program instruction set you can see you know assembly like names this is an example of an ARB vertex program a very simple one that merely transforms the vertex and moves a texture coordinate through pretty straightforward and this is a fragment program that basically samples at samples of texel from texture unit 0 on texture unit 1 and then it multiplies them together and writes it to the output fragment that's what those languages look like this point pretty straightforward but it can get pretty complicated if you want to express large programs and the complex effect so let's take a look at the high level shading language at this point why do you want to use this potentially left code to write you can express it in terms that you're more familiar with I'm sure some of you are familiar with writing assembly code but you know no matter wat you're really only writing at your inner loops you're not writing it every day with your code so being able to write and see it could be convenient you no longer need to deal with specific registers anymore that's all virtualized just declare variables and use them as you please you can depend on the compiler to some extent to take care of that for you also is a the goal of Hardware independence this language doesn't tie itself to any one piece of hardware which i think is I think is a good thing in the long run and also is a nice the nice intent of being able to have code reuse the ability to have function calls and also to be able to draw in multiple files comprising one shader is very useful you could have like a files like I got a library almost full of like useful chunks of or code the ears and you can function call into that from the shading language so that code reuse is pretty nice here's just a comparison of sort of what the two things look like side by side you can see there's the the ARB vertex program that transforms a vertex and passes its color through and then you can see the GL shading language a version of the same thing it's a more natural form of expressing it really look it just looks like see really has a main entry point and so forth so the the object model here the way the against the API kind of works is to make a vertex shader object you basically need a about you can have one or more strings which are the strings of the C code that you basically want to put in there can be more than one of these strings and they're concatenated together and you can function call in between them you need a vertex shader object and then also it's the exact same thing except for the fry and shader when you have these two these two objects created you you basically pass each of those through a compile stage which compiles it onto the machine code which then gets passed into a linker which is a interesting regardless these things now with four shaders but it goes into a linker and then it builds a final program object and that is what you then you use that program object and start drawing primitives and that the shader will be applied to those the language is he inspired it's got the usual main entry point the language supports looping and branching and all those other things just like you'd expect there are no pointers since uh since really you're not you're not accessing memory directly there's no real concept of accessing memory in this the idea of having pointers doesn't really make sense in this language so that sign tax you will not see there are new data types since the hardware registers are all up our vectors there are new data types to support that and also there's a matrix data type as well since we're introducing vector data types what happens if you want to pick a scalar out of that you want to get access to a scalar or you want to reorder the scalar components in a vector well there's use new sign textures through the length I we can pick out a certain component or reorder them there are new qualifiers you know you've got the see program for a vertex shader and then you can see program for a fragment shader they're separate how do you get communication in between them well the way that's done is via declaring a variable with a particular qualifier in both programs and having the names match that's what the linker stage does is it ties up those loose ends and then we have there are a bunch of intrinsic variable just pre defined variables that are there and either they they allow you to access OpenGL state or you write your value into these predefined variables so let's take a look at the the qualifiers the type qualifiers that exist the first stage is the attribute inside your vertex shader to access the vertex attribute you basically just declare something with the attribute qualifier they only make sense to be declared in a vertex shader and attributes they change at most once per vertex and there we don't like can't write to them think variables are declared with the uniform qualifier basically those are things that are set outside the shader and their uniform throughout the execution of that shader so they're read-only and they're frequently frequency of changes that most once for primitive things like the OpenGL state and even in fragment program that even texture the texture data is considered uniform as well the other thing just like see you've also got comped which is just in line constants you want to declare in your program obviously read only if you don't put a qualifier in your just declaring really a temporary variable just for scratch the space it's important to note like I said earlier the data in those will not persist between executions of the shader you can't write something in to attempt unexpected to be there on the deck we process of the next vertex I wasn't going to happen they are readwrite access and then there's a qualifier varying basically variables declared with the varying types are basically your your pipeline of communication between the vertex shader and the fragment shader declare something is varying the rasterizer is going to take that value from the vertex shader it's going to interpolate it and the fragment program will get that value as its interpolated across the scan line so let's take a look at some of these new data types you basically have three main classes of data type you have floating-point types there's an integer type and there's even a boolean type now you can't expect that that integer is actually going to be stored in a fixed res register so there's still no like like shifting or bitwise operations but the type still exists because some of the functions expect it to be there also it can these are the vector types and it can be two component three component and four component vectors and again use Swizzle and picard scalar components also they can be accessed as a zero-based array so if you have a vector and you want the second scalar you cadet you could do you know variable open the square bracket and actually pick it out like that so it's a two-way successive obviously that there are scalar types now if you declare something as a float it's just going to have one once killer in there and of course the flow control constructs in this language they will only accept a boolean type 2 to brunch on there's also a bunch of matrix types you'll probably want to use these for getting the transformation matrix from OpenGL store stuff in there or you might have your own matrices within your application that you want to pass in as parameters stored column major which is compatible with the way the rest of OpenGL works and again you can access these as an array of column vectors if you take a matrix I guess take it take the first element of that that will return a vector type and then with that vector you could access the first element get a scalar that's a title work the last data type is a sampler basically it's a handle to encapsulate a texture unit there are function calls in this language for getting out at exel from a texture unit and a sampler you need to pass that into the first argument and then outside of the shader in our application you then sort of you attach a texture unit to a sampler that's how they work it's just a handle and again they're always declared with a uniform qualifier because texture units don't change between executions of the shader hour during execution of the shader data from these is of course read-only so here is another sort of deviation from the C programming language because we know I have I get aggregate data types in a sense basically things that are non scalar we need these constructors not so much constructors in the c++ sense with the score even perhaps similar an idea anyhow they're used to initialize an aggregate type and of course in c you had this concert you could just do you know declare a variable V equals and then you could open the braces and fill in each of the components what a constructor is is basically when you declare a variable if you want to assign something if you want to put something into it you basically make a function call which is the same as the type as the name of the type and then in those brackets you fill in the values that's how it works so I've shown you a couple of examples of that you can see there I'm declaring a four component vector type I then have to call a function vac for with for scalars inside of that to fill it in and then you can see the similar example filling in the actual structure you make a call which is the same name as the structure and you pass in the values and you can see how they actually they net you can these constructors nest that's one thing that's a little different that you'll need to get used to but it's really it's not hard to figure out in terms of the operators you have there's really there's nothing really new here the only thing that you'll see missing is the bitwise operations there's a logical and an or and so forth but not a bitwise so Quisling just to give an example of how that works you can see here I declare v-0 and construct it with us with for scalars and you can see then but but I'm assigning v02 v1 but v-0 has a Swizzle art and you can see what that's actually doing is actually reordered the the components within the steelers within that these swizzles they can either be X Y Z W they have connections if convenient names XYZ w you'd probably use if you're if it's positional data rgba if its color space data and there's even an s TR q if it's used for storing texture units you can use them interchangeably but to make your code more readable you might want to take advantage of that also as a result of a Swizzle you can actually change the data type you can see that when I'm assigning a V 0 2 V 3 V 3 is only effect two so it has only two components you're going to need to pick the two components that you're going to assign to that which is why i'm picking q and ass and also you can see that type conversion can be used used to just for just one single scalar in terms of the the operators the in a sense they're they're overloaded in the sense of depending on the type of their working with they'll have us they'll have a different behavior no it's not don't get too scared of that it's pretty simple the behavior actually for all the operations apart from the multiply the behavior is very obvious it simply applies if you do an add a equals B plus C it's going to perform that ad for all the scalars within that the vector it's going to do for ads if the matrix it's going to do 16 of them the reason the reason multiply is a little different is that if you multiply a vector times the matrix it does a linear algebraic multiply where it's actually going to be for dot product that's what it's actually going to break down into which is really convenient because at the beginning your vertex programs you're always having to do these for dot products to transform your vertex will not just a one liner it's just you know position equals vertex times modelview matrix it's pretty convenient and me at six times matrix is also it works that way so I talked about the predefined variable you can see that a we have a predefined ones one for the the vertex color normal and access to the texture units to communicate varying types to communicate from vertex to fragment is it again texture coordinates in the front color and outputs from the fragment shader when you're right you write your final color into GL frag color and that's what will get written to the frame buffer and then there's a bunch of uniforms are declared in your modelview matrix and access to the bill lighting information from the Geo state so let's take a look at a we've talked a little about the language here let's look at how to I've shown you the program string how do you actually load these into your GL and actually make use of them in your application well the first thing you want to do is a acquire the program strength load them from a file do whatever you need to do but uh load them into two two strings we've got a type declared for that GL char our pointer load them in and then you want to actually put them in an array of strings and the reason for that is what I said you can actually have multiple strings that comprise a shader with function calls between them so in this case I just have one string so I'm obviously fitting the array size to be just one but that could be any number the second step you need to Claire three handles one for the vertex shader object one for the fragment shader object and then one for the final program object that these are both going to be attached to so you either have those declared and then other two variables are useful is basically a status status variable and also the length of a log because when you perform compiles and links you're actually going to get information back from the compiler that you might want to check or if those errors you're going to want to see them so yes there's aisle is not a log of it comes by your next stage let's create the vertex shader object you can see from the the code I've got here how to do it it's not very hard you basically create shader object on you tell you know it's a vertex shader GL shader source ARB allows me basically to attach the string to that compile shader r will actually perform the kit will start the compiler going generating the machine code for this and at that point you want to get back the status of the compile to see if it succeeded or not and also you want to get back the length of an info log and you need that length because you'll need to need to allocate some memory to store your login too hmm so you will repeat that step for the fragment shader it's exactly the same thing except with vertex pretty much replaced with the word fragment so you're going to end up with these vertex and fragment shader objects at that point you create a program object you attach the vertex and fragment shader to that and then you call link program and that's basically going to tie up varying date varying variables that you declared in the vertex and fragment shader it doesn't name match on those and connects them together and the linker you also need to check for errors as well because the link stage like like the C programming can feel I need to check that so assuming that that all succeeded and you have your program object ready to go if you're ready to start drawing geometry with that program object you make a call to use program object and then you just put it in there and it will start to draw with that very unlike our vertex and fragment program there's no GL enable/disable you don't enable and disable it rather you use a program object and then if you want to turn off the shader you call it with no not well to that will switch you back to the fixed function pipeline so I mentioned that it inside these languages you can declare variables with the uniform type basically are your way of passing parameters into your program for those familiar with our vertex and fragment program these environment parameters and local parameters well they're all environment parameters at this point and this is how you set them you basically you should just two calls required the first one is you need to get like an index back which is the position of that uniform and you do the first thing you pass in is the actual string that you declared in your program it gives you an index packet then you make call to GL uniform and there's several variants of that for you know passing in an array or some immediate inline floating point values the pass in your uniform data the same thing is true for vertex attributes you basically get the name you get an index from the name of the attribute and you set the values to take a look up there are a bunch of built-in functions that you're really nice there's a kind of a library of functions that come along with this I would encourage you to if you're interested in this in this language to go and get the the official the orange book on the shader language because the range of these functions is too much really for me to go through today but you should look through them because there are a lot of very interesting ones some of the ones that I've been finding quite interesting as a noise generation functions which are really good for image processing basically random number generators but there's a there's a lot of variations in them there's other things like you know like a like a mix function for doing a linear interpolation between two values which is convenient and also the the texture access functions naturally I want to take a look at those because those I said bubs I think I feel are probably the most important one thing that's really pretty nice about this language is we've always thought of texture access being available in fragment programs because all you want to text your map your surface well the shooting language allows you to do texture sampling in vertex programs as well this has some pretty powerful side effects you can imagine passing just a bunch of vertices into OpenGL and in the vertex program displacing them based on a texture map suddenly you know you could have a live video stream going into your GL have a bunch of vertices and have like you know like there's three eyes like a 3d fifth almost showing up there's a lot of a lot of possibilities there or you could even just store the generic vertex data in your texture units it's kind of kind of mind-bending but the things you can do there are large in terms of of texture access basically it's just a function call and you pass in a texture coordinate masama and basically it gives you back a four-component value which is the RGB a value at that point the really the functionality here is the same as ARB fragment program I got obviously with the addition that you can do it in the vertex program it does add support for depth textures where it can get at exelon automount do a comparison against a depth buffer which is useful for shadowing effect so now I actually I would like to switch over to show you a bit of a demonstration here no it's important to note that this is really the bleeding edge here so it's not done yet but I want to give you an example of sort of to show you how far along we are with this so hopefully things will go according to plan here hmm so I went ahead and did a little work here to get this integrated into shader builder so that I could show you what we have I should let me let me first show you just typing this in I'm going to make the font here a little bigger so that you can see a little more clearly okay so let's go to the fragment program here on write a simple one just declaring my main function right here and then basically in my fragment program I'm merely going to pass through the interpolated color data coming from the vertex shader to the output so this is the predefined variable geo frag color and you can see as I'm typing here again as is common and shader builder it'll do the real time sign text checking and even though there's a see it can we r still able to do that which is kind of nice so that's the fragment shader done that's a very simple one and then in the vertex shader let's pick OpenGL shading language oh dear let's uh let's try that again let me just show you what I've got here are they as I said this is a very recent code here you can see the high-level shading language being used to express a pretty simple lighting model it's just done in the vertex program the vertex shader space the fragment shader is merely passing through the color let me make the code make the code a little bigger hopefully everyone's able to see that ok but you can see the kind of stuff we're doing this is common and see you know being able to nass brackets and so forth so basically you can express these things and a lot less lines of code if I want you can see on this line i declare f1 and here i use it again well i can go ahead and like move it inside deletes outline hood we're getting there we're not done yet but this is this feature is slated for for tiger but we really want to get it to you as fast as we can so we're doing our best difficult specs to implement so we'll we'll keep you posted on that so I guess I want to wrap up at this point I hope you enjoyed hearing about this at least and I hope that when it becomes available you'll start to use it and yeah I guess at this point I'd like to invite travis up if I'm sure there'll be a lot of question