Transcript
>> Luc Semeria: Hello and welcome to OpenGL
Shading and Advanced Rendering session.
My name is Luc Semeria, and joining
on stage later will be Michael Swift.
What we're going to talk about here are what
are the techniques that are used in Quest,
like how to implement those techniques efficiently
on all of our devices, and we go also a bit further.
So here is the agenda.
We know some of you are not quite familiar with OpenGL
ES 2.0 yet and don't know everything about shaders yet,
so we do a quick recap of the programmable
graphics pipeline, and we'll do --
we go through the basics of programmable shaders.
Then we dive down in more details
on real rendering techniques
and how to implement them efficiently on the devices.
So we talk about skinning, for animated characters,
lighting, how to make nice lighting and shadows.
So if you've been programming with
the iPhone using OpenGL ES 1.1,
you are going to learn a lot about
shaders here and how to use them.
If you've been already programming with shaders on desktop
or with OpenGL ES 2.0, you may already be familiar with many
of those techniques, but we are going to show you
how you can efficiently implement these techniques
so you can have real time nice-looking effects on all
of the devices including the iPad and the new iPhone.
Let's start with the recap of the graphics pipeline.
I bet most of you have seen this picture before, right?
Start with the objects, bunch of points, those points are
broken down into vertices that get passed to Vertex Shader.
One of the things that Vertex Shader does is it transforms
the coordinate of those vertices from object space
into cube space through eye space.
Next step, you take those vertices, put them back
into a triangles, the triangles get rasterized
into separate fragments, those fragments are
going to end up as pixels on your screen.
The Fragment Shaders, one other thing it does is compute
the final cutoff of these fragments, and then it gets passed
down to a few more steps and shows up on your screen.
That's the graphics pipeline.
Now two components here that we are going to focus
on today, it's the Vertex and Fragment Shaders.
They are programmable, as I mention, and the way you
program those is using a C like language that's called GLSL.
And like, you know, any program, you need
to compile it and link it into a program,
but this time, the program runs on the GPU.
So shaders are the way you program your GPU.
And I think a programmable pipeline is great.
OpenGL ES 2.0 allows you to do very, very nice effects.
You can do better bump mapping, better environment mapping,
better image processing, good effects like refraction.
But OpenGL ES 2.0 is also very flexible, and
that allows you to create your own effects.
Say you want to have things look more
of a cartoony effect, you can do that.
You want to make things look more like a real life effect,
like we see on the horse here, you can do that, too.
It's up to you.
The other good side of being flexible is it helps
you tune your algorithms, select the right algorithms
so you get the right effects, and at the
same time, the right level of performance.
And the right level of performance is
especially important when you look at all
of the devices that support OpenGL ES 2.0.
So we've said this before, right, they all use
PowerVR SGX as a GPU, and devices are the iPhone 3GS,
the 3rd generation iPod touch, the iPad and the iPhone 4.
If you look at the screen sizes, you get about five times
more pixels to drive when you write an active iPad app,
about four times for pixels you want
to use native resolution of your iPhone 4.
So if you just your application that runs fine on
your iPhone 3GS and just try to scale it to the iPad
or the iPhone 4, you may end up being fill-rate bound.
So being able to tune the performances and select the
right algorithm to get to real time is extremely important.
Now you have the background.
Let's look at the basics of how you can write shader.
There's our pipeline again, and let's
look at the Vertex Shader first.
I mentioned before that Vertex Shader is used
to do your position and normal transformations,
you can also do a texture coordinate transformations.
This is typically where you would implement
lighting and also skinning, as we will see later.
There are several inputs to your vertex shaders.
The first set of inputs are called attributes.
Attributes are defined for each vertex on your object.
So things that you would define as
attributes are the position of those vertices,
extra coordinates for those vertices, normals, the color.
That's the first set of inputs.
The second set of inputs are called uniforms, and
they are constant for all the vertices on your object.
So example of uniforms, your ModelViewProjectionMatrix
that you use to go from object space into cube space,
ModelViewMatrix, the light position because your
light is going to be constant at the same location
for all of your pixels -- all of your vertices.
And the outputs of the Vertex Shader are first and most
importantly the final position of the vertex as well
as varyings that get passed to the Fragment Shader.
So example of varyings, your texture of
coordinates, final color of your vertex, the normals,
if you need any Fragment Shader, so on so forth.
Next, let's look at the Fragment Shader.
So Fragment Shader is typically
used to do mostly texture loading.
This is where for every pixel, you are going to
get the right value of the texture that you want
to have -- you want to use for this fragment.
That's why you do your texture environment.
If you want to do some per pixel optimization like
effects like fog, for example, that's where you do it.
Again, the Fragment Shader has several inputs.
It has both varyings and uniforms as inputs.
There is one pretty fine varying coming
in, which is a fragment coordinate.
This come directly from the rasterization stage.
And they are sets of varyings that you define.
Example of such varyings are normals, colors,
texture coordinates and they get interpreted
by the rasterization stage and passed from
the Vertex Shader to the Fragment Shader.
The other sets of inputs are uniforms.
Again, uniforms are going to be constant
for all of the fragments on your image.
Examples, if you do fog computation,
this is where you have your fog factor.
Which texture you need, you want to
load your textures from, and, of course,
the Fragment Shader is going to load the texture.
The Fragment Shader has one output which
is the final color of your fragment.
So now you've seen the interface of
those Fragments Shaders, Vertex Shaders.
How do you use them?
Well, you use the OpenGL API so that you
can compile, link and use those shaders.
So let's start with the Fragment -- Vertex Shader.
The way you use that is you create the shader, you
attach your source code, you compile that shader.
You use the same for all the Fragment Shader.
Now you have both compiled.
You can create your program, you attach those
two compile shaders, you can link your program,
you do all of this in the initialization
phase of your program.
The other thing in this initialization phase is
querying the locations of your attributes and uniforms.
Then in the main loop of your game, you update
-- you first say which program you want to use,
then you can update the values of those
uniforms and attributes and you can draw.
That's how you quickly write and compile, link, use shaders.
Let's take a very simple example, yet very useful.
Texture mapping.
You probably all know what texture mapping is, right?
You start with the view of the world.
In this case, it's represented as a wire frame.
It's the Quest World here, one specific room of the
Quest game, and we have a texture atlas that we use.
And the idea is to take a specific part of
this texture atlas and map it onto our world.
So, in this case, we take a piece of the world,
and we want to apply to the world in our model.
That's texture mapping.
That's how you can end up with a
world that looks slightly more real.
How does that look in a shader?
Well, here is a Vertex Shader.
It's pretty much as simple as it gets.
It gets your attributes and uniforms as
inputs, two attributes, in this case.
First, the position of each vertex, right?
The second, the texture coordinate for
each of those vertices on your objects.
The next input is the uniform.
It's the ModelViewProjectionMatrix which is used to
transform the position from object space into cube space.
And the output is going to be the final texture
coordinate that you pass through your Vertex Shader.
There is a main function inside the Vertex Shader,
and that's a program that's going
to apply to every single vertices.
First line is you do a matrix multiplication of your
ModelViewProjectionMatrix by the original position
of your vertex, and you end up with
the final position of the vertex.
And the second one is just you pass through
the texture coordinate to the Fragment Shader.
On the Fragment Shader side, we got a uniform, which is
which texture in it we want to load the texture from.
In this case, we just have one texture in it.
The texture coordinates that come from the
Vertex Shader and have been interpreted
for all the points on the wall, in this case.
And in the main function of the Fragment Shader,
we simply use the built-in function texture 2D
to load the texture for the specific texture of coordinates.
And that's how we end up with the final color of the vertex.
So you end up with a view of the world that
kind of looks real but doesn't look that good.
That is a start.
The next step is before you want the
add your character and animate it,
and you want to make this look good, make this look real.
So we are going to talk about these techniques now.
Mainly, we're going to talk about skinning
for animating the character, lighting,
different ways of lighting, and finally, shadowing.
Let's start with skinning.
So skinning -- the idea of skinning
is to model the deformation
of the skin based on the animation on a skeleton.
And the technique we are going to use here is
called smooth skinning or linear blend skinning
because we use linear interpolation to do skinning.
It's a pretty simple technique,
and yet, it looks pretty good.
So the way you do skinning is you start by a
skeleton, which is a hierarchy of bones and joints,
right, each bone is connected with a joint.
And that's how you animate your
character, you animate the skeleton.
That's the animation we use in this case.
The next step is to bind a skin mesh on top of
the skeleton so it starts looking more real.
And for each of the vertices on the skin mesh, we
are going to bind it to one or more bones or joints.
So let's look into the arm of our hero here.
You can see here the different vertices on
this arm, and we are going to especially look
at one vertex, which is on the elbow, this vertex.
And it's bound to two bones; the
upper arm and the lower arm.
And here, we represent it as being
bound to the joint, not the bone itself.
And there are two definitions that we are going to put.
The first one is a definition of weight.
So weight represents the influence
of a given bone or a given joint
on that vertex position, on that point on the skin.
So, in this case, we're on the elbow, which is right
in the middle between the upper arm and the lower arm.
So the weights are just 0.5 for each bone.
The second definition is for skinning matrices.
And what the skinning matrix does is it combines the
transformation of the bone, so the animation of the bone,
with the position of the vertex, position
of the skin with respect to that bone.
So here, we move a little bit the lower arm and
you can see we define two points; P1 and P2.
P1 corresponds to the position of the points,
the position on the vertex on the skin
if it were only attached to the upper arm.
So, in this case, it hasn't moved.
P2 corresponds to the position of the skin
if it were only attached to the lower arm.
And the way we get those two positions is
by simply multiplying the skinning matrix
by the original position of the skin.
So we end up with two points; P1 and P2.
Next step is to get the final position
so we can move the arm a little bit more,
and the next step is to get the final position of the skin.
And the way we do that is by a simple interpolation.
We multiply P1 and P2 by their respective weights,
sum, and we end up with the final position of the skin.
That's how we animate the different points of the skin mesh.
So we apply that for all the vertices on the body.
Once we've done that, we can do our usual texture mapping.
And by animating the skeleton, the underlying
skeleton, we can animate the whole character.
That's skinning; that's how skinning works.
How is it implemented in a shader?
Let's first look at this shader, which is what you
would find if you look on the web for a skinning shader.
It looks pretty complex, right?
You have four loop there, you get a branch.
We can probably do better, especially
from better devices, right?
How about that?
Let's assume that every vertex, every
point of the skin is only attached,
is always attached and only attached to two bones.
This way we don't -- we can unroll the loop and we
don't need to check whether or not a given bone --
a given vertex is attached to a bone or not.
We end up with this code that's pretty simple,
and that looks like what we had in the algorithm.
So let's now go through it quickly.
First we have a set of attributes, right, the
position of each vertex on the skin, then the --
which joints are attached to that specific
vertex, right, a joint 1, a joint 2,
then what are the respective weights for those two joints?
That's our attributes.
Then the uniforms are the skinning
matrices for each of those joints,
so they correspond to the animation of the skeleton.
And our usual ModelViewProjectionMatrix.
The main program for the Vertex Shader is --
well, you do your matrix multiplications
of the skinning matrices by the positions.
You end up with the two points I mentioned before, right, P1
and P2, and then we simply multiply P1 and P2 by the weights
of those two positions of the -- that corresponds to
those two bones and end up with the final position.
Then we do our ModelViewProjectionMatrix to go into iSpace
and we end up with the final position of the vertex.
We're going to do texture mapping
here, so we pass, as we did before,
the texture of coordinates to the Fragment Shader.
This case, the Fragment Shader is the same as the
one we saw before, so it's just doing a texture loop.
That's how we implement skinning, and it's pretty efficient.
So here I have my iPhone and it starts my application.
We have two characters here that
are animating and are being skinned,
so you can see they look like they
have kind of a real animation.
And you may think it walks a little bit funny because
we actually use the animation and in the Quest game,
it's carrying a big sword, so it's pretty heavy.
That's why the arm is always at the same location.
Now, I talked about performance and
performance being very important.
Let's bring up instruments and see how we are doing.
We can switch to the demo machine.
All right.
So I'm bringing up instruments, and I'm going
to -- since I'm doing an OpenGL ES application,
I'm going to set up the OpenGL instruments.
Today, I'm not very interested in getting the samples.
I'm mostly interested by what's happening on the GPU and
especially, I want to look at the utilization inside my GPU.
So there are two interesting statistics we use here.
The first one is the render utilization and that's
includes how much time you spend in your Fragment --
the time that's spent in your Fragment Shader.
Title utilization is the other side and includes the time
-- important time that you spend in your Vertex Shader.
So if you want to minimize what's going
on in your GPU, try to minimize these two.
And then we just want to make sure we are at 65 per second.
Let's mix in some bigger too.
And we have our app running.
We are going to attach to it and see how it's performing.
And so we can see that title utilization here is 17, 18%.
And in this case, I'm using the textbook
implementation that I showed you first.
Let's switch to the other implementation,
the optimize implementation that we have.
So you can see right away, right, the title
utilization goes down from about 17, 18% to now 12, 11%.
You know, for just a few lines of
code changes, that's 30% improvement.
And you may say, oh, well, you know, 18%, that's
pretty good all right because I'm not using 100%.
That was just for two characters, and there was
nothing going on in the world around it, right?
So whatever you can save on that, it's more cycles that
you can use to do better effects on your characters
and you have more vertices and better
effects you can do on your work.
So 30% is important in this case.
So that's what I wanted to show
you and tell you about skinning.
Next, Michael is going to tell you everything about
lighting and how to make all this more real using shadows.
Michael.
[ Applause ]
>> Michael Swift: Thank you, Luc.
So as Luc mentioned, my name is Michael
Swift, and I get to talk to you about lighting
and shadowing for the remainder of this session.
So let's start off with lighting.
So what you see up here is our unlit world, just
static and plain, just a simple texture fetch
that Luc showed you earlier, and I want to use this as
our starting point and turn it into something that looks
like this, which is a fully lit environment with
a shadow character which will animate and skin
and have all the lighting effects
applied to him as well as the world.
So how do we get there?
Well, it's important to know that
light contribution is determined
by three main factors, the first of which is distance.
And as the character gets further away from
the light, there's less light contribution.
The second is direction.
So the light is going to be pointed at the
geometry and as long as the light is --
or as long as the geometry is facing the
light, it will have full light contribution.
And as it is facing away, it will have less
and less of that light applied to the vertex.
And the third part is actually occlusion.
So if there's an object in between
the light and the character,
then the character will not have any light applied to them.
So those are the three factors that determine how
much light is applied to our hero in this scenario.
So to help make this make more sense, we're
actually going to break down our world
and our character into two types of content.
We have static content, which is our
environment, and we have dynamic content,
which is our skinned hero who's
moving around the environment.
So, taking that as a baseline, we're going
to start off with the OpenGL light model.
So many of you have used OpenGL ES 1.1,
and this is roughly comparable to that.
And it accounts for distance.
This is the linear light attenuation modes
and factors and also accounts for direction.
And this is -- we'll use the light vector, and
the geometry normals to compute a dot product.
And the nice thing about this is it works for all lights.
So you can have static lights, dynamic lights.
And it works for all content.
It'll work for both dynamic content and static content.
So let's take a look at how that is in the world.
And you can see it's a visual improvement
over that simple unlit environment.
And so how do we actually do this?
Well, we have our Vertex Shader,
and it has a series of inputs.
We have our normal and we have our vertex position.
And these are passed in per vertex.
And then we also have a series of uniforms.
We have our light location, we have our light color, we
have the attenuation falloff factor that we want to use.
In this case, it is the linear attenuation factor.
And lastly, we have our ModelViewProjectionMatrix.
And the result of all of this is
going to be a light color per vertex.
So let's take a closer look at the main body of the code.
So the first thing that happens is we need to transform
our incoming geometry to create the vertex in clip space.
And the second part is the linear attenuation part,
and this accounts for the distance contribution
that I was talking about in the first few slides.
And so we use our light location and
the incoming light -- oh, sorry --
the incoming vertex location to compute a length
and then use that to figure out how much we want
that light's contribution to fall off as it
gets further and further away from the light.
And also, we'll create a direction factor.
So we mentioned earlier that we're going to use a dot
product to vary the amount of light that's being applied
for each vertex depending on whether or
not it's facing the light or facing away.
And so here, since we need the vector, which was
originally in world space to actually be normalized
so we can have a same result from our dot product.
And we assume our normal is already in unit space as well.
And so we'll create the dot product
and clamp that to the range 0 to 1.
And then finally, we will create our final color by
using the original light color, the direction factor
and the attenuation factor all multiplied together.
So why would you want to use this?
Well, it's fairly straightforward.
It's very similar to what you are already using,
OpenGL ES 1.1, but it is computationally expensive.
There's a lot of math, and if you want to implement
the full light model, which is ambient color
and specular color, it takes more and more and cycles.
And you have to compute this for
each frame, so it's highly expensive.
But it is an improvement over having no lighting,
and it does account for both direction and distance.
But it has no way of knowing about
other objects in the world,
and so there's no light contribution
for geometry that is occluded.
And that brings us to prebaked content.
So what's real important here is we can simplify our scene
to deal with just the static lights and a static geometry,
and this allows us to precompute the light
contributions for the world in an offline pass.
And this makes a lot of sense because all of
that computation we did in the Vertex Shader
and in the Fragment Shader, we're actually going to
hoist out, and it'll make things run a lot faster.
And so you can create these lightmaps, and you
can atlas them as part as an optimization phase
to make your game and your content run really fast.
And it accounts for all of the distance
and direction and occlusion information
of the static lights and the static geometry.
So, let's look at the first class of these.
And specifically, it's the per-object lightmaps.
These are only for static geometry.
So what we actually do is in your 3D modeling
software or your level editor environment,
you actually use like a radiosity, so
it's like a nice soft shadowing algorithm
or a direct illumination algorithm, so some hard shadows.
And then prebake all this light contribution and create
an atlas like you see on the lower left-hand side
which we can then use to draw the entire world.
And on the lower right, you can see that atlas being
applied to the first room in the Quest environment.
So let's take a closer look at that.
So we have our lightmap which is -- which was
created for each piece of geometry in the world,
and then we can multiply that against our original diffused
texture, and we get something that looks like this.
So it's a lot more interesting than what we got out of the
OpenGL ES light model that you saw on the first few slides.
So how do we actually make this work?
Well, the only thing that's different
here is the Fragment Shader.
The Vertex Shader is just like
the ones we've showed you earlier
which is just passing then the texture
coordinate and transforming the incoming geometry.
So in the Fragment Shader, we have samplers -- oh --
we have two samplers; one for the lightmap
and one for the diffused texture map.
And we have two sets of texture coordinates
because the -- they're atlased separately.
And we do the two sample operations
and then multiply them together.
So very simple, very straight forward,
very efficient on this hardware.
So what I've been talking about is working
with static lights and static geometry.
And now we're going to take a look at
how you can deal with static lights
with dynamic geometry because it's a little bit different.
You can't know during your offline phase
where your dynamic geometry is going to be
and how much light contribution it will ultimately have.
And so you cannot account for direction because you
don't know if the content that's moving is going
to face the light, face away from the light or
be somewhere else or be animating or skinned.
You have no idea.
So you can't use those per-object lightmaps.
Instead, you get to approximate
it by using world space lightmaps.
And for our Quest example, we actually
are doing this all in 2.5D.
The game itself was constructed such that we could
use just a single top-down lightmap that is --
that has all the light contribution
and -- I'll move it to the side --
and apply that to the actual character
as he moves around the environment.
And we do this by having a reference point.
We need to know how to get our X, Y,
Zs into UV texture coordinate space.
And by having this reference point,
we can actually make that possible.
So we'll take our character, and
we'll transform him or flatten him
down to a specific part of the world as he moves around.
And so here are two screen shots.
Let's zoom in on the one on the left.
And as you can see, on the left-hand side of the
hero, he's lit with a nice, bright, white light.
There's less light contribution on his right-hand side.
And then on the second image, the hero is actually
in front of it of a grate and so the geometry that is
of the grate itself is hiding the red light that's behind
the grate, so you have him lit with the dark red on top
and on bottom, but there's less
light contribution in the middle.
So looking at the shaders themselves.
So we're going to start with the skinning
shader that Luc showed you earlier,
and we're going to add in a couple of new pieces.
So, as I mentioned before, what we're doing is we're
transforming our X, Y and Zs into the UV coordinate space,
and we can do that using a simple matrix multiplication.
And so, our new input is a lightmap projection matrix,
and then we're going to create a
set of lightmap UVs as the output.
And so this is one extra line of code, which is
simply the matrix multiplication of those two values,
of the matrix with the postskinned location of the geometry.
So these are great.
These work really fast.
You only need one top-down lightmap.
It's very straightforward, and
you don't have to add much code.
However, this does not handle the direction contribution
as I mentioned in the first couple of slides of this.
And so you can get some weird artifacts.
So, specifically, we have our hero,
he's standing on top of the grate,
and there's actually a light underneath the grate.
And so you can see the striated, dark and light shadows on
top of the character, and that's not what you really want.
You want something that's actually correct.
So how do we make that happen?
Well, we talked earlier about the OpenGL ES lighting
model, and we're actually going to use part of that.
We're going to use that -- the normal and the light
vector and this dot product to figure out whether
or not the character is facing that
light or facing away from that light,
and then we're going to still use these world
space lightmaps for each one of the lights.
But the difference here is instead of just a single world
space lightmap, we're going to have a whole slew of them,
and you're going to change through
them as you move in your environment.
And then it's the same straightforward
process you saw earlier.
We're going to multiply the two of them together.
So, how does this actually look?
So let's first take a look at the lightmaps themselves.
And so we have our first group -- our first
light, second light and the third light.
And so we're going to constrain our code to what would
just be three lights, at least in the first room,
so we can keep the greatest contribution
as the character moves around the world.
And the results of those three you can see here.
So the first light creates shadows on the right-hand side
of the character, and the second light will create shadows
on the left-hand side of the character, and then finally,
the third light, which is actually behind the character
and higher up, will have a small amount of light
contribution on the shoulders and the head.
So once you put this altogether, you
get something that looks like this.
And it's actually fairly interesting.
We're going to go into a demo in a few
minutes and actually see this in action.
It's a lot more nuanced than what you see here.
So how do we make this happen?
Well, we're going to start out with
our Vertex Shader just like before,
and it's slightly larger than you would like, probably.
But there we go.
So we're start off with our skinning
part that we mentioned earlier.
And the next part was that lightmap UVs
from world space X, Y, Z transformation.
And what's new is just like skinning the geometry
locations, we also need to skin the normals
because as the character animates, we need the make
sure the normals are also appropriately changed
as the animation happens.
So same process and what's different
here is the normals are transformed,
not by the ModelViewProjectionMatrix,
but by just the model view matrix.
And then we're actually going to do
that dot product we mentioned earlier.
So for each one of the lights, we have three lights,
we're going to create the light vector and then dot
that light vector, and we're packing here in the first
or X part of the vec3 varying light
factor, the second and then the third.
So this is how we put in each of the
three light contributions and make sure
that we can subsequently read them in the Fragment
Shader, which we're going to jump to next.
So, the Fragment Shader has the original diffused texture
and our three lightmap textures that we talked about
and has our lightmap UVs, our diffuse UVs and it has
this new light factor into which we packed the N.L test.
And we're just going to sum all
those lights contributions together.
We're going to do a texture fetch and
then multiply it by the light factor,
texture fetch the diffuse color,
and lastly, multiply it all.
So that creates the final result.
So, what's interesting about this is this is a lot more
work, and this will solve those artifacts we talked
about earlier, but they could also be -- those
same artifacts could be solved by some tricks
and things while you're exporting
your lights and your content.
And for a game like Quest, they chose to modify their
assets rather than implement a more complete algorithm
like this because there is an increased cost.
And as a result, this kind of algorithm is more suited
for a first person shooter where you're really in close
and can see all the various light changes on
the character as he moves and animates and dies
and falls over and all those kind of fun stuff.
So, this gives you a really nice increase in quality, but
it may be really subtle based on where your camera is.
And so you might -- you need to do some tradeoffs
to figure out which algorithms are appropriate
for the kind of content that you're working on.
But the advantage of it is it will fully
account for your distance, your direction
and your occlusion information of the static geometry.
So let's hop into a demo.
There's the skin you saw earlier.
There is the optimized skinning.
As you can see, there's no difference.
And so here we have our character
with the OpenGL light model,
and it's running on a nice fluid
65ths, nice and smooth, as you can see.
Let's zoom back in.
And you can see the character has a little
bit of change of lighting as he moves around,
but it's fairly straightforward and kind of plain.
And this is what you can do right now in OpenGL ES 1.1.
And so we're actually showing the prebaked world lighting.
And you get a lot more nuanced colors and shadows
and you can -- it actually runs a lot faster.
If we were to look at instruments, the utilization
is probably about a third of that previous frame.
And so as you can see, as the character moves around,
there's different moves amounts of
light being applied to the character.
It would be most noticeable as he goes
on the right-hand side of the screen.
So this is just the single top-down
lightmap, and its looks pretty good.
So let's jump back to the main room with the same lighting.
And it's kind of interesting; it's not great.
The interesting part about this screen, though, is that
the grate is exactly in the center of those three lines,
so it kind of has a very even light contribution.
But when we add the direction test
and use those per-light lightmaps,
we get a much more nuanced kind of lighting effect.
As you can see, there's different parts of him
that get darker and brighter as he moves around
and just a lot more interesting effects, and
it becomes more noticeable as you're zoomed in.
If you're zoomed all the way out, you can
kind of tell, but it's not -- it doesn't --
not quite as different from the
previous single top-down lightmap.
So in summary, there's a whole bunch of
choices of what you can do with lighting.
There's the OpenGL light model that you saw which
works for static lights, dynamics lights and static
and dynamic geometry, but it's kind of expensive, and
it's hard to make it run really efficiently on the GPU
and also these other methods I just
showed you actually look nicer.
And so there's the per-object lightmaps that we mentioned
which is solely for static lights and static geometry.
And then you have these tradeoffs of performance versus
quality for using these top-down lightmaps or any other kind
of world space lightmaps based on your content.
And that brings us to shadowing because thus far,
we've talked about how do you deal with static lights
and their light contribution on dynamic geometry.
Now we want to talk about how you do you handle the dynamic
geometry actually shadowing both the environment and itself?
And we're going to focus today on shadow volumes.
And we're doing this because they work well with
the per-light lightmaps that we mentioned earlier,
and they can do a per pixel test whether
something is shadowed or something is lit.
And it's also one of the full shadowing solutions
that will implement self-shadowing and
shadowing of the rest of the world.
So let's take a closer look on this.
And specifically, we want to talk
about how do you count shadows?
Well, we have these three pieces of geometry, A, B
and C, all of which are casting shadows on the things
that are on the opposite side of the light.
And what we want to do is we have a camera
and he's going to actually look through,
and we want to figure out for a given point on
that line, how many shadows are we inside of?
Now you can see, on the left-hand side it's 0,
as it enters A, it becomes 1, and then it exits A
and goes back to 0 and so on and so on and so on.
And this is actually really important because we can
use that entering and exiting information to figure
out is something shadowed or is something lit?
So let's do a brief overview of how all this works.
And you start out by you render your world.
In this case, you want to render
our ambient light contribution.
This sets up our color and also sets up our depth
information which is critical for the counting algorithm.
And then for each light, we want
silhouette or geometry with --
or from the light's perspective effectively creating that
shadow as a volume and this creates just the front edge.
Which we want to then extrude that out to infinity.
This, in fact, casts the shadow.
And once we have these pieces of
geometry, we just have to stencil buffer
and we render the volume from the original camera's view.
And what's important here is we're actually
counting how many volumes have we entered
and exited before we hit the first thing in our scene.
Because we want to stop as soon as we have a matching
depth so we can accurately say for this location
on the screen, was this fragment shadowed or lit?
And then we're going to use that count information and
specifically get the stencil to 0 then it's fully lit.
And if it's non-0, then it's shadowed.
And so we're going to render our world through that
stencil test and then add that light contribution.
So we're going to do this for each one of the three lights,
and then we're going to sum all of
these to create our final image.
Let's take a closer look at this.
So when we're generating the shadow volumes, as I mentioned
before, we need to do this silhouette determination.
This breaks down to two key things.
What we need to do is we need to figure out what edges
or what geometry are on the edge of the silhouette.
And we do this by using the dot product
test that I mentioned so we can tell
if the geometry is facing the light and facing away.
And it's actually when there's a shared edge between
two triangles that face both towards the light
and away from the light that we can
actually say this edge is in the silhouette.
So once we have our silhouette, we can then extrude it.
And we're going to do this by taking our
geometry, copying it, and then setting the W to 0.
Now, this is on a quick little hack because
the ModelViewProjectionMatrix that we said
that can be equal to 0, it goes off to infinity.
So this works great.
So once we have our volumes, we then need
to set up the stencil buffer or count them.
And as I mentioned earlier, it's only when we pass the depth
test or in front of the fragment that we want to shade it --
that we want to test the volumes against
that we're going to increment the stencil
when we enter a volume and decrement
when we exit the volume.
Now the good news is OpenGL ES 2.0 as part of the
core spec supports the increment WRAP extension,
and you can set them both separately.
So instead of having to do two passes of your
silhouette geometry, you can instead use one
and then have both a front test and a back test
allowing this to happen in just a single pass.
So that's great.
This also works in OpenGL ES 1.1 through some extensions.
So, once your stencil buffer is set up, you
then want to test is it lit or is it shadowed?
And so when we actually do this test, we're going
to do the stencil test of equal as the operator
and the value of 0 because 0 we chose to be fully lit.
And then our depth test once again to be equal
so we only draw the pixels where we've matched,
and then we add in that light contribution by
drawing our world and the skinned characters.
So, let's take a look at the volumes.
So, here we have three lights which are casting white light,
but we've chosen to color them as red, green and blue.
And the first volume here is the
red volume cast by the red light.
And the next one is the green one
from the green-colored light.
And lastly, the top-down blue light.
So let's get a little closer look at this.
So we have our three volumes that have been
extruded, and we're showing here in the world,
and then we're going to turn those into actual shadows.
So I'm going to zoom in on this, and you can see you
get really nice, crisp per pixel shadow versus lit.
And it looks really good.
And happens to work fairly well on the PowerVR SGX.
So that's how all that works.
And it changes only a small amount when you're working with
prebaked lighting because we want to take all these things
and mix them altogether to get a
really, really good visual result.
And so the first thing we need to do is we need
to draw our world with the full prebaked light.
So that was basically what we saw
at the end of the lighting section.
And then we're just going to silhouette
our geometry, extrude it,
and we're going the set up sensor
buffer exactly the same way.
But instead, we're going to add a little twist here.
Instead of adding light contribution, since
the prebaked lighting already counted for all
of the light's contributions, instead we're going to say
if we are in shadow, then remove that light's contribution.
So by doing this, we're all going to incur the cost of
removing the shadows, and that's a lot less fill rate.
So, once again, we have our environment
with the three volumes visualized.
Zoom in on that real quick.
And you can turn these into actual shadows.
Now what you'll notice is the shadows are very
subtle and this is because what we subtracted
out were those individual lightmaps
for each one of the lights.
And as they get -- as the character and shadows get further
away from those lights, there's less and less contribution.
Let's watch a quick little zoomed in movie of this.
As you can the character is moving
around and you can get some nice shadows
and they're casting all their world geometry.
And then finally, you can see on the lower right-hand
side, that shadow is cast from the right-hand side
up towards the upper left, has a really strong
shadow initially, and then it fades off nicely.
So this is kind of a way you can kind of mix
kind of soft shadows with the hard shadows
of stencil volumes by using these prebaked lightmaps.
So there's a couple of things to know about this.
It is expensive.
But by using the prebaked lighting, we only incur the cost
for when things are shadowed and not when things are lit
because as you noticed in our environment, there's
a lot less things that are shadowed than lit.
So we saved all of that fill rate.
And shadow volumes are quite large, and they have an extra
cost both on the vertex processing in the shader stage
as well as when you are actually tiling this
because this is a tile-based architecture.
And we can make some of this tricks and hacks
and we can use the prebaked lights specifically
because the lights themselves contain the occlusion
information of the static geometry which means you only have
to deal with the contribution of the dynamic geometry.
So, in this case, our environment is 50, 60.000
tries, and our skinned character is only about 1,700.
So those are vast orders of magnitude that you
won't have to generate silhouettes and volumes
because there's a small amount
relative to the rest of the environment.
And the best news is depth testing and
stencil testing is really efficient on SGX.
And so you can do all of this in just a single pass
unlike some other algorithms you can do for shadowing.
So let's hop into a demo and see how this looks in action.
So here we were with our lightmaps.
We topped our lightmaps with direction
and let's jump to the next one.
So here are our visualized shadow volumes, as you
can see, if I zoom out, they do extend to infinity,
and you have our blue light which is in the upper --
the top and behind, hence the shadow is cast
very short and the red light and green light.
So zooming and then switching to the actual shadows,
you can see how they're nice and sharp and crisp,
and it's all really fluid as the character zooms around.
So moving out of the sky box environment,
we actually put him in the world.
And so, as we zoom in, you can see just the same effects
of the directional lightmaps being applied to the character
and also the shadows animating and falling off as he
gets closer and further away from the light sources.
And if we zoom out, we can actually get
-- you can see the shadows being cast
on the various pieces of geometry in the world correctly.
So those are shadow volumes.
[ Applause ]
And so, all of that is happening
in real time on the new iPhone 4.
So you can do the same algorithms and you can save all
of that fragment cost by switching your algorithms around
and removing the light contribution
as opposed to adding it in.
It saves a lot of fill rate.
And so, here was our agenda for today,
and let's just jump into quick summary.
So the key to all of this is OpenGL ES 2 allows you to
choose the algorithms that are right for your content.
You can choose algorithms that allows you to
do a rough approximation that's really fast
if you are zoomed far away from
your skinned and animated content.
Or if you want to be up close for like a first
person shooter, you can choose higher quality ones.
And you can make those tradeoffs, and you need to
choose which ones are best suited for your content.
And the best part is since you can do all
these tricks and change the math and like --
and prebake all this stuff, it allows you to simplify
your GLSL in your Fragment Shaders to just a few lines
or to just -- or to use more expensive shaders
just for smaller amounts of dynamic content.
And so you can optimize both for performance as
well as optimizing for the visual quality based
on where your camera is and how your world is constructed.
And you can hoist it all out to a preprocessing stage
and make your games that much more visually interesting
and just really fun to watch and like -- and
see everything nice and smooth and run at 65ths.
So, please contact Allan Schaffer if you have any questions.
His contact information is here.
There's also the Apple Developer Forums.
Many of us on the OpenGL and Driver
teams are on there making comments
and trying to help out anyone who has questions.