Transcript
[ Music ]
[ Applause ]
>> Hi, everybody.
My name is Jaap van Muijden and
welcome to this session on
Modern Rendering with Metal.
In the first part of the
session, I will go over some of
the more advanced rendering
techniques that you can use in
your apps today.
Then my colleague, Srinivas
Dasari, will talk to you about
moving your CPU render loop to a
more GPU-driven pipeline.
We'll end the session by showing
you how we can use our new GPU
families to easily write cross
spectrum code.
Whether you're starting from
scratch or you want to improve
your existing Metal app, or you
have an amazing rendering engine
that you want to move onto the
Metal platform, we'll show you
how you can make the best use of
the available hardware with the
rendering technique that fits
your needs.
We'll start by taking a look at
some of the range of rendering
techniques that are used by
games and apps today.
We'll start with basic deferred
rendering.
This is the most commonly used
rendering technique used by
games and graphical apps on all
platforms.
We'll discuss the classic
two-pass setup.
We will show you how to
implement this in Metal and how
to optimize this for the iOS
platform.
We'll then move on to tiled
deferred, which extends the
lighting pass of deferred
rendering and is perfect if your
art direction requires you to
have complex lighting setups.
We'll then take a look at
forward rendering.
That's a really good alternative
for the Metal apps that require
complex materials,
anti-aliasing, transparency, or
special performance
considerations.
Our last technique we're going
to talk about is visibility
buffer rendering, which defers
the geometry logic all the way
back to the lighting pass and
now in Metal 3, is easier to
implement than ever.
But before we get there, let's
start with deferred rendering.
Deferred rendering splits the
rendering of your scene up into
two passes.
There's the geometry pass, where
you basically render your entire
scene into an intermediate
geometry, or g-buffer.
And the textures in this buffer,
all the normal, the albedo, the
roughness, and any kind of
surface or material property
that you need in your writing
model or your postdressing
pipeline.
Then in the second pass, the
lighting pass renders the light
volumes of your scene and builds
up the final lit scene in
accumulation texture.
The deferred light shaders were
bind all the textures in your
G-buffer to calculate their
contribution to the final lit
surface color.
So, let's define the data flow
of this technique and then move
onto a Metal implementation.
So, here we have our two render
passes, and we'll be running
these two consecutively on the
GPU.
In our geometry pass, we need to
write out depth.
The depth is used to do depth
calling during your geometry
pass, but it's also used to
calculate the pixel location and
world space for your lighting
pass.
And we also output our G-buffer
textures.
In our example here, we'll use
normal, albedo, and roughness
textures.
Then in our second pass, the
lighting pass, we read back the
G-buffer textures and then we
draw light volumes and
accumulate them in our output
texture.
So, let's see how we can
construct this data flow in
Metal.
To set up a render pass in
Metal, you have to start with a
Render Pass descriptor.
The most important part of a
Render Pass descriptor are its
outputs.
In Metal, these are defined with
attachments.
Every Render Pass can have
single depth attachments and
multiple color attachments.
For every attachment, we have to
define the texture which points
to the data that stores our
attachment data.
We have to define our load
action which shows us how to
load the existing data from the
texture and the store action is
how to store the results of your
rendering back into the texture.
When you've defined these
properties of all your
attachments, you can then create
your Render command encoder that
you can then use to finally draw
your objection to your Render
Pass.
Now, let's see how we build this
in Metal, starting with our
setup code.
So, here we have our setup
function.
We'll start by creating a Render
Pass descriptor.
And now we just start filling in
all these attachments.
So, we'll start with the depth
attachment.
And since we're using the depth
attachment to do our depth
calling, we need to make sure
it's clear before we start
rendering our scene.
So, we set our load action to
clear.
Of course we want to store depth
for the second pass.
So, we set R, store action to
store.
Now, we move onto our color
attachments.
The color attachments, we need
one for every texture in our
G-buffer.
And because all these textures
are going to be handled kind of
the same way, we'll just show
you the albedo.
Because we're probably going to
be using like a skybox or a
background during rendering, so,
we're sure that we're going to
be overriding every pixel in our
frame, every frame.
Which means we don't really care
about any previous values in our
G-buffer textures.
So, we can set our load action
to dontcare.
Of course, we want to store the
results of our G-buffer, so we
set our store action to store.
Now we can start with our
lighting pass descriptor.
We create another descriptor
object, and then we defined
attachment for accumulation
buffer.
Since we're accumulating data,
we need to clear it before
starting so we put our load
action to clear.
And of course we want to save
our final image, so our store
action's going to be store.
Now, let's look at render loop
while we're using these Render
Passes to actually draw our
scene.
So, we'll start again with our
geometry pass.
We create our Render command
encoder using our descriptor.
And then we start just iterating
over all the measures in our
scene.
This is a very simple way of
rendering your scene.
And my colleague, Srinivas, will
show you in the second part of
this session how you can move
this basic CPU render loop into
a more GPU-driven pipeline with
all kinds of calling and LOD
selection.
Okay, so now we've encoded our
entire geometry buffer.
Sorry, our entire geometry pass,
and we move on to our lighting
pass.
We create another render command
encoder.
And we now we start iterating
over all the lights for our
lighting pass.
And every light, every deferred
light shader will bind those
textures from the G-buffer to
calculate its final light
colors.
Well, this two-pass system works
perfectly fine on macOS and iOS
across all platforms.
It's a really good fit for all
types of hardware.
But there are some things that
we can do to further optimize
our implementation on iOS.
So, let's step back from the
code back into our overview.
You can see there's this big
buffer between our two render
passes.
The geometry pass is storing all
its data into these G-buffer
textures.
And then the lighting pass is
bringing them all back.
And if we're having multiple
lights shining on a single
pixel, we're doing this readback
multiple times.
Using a technique called
programmable blending in Metal,
we can get rid of this
intermediate load store into
device memory by leveraging the
taut architecture of iOS
devices.
So, how do we take advantage of
this technique?
Well, to enable programmable
blending, we merge the geometry
and the lighting pass, and
create a single render encoder
for both Geometry and light
rules.
So, due to the nature of iOS
architecture, the attachments
are kept resident in tile memory
for the entire duration of our
encoder.
This means we can't only write
to our attachments but we can
actually read them back.
We can read back the values of
the same pixel we're writing,
and this is exactly what we
want.
We want, when we're calculating
the light in our lighting pass,
we want to retrieve the written
G-buffer attachments of the same
pixel.
So, let's see how this will
affect our light shaders.
So, here we have a light
fragment shader from our
lighting pass.
And as you might know, you just
start by binding all the
textures that you need to get
your G-buffer data.
And then you have to actually
read all these textures across
all your G-buffer textures to
get all the material and surface
information.
Only then can you push this into
your lighting model to get your
final vid color.
Now, let's see what happens if
we use programmable blending.
Instead of binding all the
textures, we simply bind all the
color attachments.
And now we can directly use
these values in our lighting
model.
As you can see, we've created a
new linear depth color
attachment for our G-buffer and
this is because when you're
using programmable blending, you
cannot access the depth
attachments.
So, now that we're no longer
binding or sampling any
textures, let's see how we can
use this to further optimize our
memory layout.
When using programmable
blending, we're no longer
writing or reading from the
G-buffer textures.
So, we can put the store action
of our color attachments to
dontcare.
Okay. So, this solves our
bandwith problem, but we still
have these Metal texture objects
taking up space in our device.
And we need to tell Metal that
we no longer need any physical
memory for our G-buffer
textures.
And we do this by setting the
storage mode of the texture to
memoryless and we tell Metal
that we're no longer going to
basically be performing any
store actions on it, so we don't
need to actually allocate the
memory.
With these steps, we've now end
up with an iOS implementation
that has all the benefits but
without any of the memory or
bandwith overhead of a G-buffer.
Before I move on to tile
deferred, let's summarize.
The separation of the geometry
and lighting pass makes this a
very versatile technique.
It handles both complex geometry
and lighting very well.
And a G-buffer can be used to
facilitate a really deep
postprocessing pipeline.
And an entire pipeline can be
put in line using this
programmable blending method.
On macOS, you still have to deal
with a G-buffer, memory, and
bandwith overhead.
So, now let's move on to tiled
lighting solution, which is
ideal for those of you who want
to render maximum amount of
light but you still want to
reduce your light pass overhead.
The tiled deferred rendering
technique tries to solve the
performance issues of rendering
large amounts of lights.
In classic deferred, we render
every light separately, and this
causes a lot of the G-buffer
overhead for overlapping lights.
Tile deferred rendering extends
the lighting with an additional
compute prepass that allows our
shading to happen not on a per
light level but on a per tile
level.
The prepass first divides our
screen into a 2D grid of
lighting tiles and generates a
light list per tile.
Then in a second step, the
lighting step itself, these
lights are then used to
efficiently light the tile using
a single light fragment shader,
but it's ranging over the lights
in the light list.
Before we dive into the
implementation details, let's
take a quick look at how these
light lists are being generated.
Well, we first split up our view
frustums into these small
subfrustums, one for each tile.
Then we use our compute shader
to further fit down the
subfrustums using the location
of our tile, but also the depth
bounds of the tile.
And we can do this because we've
already ran the geometry pass.
So, the depth buffer is already
populated.
So, when we fitted down these
subfrustums, we can then just
test all the frustums against
the light volumes and add any
intersections to our light list.
This entire process can be done
in parallel across al the tiles
and is a perfect fit for a
compute kernel.
So, how do we integrate this
into our deferred pipeline we
showed before?
Well, before we had this
two-pass deferred setup.
And now we've added this compute
pass to the middle of it.
And that will create in the
light list for us and we need to
store these light lists in a
light list buffer to be stored
in device memory.
So, again, this solution works
fine on all platforms and it
simply requires us to create
additional compute and to move
our lighting logic from a single
light per shader to an iterative
loop in our lighting shader.
Just like with the previous
rendering, we can now take
advantage of the hardware tiling
on iOS to further optimize this.
So, let's take a look of how
this fits in our single encoder
iOS implementation.
So, here we have our single pass
solution we showed before, and
we need to get this computer in
there, but we need to stay
within the single render command
encoder to use the programmable
blending.
Metal provides an efficient way
of using tile-based hardware
architecture to render compute
work on each tile that we're
restorizing.
For this purpose on iOS, the
render command encoder can
encode tiler shader pipelines to
run the compute functions.
And this is a perfect fit for
outside lighting because now we
can take our lighting tile
concept and map that directly on
our hardware tiles.
So, now that our light calling
prepass can be run directly on
our hardware tile, we can use a
second Metal feature called
persistent thread group memory
to store the resulting light
list alongside our attachments
in tile memory.
Which then can be read back,
similarly to the attachments but
all the draws in the render
command encoder.
Which in our case is going to be
our per light draws.
We now move our lighting back
end to execute in line with our
graphics and completely within
tile memory.
So, what does this look like in
Metal?
Let's move back to our setup
code.
Creating a tile shader is very
similar to setting up a normal
render pipeline state.
We create our descriptor.
We set up all our color
attachments.
We then set up our compute
function we want to execute.
And we simply create our
pipeline state.
Because we are using the
precision thread group memory,
we need to reserve a little bit
of memory in our tile.
So, we go back to our render
pass descriptor and then we
simply reserve just enough data
to store our light list.
Now, let's move on to the render
loop to see the dispatch size of
these things.
So, our render loop starts this
time with a single render
command encoder.
And we again loop over all the
meshes in our scene.
And then instead of directly
going to the lighting pass, we
first have to execute our tile
shader.
So, we set up our pipeline
state, we set a buffer that
holds all the lights within our
scene, and then we bind the
thread group memory buffer into
our tile memory.
And then we simply dispatch our
tile shader.
Now that we've executed our tile
shader, the thread group memory
we hold, the light list, we can
then use in the lighting draw
where we can have every pixel
having access to its tiles light
list using the persistent thread
group memory and now very
efficiently can shade its
pixels.
Now finally, after all is set
up, let's see what this looks
like in our shaders.
So, here we have two shaders.
The top one is the tile shader.
It's binding the output light
list into a persistent thread
group memory buffer.
Then it basically loops over all
the lights in some way and it
outputs the light mask into the
persistent thread group memory,
which can then be read back by
the second shader, our actual
lighting shader.
And then it's right over all the
visible lights within its tile
and shade our pixels.
Now that we've seen all these
key points of implementing the
tiled lighting technique for our
tile deferred technique, let's
see how we can use this
principle to extend our renderer
to frame additional forward pass
efficiently.
Because we've set up our light
list in a persistent thread
group memory, we can use the
same data to accelerate an
effective tiled forward pass.
Whenever we're shading our
forward geometry in our forward
pass, we can simply use that
same persistent thread group
memory to read our tile light
list and use the same light loop
that we've been using in our
deferred lighting to very
efficiently shade our forward
pixels.
This forward pass really rounds
out the capabilities and allows
for transparency, special
effects, and other complex
shading that would normally not
be possible with just deferred.
However, there's always some
limitations to a deferred
pipeline.
Anti-aliasing.
Complex material expressions are
still a problem due to the
intermediate G-buffer
representation.
What we have seen is using this
tile technique we can very
efficiently accelerate forward
rendering using the tiled
lighting technique.
So, let's take a step back and
focus purely on that forward
pass, because alongside with
tiled lighting, this becomes a
viable solution in its own
right.
To create a forward only
renderer, we simply remove the
deferred geometry and lighting
pass.
However, our light calling
technique needs that depth to
fit its subfrustums.
So, we need to replace our
geometry pass with a depth
prepass to fill this depth
buffer.
And if your engine already has
such a depth prepass, this is
the perfect solution for you.
So, if you have one beat for
overdraw, optimization,
inclusion calling, or
self-blending, this solution
will fit your needs.
However, on iOS hardware, such a
pass is often unnecessary.
And for those cases, a different
lighting solution, called
clustered lighting, might be a
better fit for you.
This clustered solution has a
different way of creating the
light lists that does not
require any depth.
Because for clustered lights, we
don't create any depth bounds
for our tiles but we simply
subdivide the frustums along the
depth axis.
And we then emit a 3D light list
map instead of a 2D light map.
It might not be as efficient as
our fitted subfrustums from our
tiled lighting but it will give
us a measurably improved
performance on lighting, where
every pixel is only shaded by a
local light list.
Using clustered calling combined
with tile shading and persistent
thread group memory, this gives
us very optimized forward
rendering.
We've seen now a few of the most
popular pipelines and how to
implement them on Metal.
Now we're going to look at the
visibility buffer rendering
technique that tackles the
G-buffer overhead in a different
way, making it more suitable for
old hardware that does not
support hardware tiling.
Let's go all the way back to our
deferred renderer.
So, most of the optimization
we've shown so far only work on
the iOS architecture.
The visibility buffer technique
tries to minimize the
intermediate buffer bottleneck
in another way, namely by
storing the absolute minimum
amount of data in that buffer.
Instead of storing all the
surface and material properties
per pixel, we only store a
primitive identifier and
barycentric coordinates.
This data cannot be used
directly to shade your entire
scene, but it can be used to
reconstruct and interpolate the
original geometry, and then
locally run your entire material
logic within the lighting
shaders.
Since this reconstruction step
is very costly, this works very
well with the tiled lighting
technique because that
guarantees we're only going to
use reconstruction step once per
pixel.
When we're implementing this
technique, the biggest problem
is usually how to create these
primitive indices and how to
create these barycentric
coordinates without a lot of
additional processing.
We're now happy to tell you that
in Metal 3, you can now retrieve
the index of your current
primitive and the barycentric
coordinate of your current pixel
within your fragment shader
using these two new attributes.
The resulting geometry shader is
now extremely simple, making
your geometry pass faster than
ever and making it easier to
implement than ever using Metal
3.
We've now gone over all these
different options that you can
use to render your scene in
Metal.
Now let's look at a little demo
that showcases some of these
rendering techniques.
So, here we have our test scene,
which has some rather complex
geometry and setup PBR materials
and array of different material
shaders.
We can use deferred or tile
deferred, or even forward to
render this scene on any of your
devices.
Let's start with the normal
deferred renderer.
The deferred renderer has two
passes, as we've shown before.
And the first pass is now
rendering everything through
these intermediate G-buffers.
Let's look at some of those
G-buffer textures now.
So, here we have the albedo.
We have the normal.
And we have the roughness
texture of our G-buffer.
If you have temporal and
aliasing, or more complex
lighting models, you'll probably
need to store more in the
G-buffer.
The scene we're seeing right now
is being lit by that second
lighting pass.
So, let's go to a night scene to
visualize our lights a little
bit better.
Now, in this scene, to get it
lit up like this, we need to
render a lot of lights, which we
are visualizing here.
In normal deferred, we should be
rendering all of these lights
one at a time, which is very
inefficient.
And you can see there's lots of
overlap between the different
lights.
So, let's move on to a tile
deferred lighting.
So, here we have the same scene
rendering using tile deferred
renderer.
What we want to show here was
all the possible visualizations
we had for how the different
tiles, show you the amount of
lights that are being rendered
in each of these different
tiles.
And you can see that it really
makes a difference in using
these tiled subdivisions,
relative to just lighting all
the pixels with all the lights
at once.
Now, we've shown you some of
these possible rendering
techniques that you can use to
render your scene.
And now my colleague Srinivas
will show you in the next part
how you can turn your CPU heavy
render loop into a GPU-driven
pipeline.
[ Applause ]
>> Thanks, Al.
With Metal 2, we introduced the
GPU-driven pipelines which
consist of augment buffers and
indirect command buffers
with which you can now move your
CPU-based rendering operations
to the GPU.
My colleague just showed you how
to implement various advanced
rendering techniques with Metal.
In this talk, I'm going to show
you how to move your entire
CPU-based render loop to the
GPU.
Now, this will not only make
your render loop more efficient
but it allows you to free up
your CPU for any other
processing that you may want to
do, like complex physics
simulations REI.
Now, before diving into details,
let's first see what operations
are usually performed in a
render loop.
Now, large scenes require
complex rendering operations.
So, usually you do a series of
operations to efficiently render
the scene.
So, the first thing that you do
is frustum culling, where you
remove the objects that fall
outside the view frustum.
We only show draw calls for
those.
Next, occlusion culling.
Here you eliminate the objects
that are occluded by other,
bigger objects.
Another thing that usually done
is level of detail selection,
where you select between a range
of levels of details of the
model based on its distance to
the camera.
So, your CPU-based render loop
with all these operations
typically looks like this.
Basically, you first encode your
occluded [draws] and
occlusion test into your command
buffer and you execute it in a
render pass on the GPU to
generate occlude data for the
next frame.
Next, you do frustum culling,
eliminate the objects that are
outside the view frustum.
And LOD selection to pick a
level of detail of the model and
occlusion culling, to eliminate
the objects that are occluded
other, bigger objects.
So finally, you enclosed the
pass for visible objects and
execute it in a render pass to
generate your scene.
Now, this works fine but there
are a couple of inefficiencies
here.
First let's take occlusion
culling.
Now, to do occlusion culling,
you need occlude data for the
current frame.
But because you don't want to
introduce any synchronizations
in the current frame, you
usually rely on the previous
frame's occluder data, which is
usually obtained at a lower
resolution.
So, it can be approximate.
It can lead to false occlusion.
And so you probably have to take
some corrective steps in your
gains.
And second, there are operations
here that are highly
paralyzable.
For example, frustum culling.
On a single CPU thread it looks
like this, where you'll be doing
frustum culling of each object,
one after the other.
Now, you can definitely
distribute this processing
across multiple CPU threads but
there are only a few CPU threads
available.
And if you include all other
operations that you want to
perform per object, you'll
probably be doing something like
this.
But are these operations highly
paralyzable?
So, if you have more threads,
you can pretty much process all
the scenes, all the objects in
our scene in parallel.
But typically, there are
thousands of objects in a scene
and so to paralyze all of them,
you need thousands of threads.
So, the perfect choice for these
operations is the GPU.
Now, GPU is a massively parallel
processor with thousands of
threads available to schedule
operations on.
So, it is possible to assign an
object to a dedicated thread and
perform all the operations that
we want to perform on that
object.
And with thousands of threads,
you can process thousands of
objects in parallel.
So, your render loop is going to
be more efficient if you move it
from CPU to GPU and as I
mentioned before, it will also
be freeing up your CPU for any
other processing you want to do.
So, how do you move all these
operations to the GPU.
You can do it with a combination
of compute and render passes on
the GPU so that we can drive the
entire render loop on the GPU
without any CPU involvement.
This idea, I mean, this is what
we need.
The entire render loop here is
on the GPU.
It's completely GPU driven.
Now, let's go through these
passes to see how this
GPU-driven render loop really
works.
Now, we need occluder data for
occlusion culling.
So first we have a compute pass
that takes a scene data, does
frustum culling of occluders,
and enclodes commands for
rendering the occluders.
Now, these encoded occluded raw
commands are executed in a
render pass, so we generate any
required occluder data.
Now, this occluder data can be
in various forms, depending on
how it gets generated.
So, you may want to further
process that data.
For that, we can have another
compute pass.
Now in this pass, the occluder
data can be converted into a
form that is more suitable for
occlusion culling.
We need one more compute pass
for the operations we talked
about.
That is, culling, a level of
detail selection and for
encoding of scene raw commands.
So, one thing to find out here
is that occlusion culling here
is no longer dependent on
previous frames' data.
Required occluder data is
generated for the current frame
in the first two passes that we
just talked about.
And also because we are
generating it for the current
frame, it's also more accurate.
And finally, we have another
render pass that executes a
scene raw commands for rendering
the scene.
So, in this GPU-driven render
loop, everything's happening on
the GPU.
There is no CPU-GPU
synchronization anywhere, no
previous frame dependencies.
So, how can we build this
GPU-driven pipeline?
Now, it is clear that we need at
least two things to be able to
build this render loop on the
GPU.
First, draw commands.
We need a way to encode draw
commands on the GPU so that the
compute pass can encode the
commands for the render pass.
And the building block that
Metal provides to support this
is indirect command buffers.
And we also need scene data.
We should be able to access the
encoded scene data on the GPU
through the frame wherever it is
needed.
And with this scene data, we
should be able to pretty much
describe the whole scene, like
geometry, shared arguments,
materials, etc. So, in the
building block that Metal
provides to support this is
argument buffers.
Now, let's take a more closer
look at these two building
blocks.
Now, argument buffers let you
describe the entire scene data
with complex data structures.
And they let you access the
scene data anywhere in the loop.
And indirect command buffers
allow you to build draw calls on
the GPU and basically it
supports massive parallel
generation of commands on the
GPU.
Now, let's dive into these
argument buffers a little
further with an example scene
object model.
Now, the first thing that we
need is access to scene data.
So, what does scene data really
consist of?
First, meshes.
Now, here is the meshes.
It is an area of mesh objects,
each describing its geometry.
And Metal is an area of Metal
objects each with a set of Metal
properties, any textures it
needs, and the pipeline steered
object that describes the shadow
pipeline.
And scene also consists of an
area of models.
Here, each model can have an LOD
so in this example, we have each
model consisting of area of
meshes and materials, one per
LOD.
Finally, we have a scene object
that relates meshes, materials,
and models that are part of our
scene.
So, let's see how this object
model can be expressed with
argument buffers.
It is a very simple 1-to-1
mapping from our object model to
argument buffers.
For example, scene argument
buffer here simply consists of
the objects that we just
described in our object model.
That is areas of meshes,
materials, and models.
Basically, the entire scene can
now be described with argument
buffers.
Now, let's look at how this can
be constructed and accessed in
the shader.
Now, each of the argument
buffers we just discussed is
simply represented by a
structure.
That now contains members that
we have described in our object
model.
Since each argument buffer is a
structure that is completely
flexible, you can add things
like arrays, pointers, even
pointers to other argument
buffers.
For example, here's a Metal
argument buffer.
It can contain Metal Constants
any textures it needs, and of
course the pipeline straight
objects that describe the shadow
pipeline.
So everything that is needed for
a Metal is in one argument
buffer.
And the scene argument buffer is
just like how we described it in
our object model.
So, it's just very easy to
construct object models with
argument buffers.
Now, let's look at how we can
access these argument buffers in
the sharable.
Now, this is a compute kernel
that does frustum culling that
we just talked about.
It encodes the draw commands for
visible objects into an indirect
command offer.
Each thread that executes an
instance of this kernel
processes one object and encodes
a single draw call if it data
mines that object is visible.
So, let's see how this does it.
Now, first we pass in our
high-level scene argument buffer
to the share.
Now, once we have access to our
shader, our scene, then it is
very easy to access anything
else we need.
And command R here contains the
reference to the indirect
command buffer that we want to
encode into.
We first did the model from the
scene based on thread ID.
Notice that all threads of this
compute kernel are doing this in
parallel, each operating on a
particular object.
We do frustum culling to see if
the object is falling outside
the view frustum.
And once we determine that
object is visible, we calculate
its LOD based on its distance to
the camera.
So once we have the LOD, it's
very straightforward to read its
corresponding mesh and material
argument, argument buffers that
apply to that LOD.
This is straightforward mainly
because of the way argument
buffers lets us relate resources
that we need in our scene.
And we have acquired all the
information we need; now it's
time to encode.
Let's just see what encoding
into an indirect argument
buffer, indirect command buffer
really means.
So, indirect command buffer is
an area of render commands.
Each command can have different
properties.
A command can include a pipeline
straight object that describes a
shared pipeline and any vortex
and fragment buffers that the
draw call needs.
And the draw call itself.
So, encoding basically means
that once we determine that an
object is visible, we read it
with all its properties and
encode those into the indirect
mine buffer.
Now, each thread that processes
an object can encode into a
particular slot in this indirect
command buffer.
And since all threads are
running in parallel, commands
can be encoded concurrently.
Now, let's continue with our
culling kernel example to see an
actual example of the encoding.
Now, we first need a position in
the command buffer to encode the
raw command.
So, we use that raw ID to get
ourselves a slot in the indirect
command buffer.
And like we discussed, we need
to set any parameters that the
draw call needs.
Now, the material and mesh
argument buffers that we just
acquired have all the
information we need to set the
parameters.
So for example for material we
can set the pipeline straight
object that we need to set.
And from the mesh object, we can
set any vortex buffer or any
vortex uniforms that we need to
set.
And of course the fragment needs
the material, so we set that.
And finally this is how we
encode the draw.
So that's it.
Encoding the draw call is very
simple and easy.
Now, let's see how you can set
up your path in your game.
Now, we first need an indirect
command buffer to encode
occluder draw commands, because
that is the first thing that we
talked about when we discussed
our GPU-driven render loop.
So to render the occluders, we
start up a compute dispatch that
does custom culling of occulders
and encodes the occluder draw
commands.
And because each thread is doing
independently encoding a draw,
there can be multiple state
settings, written and state
settings in the indirect command
buffer.
So, optionally we optimize the
indirect command buffer to
remove any driven end stage
settings.
Now, this is a random pass that
executes the occluder draws in
the indirect command buffer.
And similarly, the rest of the
paths can be set up easily.
For example, here is our, the
main compute dispatch that
launches our culling kernel that
we just talked about that does
culling tests, LOD selection,
and encoding draw commands.
Now we are ready to launch our
final render pass that executes
the commands in the indirect
command buffer.
So that's it.
That's all it takes to draw the
scene.
Now, let's take a look and see
how the indirect command buffer
looks like after the encoding of
draw commands.
Now, it can be sparse with
holes.
This is mainly because as we
have just seen in our culling
kernel example, the thread that
is processing an object doesn't
encode the draw command if it
finds that object is not
visible.
For example, objects one and
three, in this case.
That means those slots in the
indirect command buffer are
empty.
So if you submit this command
buffer to the GPU, it'll end up
executing a bunch of empty
commands, which is not really
efficient.
So, the ideal thing to do is to
tightly pack the commands like
this.
That is, we need a way to pack
the commands as we encode the
draws.
For that, we have indirect
principle.
With indirect ranges, you can
tell the GPU with execute call
where to get the range of
commands to execute.
Basically, you can have indirect
range buffer that has a start
location and number of commands
to execute, and this buffer can
be populated on the GPU as
you're doing your encoding of
draw commands.
And the execute call will pick
up the start location and the
number of commands from this
buffer.
It can be used for both packing
and the range.
Now, let's go to an example and
see how this really works.
This is our culling kernel that
we just discussed before,
modified to use indirect range
buffer.
Let's see how this kernel packs
the draw commands.
We first pass in our pointer to
the length member of the
indirect range buffer.
And when we are retrieving the
command to encode, we can
automatically increment the
length.
Now, each thread is atomically
incrementing the length, and so
when this compute work is done,
the length is automatically set
up in the indirect range buffer.
At the same time, the draw
commands have been packed.
Because the indirect that is
returned by this atomic
instruction in this code is the
previous value of the length.
And so for example, if you start
at zero, the thread that is
using the zero slot is
incrementing the length to 1.
And the thread that is using the
first slot is incrementing the
length to 2 and so on.
So, this is great because now we
not only pack the commands; we
also updated the range at the
same time.
Now, let's see how we can set up
the indirect range buffer in the
application.
First, you create a range buffer
for the compute pass to update
the range.
Next, you set up the range
buffer as a kernel argument for
the culling compute kernel.
And then we do the compute pass
that launches the culling kernel
that does the object first thing
and also updates the range
automatically.
And finally, you schedule the
pass with execute commands in
buffer with indirect range API.
Now, this call will pick up the
start location and the number of
commands that is executed from
this indirect range buffer.
So, with indirect ranges, you
can get more efficient execution
of indirect command buffers.
Now, so far in our GPU-driven
pipeline, all these draw
commands are built in compute
passes on the GPU.
And these compute passes where
regular dispatch is happening in
your game.
So, one question that comes to
mind is building compute
dispatches on the GPU.
Can we encode compute dispatches
into indirect command buffer?
So, I'm very happy to let you
all know that a new addition we
are now putting into Metal 3 is
support for encoding compute
dispatches.
Now, you can build your compute
dispatches on the GPU too.
In terms of functionality,
compute indirect command buffers
are just like render.
They can also be built once and
can be reused again and again.
So, they also help in saving CPU
cycles.
And the great thing is both
render and compute can now be
driven on the GPU.
It's really great because now
you can build more flexible
GPU-driven pipelines.
Now, let's see an example with
this with the use case.
Per patch tessellation factors.
So, let's say we have a mesh
that is made up of a bunch of
patches and we want to generate
tessellation patches for each
patch.
We can definitely do this in the
culling compute kernel that we
talked about that does culling
tasks and encodes draw commands.
That is a GPU thread that is
processing an object can go
through each patch of the object
and can generate tessellation
factors.
But that's not really an
efficient thing to do because
generating tessellation factors
is also a paralyzable operation
by itself.
So, the efficient thing would be
to distribute this per operation
across multiple threads so that
all patches are processed in
parallel.
That is, each thread of the
culling compute dispatch that is
processing an object can encode
compute dispatches for test
factor generation.
And those dispatches can be
executed on another compute
pass, paralyzing the operation.
So, with GPU-driven dispatches,
we can now do this.
So, let's see how we can
exchange our GPU-driven pipeline
to accommodate this processing.
Here is the main compute pass
that we talked about before that
does the culling tasks, LOD
selection and encoding of draw
commands.
We can now action this pass to
also encode dispatches for test
factor generation.
For example, after a thread
determines an object is visible,
it can encode dispatches for
test factor generation into an
indirect command buffer.
And then those commands can be
executed in another compute pass
before the main render pass.
So, the GPU-driven dispatches
combined with GPU-driven draws
lets us build more flexible
GPU-driven pipelines.
So, we built a sample to show
you what we talked about in
action.
Let's take a look.
Now, here is a bistro scene that
you saw before.
This, we are actually doing a
fly by through the street here.
This scene is made up of about
2.8 million polygons and close
to 8000 draw calls.
And that's for one view.
And if you consider the shadow
cascades that have been used
here for shadow processing, this
render is handling about four
such views.
So, that is quite a few API
calls if this scene gets
rendered on the CPU.
But in this sample, we are using
indirect command buffers and so
everything is on the GPU.
It's completely GPU driven.
The entire render loop is on the
GPU, and so it's saving the CPU
from a lot of work.
Let's look at one more view.
Now, we are looking at the same
view, same fly by, but we are
looking at the camera as it's
passing through the street here.
To be clear, we also are showing
the camera that white object,
that is the camera.
We are showing the geometry that
is tinted with magenta color is,
falling, geometry that is also
the view for some of the camera.
So, as you can see, as the
camera is passing through the
street, there's quite a bit of
geometry that is also the view
frustum of the camera.
And our culling compute dispatch
that does frustum culling on the
GPU determined this geometry,
this tinted geometry as
invisible.
And so this geometry doesn't get
processed or rendered on the
GPU, saving significant
rendering cost.
Let's look at one more final
view.
And here is one more view.
Here we are showing both frustum
and occlusion culling at work.
We are, we tinted the geometry
that is occluded with cyan color
and the geometry in magenta is
outside the view frustum.
And you can see there's quite a
bit of geometry on the right
here that is occluded by the
bistro, so it is in cyan color.
And as you can see, there's a
lot of geometry here.
Also the view frustum are, is
occluded.
And again, our culling compute
kernel that does both frustum
and occlusion culling on the GPU
determined these geometry to be
invisible.
So, this tinted geometry doesn't
get processed or rendered on the
GPU, saving significant
rendering cost and increasing
performance.
So, before we end this talk, I'm
going to show you one more
thing.
I'm going to show you how we are
making it easier than ever to
write a cross ref on Metal core.
I'm also going to show you how
to more easily target features
that are iOS, tvOS and macOS
specific.
And before we do that, let's
take a quick look at Metal
features that are now available
across all our platforms.
Now, we have several features
new to iOS and tvOS.
In the previous sections, we
showed you how setting pipeline
states in indirect command
buffers helps you to fully
utilize GPU-driven pipelines.
We also showed you how indirect
ranges allow you to more easily
and more efficiently pack and
execute indirect commands.
And finally, we are bringing
16-bit depth texture support to
iOS and tvOS.
This has been a popular request
that helps to optimize shadow
map rendering.
We also have several important
features new to macOS.
We can render now without
attachments in cases where you
need more flexible outputs to
memory buffers.
You can query the time your
command buffer takes on the GPU
so you can adjust your
representation intervals
dynamically.
And finally, macOS now supports
casting between sRGB and
non-sRGB views to better
accommodate linear versus
nonlinear lighting.
So, now let's take a look at the
new GPU family API.
Now, you previously used Metal
feature set queries to condition
your applications based on
available features and limits.
But the number of features,
feature sets has grown and they
currently number, numbers in the
dozens.
The GPU family queries replace
feature sets and makes it easier
to query the capabilities of the
system.
First, we have consolidated into
four families and organized them
to simplify cross-platform
development.
Second, each family supports a
hierarchy of features organized
into one or more instances.
So, support for one instance
means all earlier instances are
supported.
Third, the new API separates out
Metal software version query to
track how to instances of a
given family change our software
delivers.
And finally, a GPU family
defines a small set of device
queries for optional features
that don't neatly fit into
families.
Now, with that said, let's take
a closer look at the new GPU
family definitions.
All iOS and tvOS features are
now organized into their family,
a single family of five
instances.
With each instance supporting
all features included within the
earlier instances.
So, I'm not going to enumerate
all the features here, but the
resource section of this talk
will have a table that'll map
features to families and
instances.
Mac features are similarly
organized around only two
instances.
Again, the Mac II supports all
the features from Mac I.
Now querying these features,
these families greatly
simplifies writing flat,
unspecific code.
But what about when you want to
target all of the platforms?
For that, we have the new common
families.
The common family organizes
Metal features into
cross-platform hierarchy.
Common 1 is universally
supported by all Metal GPUs and
is a great choice for apps that
only use Metal lightly.
Common 2 provides all the
building blocks necessary for
great game development such as
indirect draw, counting
occlusion queries, tessellation,
and Metal performance shadow
support.
And common 3 provides all the
features needed by advanced
applications such as indirect
command buffers, layered
rendering, cube map arrays, and
vortex position invariants.
And finally, Metal 3 provides a
special family for iPad apps
targeting the Mac.
That is, tuned for that
experience.
The two iOS mac instances
support a combination of
features critical for great
performance on the Mac.
In particular, they make
available the Mac-specific block
compression pixel formats and
manage texture modes for use
within an otherwise completely
iOS application.
Now, iOS Mac 1 supports all the
features of Common 2 plus
several features from Common 3.
Besides the BC pixel format and
managed textures, it supports
cube texture arrays, read/write
textures, layered rendering,
multiple viewport rendering, and
indirect tessellation.
iOS Mac II supports all the
features of Common 3 in addition
to the BC pixel formats and
managed textures.
So, that's the four new
families.
Now, let's take a look at how
you'll use the new QD API in
practice.
Now, in this example we'll check
whether the Mac II features are
available.
We start by checking whether the
OS supports the new family API.
And if the new family API is
available, then we use to check
for Metal 3 features that are
available.
Since Metal 3 is new, you don't
need to strictly check for it,
but that's a good practice.
And if Metal 3 is available,
then we check for the family we
would like to use.
Cross-platform applications
here, check for one of the
common families, as well as one
or more Apple or Mac-specific
families.
If either the API or the
version are not available,
then we fall back to the
older feature set API on
earlier Metal versions.
So, now let's take a look at the
setup option features you can
query for.
Now, when a family specifies a
general behavior of GPUs in that
family but some important
features and limits are not
supported uniformly across a
family.
Such as a depth 24 stencil 8
pixel formats, and the number of
MSA samples in a pixel.
So to handle those cases, the
Metal device provides an API to
query for each of those features
directly.
But as you can see, that is not
many features that fall into
this category.
So, to end this section, let's
look at how the many techniques
we have discussed so far are
supported by the new GPU
families.
Classic deferred shading is
supported across all our
platforms and programmable
blending is supported across all
Apple GPUs, making it a good
default choice for your games.
Tile deferred and forward
rendering are also broadly
supported with Apple-specific
optimizations requiring more
recent hardware.
And finally, the visibility
buffer technique is only
supported by the Mac family.
It just happened to have very
demanding resolution
requirements.
Now, let's end this section by
looking at how our GPU-drive
pipeline features are supported
across our families.
Now, some features require broad
support to become a core part of
render engines, and we believe
that GPU-driven pipelines
require that kind of support.
So, we are therefore very happy
to let you all know that
argument buffers and indirect
command buffers for both
graphics and compute are now
supported by a Common family 2
and later.
Now that brings us to the end of
this session on Model Rendering
with Metal.
We hope you can apply all these
techniques to your games and
apps.
Let's do quick recap.
My colleague, Jaap, just showed
you how to implement many
advanced rendering techniques
with Metal, techniques like
deferred shading, tile forward
rendering are excellent matches
for iOS when combined with and
optimized with programmable
blending and tile sharing.
On Mac, you can use the new
barycentric coordinates and
query LOD to implement the
visibility buffer technique and
render it high resolutions.
But no matter what technique you
choose to use, you can move your
entire render loop to the GPU.
Frustrum culling, occlusion
culling, LOD selection, can all
be done on the GPU with argument
buffers and indirect command
buffers.
And now we can also encode
compute dispatches into indirect
command buffers on the GPU.
Whether you want to target a
wide range of hardware on both
iOS or macOS, or only want to
use a few advanced Metal
features, you can now use the
newly redesigned GPU family API
to check for feature
availability at runtime.
Now, please visit our session
website to learn more about
Metal features and GPU-driven
pipelines.
We will be posting the sample
app that we used in this talk.
You can explore those techniques
and integrate them into your
apps and games and please join
us in our labs.
In fact, there is one right
after this talk.
Thank you, and have a great
conference.
Thank you.
[ Applause ]