Transcript
[ Music ]
[ Applause ]
>> Good morning everyone.
My name is Sean and I'm an
engineer on Apple's GPU Software
Team.
In this session we're going to
talk about Ray Tracing.
So, let's first review what Ray
Tracing is.
Ray Tracing applications are
based on tracing the paths the
rays take as they interact with
a scene.
So, Ray Tracing has applications
in rendering, audio, physics
simulation, and more.
In particular Ray Tracing is
often used in offline rendering
applications to simulate
individual rays of light
bouncing around a scene.
This allows these applications
to render photo realistic
reflections, refractions,
shadows, global illumination,
and more.
Recently, Ray Tracing has also
started to be used in real-time
applications such as games.
And this actually introduces
some new requirements.
First, in real-time applications
objects tend to move around.
So, we need to be able to
support both camera and object
motion.
Second, performance is now even
more critical.
This means that the ray
intersection itself has to be as
efficient as possible and we
also have to make effective use
of our limited ray budget.
So, we need to careful with our
sampling strategies, random
number generation, and so on.
Finally, even with these
techniques we won't be able to
cast enough rays to remove all
the noise.
So, we need a sophisticated
noise reduction strategy.
Fortunately, Metal has built in
support for Ray Tracing and
Denoising.
So, it's easy to get started.
So, let's first review how Ray
Tracing works in Metal and then
we'll move on to some more
advanced topics.
So, if you look at a typical Ray
Tracing application, they all
follow roughly the same outline.
First, we generate some rays.
Each ray is defined by its
origin point and its direction
vector.
Those rays are then intersected
against the geometry in the
scene, which is usually made of
triangles.
The intersection data could just
be the distance to the
intersection point, but it
typically includes additional
data such as the index of the
triangle that was hit and the
very center coordinates of the
intersection point.
The next step consumes the
intersection results.
For example, in a rendering
application this is typically a
shading step which outputs an
image.
And this step may also generate
additional rays, so we repeat
this process however many times
we need until we're done.
Applications typically intersect
millions of rays with the scene
every frame.
And this core intersection step
is common to all Ray Tracing
applications.
So, this intersection step is
what we accelerate in Metal.
Last year, we introduced the
MPSRayIntersector API, which is
part of the Metal performance
shaders framework.
This API accelerates ray
intersection on the GPU on all
of our Mac and iOS devices.
We talked a lot about this API
in last year's talk, so we'd
encourage you to go back and
review this talk for more
information.
At a high level, this API takes
in batches of rays through a
Metal buffer.
It finds the closest
intersection along each ray and
returns the results in another
buffer.
And all of this work is encoded
into a Metal commandBuffer at
the point in your application
where you'd like to do
intersection testing.
Much of the speed up comes from
building a data structure we
call an acceleration structure.
This data structure recursively
partitions triangles in space so
that we can quickly eliminate
triangles which cannot possibly
intersect a given ray during the
intersection search.
Metal takes care of building
this data structure for you.
All you have to do is specify
when you'd like to build the
acceleration structure, then
simply pass it to the
intersector for intersection
testing.
Now, building this data
structure is typically a fixed
cost paid when your app starts
up.
In last year's version of the
API this data structure was
always built on the CPU.
This year, we've moved the
acceleration structure build to
the GPU, which can significantly
reduce the startup cost.
And even better, the GPU will be
used automatically whenever
possible.
So, you don't need to do
anything to see the speed up in
your applications.
So, let's now revisit our
typical Ray Tracing application
and see what we need to do to
translate it into a Metal Ray
Tracing application.
So, like I said, we'll start by
generating rays.
This is typically done using a
compute kernel, but it could
also be done from a fragment
shader or really any mechanism
that can write to a Metal
buffer.
We then pass the ray buffer to
the intersector.
It'll find the intersections and
return the result in our
intersection buffer.
And remember that to use the
intersector we need to provide
an acceleration structure.
We can often build this just one
and reuse it many times.
Finally, we'll launch one last
compute kernel which will use
the intersection data to write a
shaded image into a texture.
And this compute kernel can also
write additional rays back into
the ray buffer for iterative
applications.
So, let's see how this works in
a real application.
For this example, we'll talk
about how Ray Tracing is being
used in AR Quick Look.
AR Quick Look was introduced
last year and allows you to
preview 3D assets in augmented
reality.
We talked a lot about AR Quick
Look in this morning's session.
So, I'd encourage you to watch
that talk as well.
For this talk, we'll focus on
how AR Quick Look is using Ray
Tracing to render an Ambient
Occlusion effect.
We'll talk more about Ambient
Occlusion later, but for now,
what you need to know is that
Ambient Occlusion computes an
approximation of how much light
can reach each point in the
scene.
So, this results in the
darkening of the ground
underneath the robot model as
well as soft contact shadows
between the robot's legs and the
ground.
The effect is somewhat subtle,
but if we turn it off, we can
see that it actually goes a long
way towards grounding the robot
in the scene.
And this is really important for
AR applications to prevent
objects from looking like
they're floating above the
ground.
In last year's version of AR
Quick Look, the shadows are
actually precomputed, so they
wouldn't move as objects moved
around.
This year, we've used Metal
support for dynamic scenes to
render these shadows in real
time.
So now as objects move their
shadows will move with them.
And this even works for
deforming objects such as
skinned models.
We can see the shadows follow
the motions of the fish as it
swims around the scene.
So, what we just saw actually
has three types of animation
going on.
If we were just using the
rasterizer, we'd simply
rasterize the triangles in their
new position.
But because we're using Ray
Tracing, we need to maintain an
acceleration structure.
So, the first type of animation
is simple camera movement.
And this is movement due to just
moving the iPad around.
We don't need to update the
acceleration structure just
because the cameras moved.
So, we actually get this type of
animation for free.
We can simply start firing rays
from the new camera position.
The other two types of animation
do require updating the
acceleration structure.
So, the first is Vertex
Animation.
This could be skinned models
like the fish, but it could also
be plants blowing in the wind,
cloth, or other types of
deformation.
Metal includes a special
acceleration structure update
mechanism optimized for cases
like this.
And the last type of animation
is rigid body animation.
This is where objects can move,
rotate, and scale, but otherwise
completely maintain their shape.
So, a large portion of the
acceleration structure is
actually still valid.
So, Metal also includes a
special mechanism to reuse the
parts of the acceleration
structure that hasn't changed.
So, let's first talk about
Vertex Animation.
As the geometry changes, we need
to update the acceleration
structure.
We could rebuild it from scratch
every frame, but we can actually
do better.
In Vertex Animation use cases
objects tend to mostly retain
their shape.
For example, a character's hands
will stay connected to their
arms.
Their arms will stay connected
to their body and so on.
So, the spatial hierarchy
encoded into the acceleration
structure is mostly still valid.
It just needs to be adjusted to
the new geometry.
Let's look at an example.
So, here's the acceleration
structure we saw earlier.
If the triangles move, we can
see that the bounding boxes no
longer line up with the
triangles.
But the tree structure itself
mostly still makes sense.
So rather than rebuild it from
scratch, we can simply snap the
bounding boxes to the new
triangle positions from bottom
to top.
We call this operation
Refitting.
As we can see, this still
results in a valid acceleration
structure, but it's much faster
than building from scratch
because we can reuse the
existing tree.
This also runs entirely on the
GPU, which makes it even faster,
but also means that we can
safely encode a Refitting
operation after say a compute
kernel which updates the
vertices.
The downside is that we can't
add or remove any geometry
because the tree will still
encode references to the old
geometry.
This also potentially degrades
the acceleration structures
quality which can impact Ray
Tracing performance.
This is because the triangles
were originally partitioned
using a set of futuristics which
won't be accurate after the
triangles move.
The impact is usually minor, but
extreme cases like teleporting
geometry could cause performance
problems.
Nonetheless, this works great
for typical deformation and
character skinning use cases.
So, let's see how to set this up
in code.
First, before we build the
accelerations structure, we need
to enable support for Refitting.
And note that just enabling
Refitting is enough to reduce
the acceleration structure's
quality.
So definitely only turn this on
if you really need to refit the
acceleration structure.
Then we simple call encodeRefit
into a Metal commandBuffer.
And that's all we need to do for
Vertex Animation.
So next, let's talk about Rigid
Body Animation.
So as the name implies, this is
animation where objects can
move, rotate, and scale, but
otherwise completely maintain
their shape.
So, in the example on the right,
even though it looks like the
robot is deforming, actually all
of its joins are moving rigidly.
So, this is still an example of
Rigid Body Animation.
So, in a typical scene, most of
the geometry is probably only
moving rigidly.
In fact, most of the geometry is
probably not moving at all.
We may also have multiple copies
of the same objects in the
scene.
It would be wasteful to
replicate these objects multiple
times in the accelerations
structure and it would also be
inefficient to refit or rebuild
the entire acceleration
structure just because a subset
of the geometry is moving.
So, to solve both of these
problems, we can use what we
call a Two-Level Acceleration
Structure.
So, what we'll do is first build
a high-quality triangle
acceleration structure for each
unique object in the scene.
And we can do this just once
when the app starts up.
We'll then create two copies of
those -- of those triangle
acceleration structures using a
second acceleration structure.
Each copy is called an instance
of one of the original triangle
acceleration structures.
Each instance is associated with
a transformation matrix,
describing where to place it in
the scene.
So, we'll do this all using two
buffers and each buffer will
contain one entry for each
instance in the scene.
The first buffer will contain
the transformation matrices for
all the instances.
The second buffer will contain
indices into an array of
triangle accelerations
structures describing which
acceleration structure to use
for each instance.
We'll then build a second
acceleration structure over just
the instances in the scene.
We can then quickly rebuild just
the instance acceleration
structure as the objects move.
So, let's see how to set this
up.
First, we'll create what's
called an
AccelerationStructureGroup.
All the acceleration structures
in the instance's hierarchy must
belong to the same group.
And this allows them to share
resources internally.
Next, we'll create an array to
hold our triangle acceleration
structures.
Then finally, we'll loop over
all the unique objects in the
scene, building a triangle
acceleration structure for each
of them.
Adding them to the array as we
go.
We're now ready to create the
second level acceleration
structure.
We do this using the NPSInstance
AccelerationStructure class.
We'll start by attaching our
array of triangle acceleration
structures as well as the two
buffers I talked about
previously.
Then finally, we'll specify the
num of instances in the scene.
Then, whenever the objects move
or if an object is added or
removed from the scene, we can
simply rebuild just the instance
acceleration structure.
This acceleration structure is
typically much smaller than a
triangle acceleration structure,
so we can afford to do this
every frame.
But note that similar to
Refitting, there is some
overhead when using instancing.
So, if your scene only has one
object or a handful of objects,
or especially if none of the
objects are moving, it might be
worthwhile to pack those into a
single triangle acceleration
structure.
This will increase your memory
footprint, but it should gain
back some of the performance.
So, you need to experiment to
find the right tradeoff for your
application.
So that's it for dynamic scenes.
We talked about how to support
Vertex Animation and Skinning
using Refitting.
As well as how to support Rigid
Body Animation using a Two-Level
Acceleration Structure.
So next, let's switch gears and
talk about Denoising.
So far, all the images that we
have seen have been free of
noise.
That's because they've all been
using a denoising filter.
If we turn it off, we could see
what it would have looked like
without the denoiser.
We can see that these images are
too noisy to use in a real
application.
That's because we're only using
a handful of samples per pixel.
Usually we would just solve this
by averaging together more
samples over time.
But if the camera or objects are
moving it's not quite that
simple.
Fortunately, Metal now includes
a sophisticated Denoising
filter.
Let's see how this works.
Ideally what we'd be able to do
is simply take the noisy image
output by a renderer, run it
through a denoiser and get back
a clean image.
In practice, the denoiser needs
a little more information about
the scene.
We'll start by providing the
depths and normal for the
directly visible geometry.
Many renderers have these
textures lying around, and if
not, it's easy to produce them.
The denoiser will then run a
bunch of image processing
operations and output a cleaner
image.
But since we started with just a
handful of samples per pixel,
the result will still have some
noise.
So, we'll revisit the idea of
combining samples over multiple
frames.
So, we'll first set aside the
clean image to reuse in the next
frame.
We'll also set aside the depth
and normal so we can compare
them to the next frame.
Then finally, we'll provide a
motion vector texture which
describes how much each pixel
has moved between frames.
In the next frame, the denoiser
will churn through all of these
textures to produce an even
better image.
And this image will continue to
get better over time even if the
camera or objects move.
The denoiser will use the depths
and normal to detect cases where
the history for a pixel has
become invalid due to an object
moving or getting in the way.
So, this is all implemented
using the MPSSVGF family of
classes.
This is an implementation of the
popular MPSSVGF denoising
algorithm.
This algorithm makes a good
tradeoff between high quality
and real-time performance.
So, the denoising process is all
coordinated by the
MPSSVGFDenoiser class.
Meanwhile, low-level control is
provided using the MPSSVGF
class.
This class provides the
individual compute kernels used
by the denoiser and exposes many
parameters you can use to fine
tune the Denoising in your
application.
And you also just call this
classes' methods directly to
build a customized denoiser.
Now the denoiser creates and
destroys quite a few temporary
textures throughout the
Denoising process.
To the MPSSVGF texture allocator
protocol serves as a cache for
these memory allocations.
You can either use the default
implementation or implement this
protocol yourself to share
memory with your own
application.
So as usual, we've optimized
these classes for all of our Mac
and iOS devices.
The denoiser can process two
independent images
simultaneously.
For example, you might want to
split your direct and indirect
lighting terms into separate
textures.
There's also a fast path for
single channel textures such as
Ambient Occlusion or shadow
textures, which is faster than
Denoising a full RBG image.
Let's see how to set this up.
So first we'll create the
MPSSVGF object and configure its
properties.
All we need to provide is the
Metal device we want to use for
Denoising.
Next, we'll create the
TextureAllocator.
In this case we'll just use the
default implementation.
Then finally, we'll create the
high level Denoiser object which
will manage the denoising
process.
So now we're ready to do some
denoising.
We'll start by attaching all of
the input textures to the
Denoiser.
Now we simply encode the entire
denoising process to a Metal
commandBuffer.
And finally, we can retrieve the
clean image from the denoiser.
And that's all you need to do to
enable denoising your
applications.
So, we now talked about all of
the basic building blocks
available in Metal for Ray
Tracing and Denoising.
We reviewed how to do basic
ray/triangle intersection using
the MPS Ray Intersector API.
We then talked about how to
extend this to dynamic scenes
using Refitting and Two-Level
Acceleration Structures.
And finally, we talked about how
to remove the noise from your
images using the MPSSVGF
classes.
Now, don't worry if this is all
a little bit overwhelming.
We've written a sample, which
demonstrates how to use all of
these concepts which is
available online.
Now, I mentioned earlier that we
need to be careful with
performance.
Especially in a real-time
setting.
So next, I'd like to bring out
my colleague Wayne who will talk
about how to make all of this
work on real devices with real
performance budgets.
[ Applause ]
>> Hi everyone.
Now, what I'd like to show you
in this part of the talk is how
to use the Ray Tracing features
that we have in Metal to
implement a few different
rendering techniques in your
applications.
So, in particular, I'll be
focusing on hard and soft
shadows, Ambient Occlusion, and
global illumination.
So, let's start with hard
shadows.
Now, the way that we model this
with Ray Tracing is to take
points on our surface and fire
rays up in the direction of the
sun.
If a ray hits something, then
the associated point is in
shadow.
Otherwise, it's in sunlight.
Now, to incorporate this into an
existing application, I'm going
to assume that you're starting
with something a bit like this.
You've rasterized a G-Buffer and
run a compute pass for your
lighting.
And the output of that is your
final shaded image.
Now, to take advantage of Ray
Tracing here we'll start by
taking the G-Buffer and then run
a compute shader to generate
some rays.
We'll then pass those rays to
Metal to intersect with an
acceleration structure.
And Metal will output the
results to an intersection
buffer.
You can now use this buffer in
your shading kernel to decide
whether your surface points are
in shadow.
Now the main part I'd like us to
focus on here is Ray Generation.
So, let's start with a quick
reminder of how rays are
described in Metal.
So, Metal provides a few
different ray structures for you
to use, and at a minimum these
contain fields for your ray
origin and your ray direction.
So just fill out one of these
structures for each ray that you
want to trace and write it out
to your ray buffer.
Now, the way in which you
arrange your rays in your ray
buffer, that has a performance
impact.
So, often you might start like
this.
We call this row linear order.
Now, the problem here is that as
Metal works its way through
these rays, they tend to hit
very different nodes in the
internal data structures that
Metal uses to accelerate ray
traversal.
Now this in turn can flash the
underlying hardware caches.
So, a better approach is to use
block linear ordering.
So, rays from nearby pixels on
the screen, they tend to hit the
same parts of your acceleration
structure, and so by storing
your rays like this it enables
Metal to drive the hardware much
more efficiently.
Now, in the visualization here,
I'm showing you a block size of
4 by 4.
In practice we found that 8 by 8
works really, really well.
So, optimizing your ray storage
is a great way to improve
performance.
But where possible, an even
better way is just not to fire
rays at all.
Now, in the context of shadows,
the reason that you might want
to do this is because not all
pixels need a shadow ray.
For example, pixels on your
background, on your skybox, or
on surfaces that are facing away
from the sun.
Now, it's likely that your ray
buffer contains a ray structure
for each pixel on the screen.
So, what we need here is a way
to tell Metal to skip firing
ways for the pixels that we just
don't care about.
Now, there's a few ways to do
this.
The approach I'm showing you
here is simply to set the
maxDistance field in your ray
structure to a negative value.
And that's the main things you
need to know for Hard Shadows.
As you can see, Ray Tracing
gives really great results.
The shadows are very crisp and
they're very precise.
But in reality, shadows cast by
the sun, they tend not to look
that sharp.
they look more like this.
They're soft around the edges
and that softness varies with
distance.
And you can see a great example
of this on the left there.
the shadow from the lamp post
starts off hard at the base and
it softens as the distance to
the ground increases.
So, to model that with Ray
Tracing, instead of using the
parallel rays that I was showing
you earlier, we'll instead
extend the cone from our surface
point all the way up to the sun.
And then, we'll generate some
ray directions randomly within
this cone.
Now you can see there that some
rays intersect geometry and some
don't.
And it's this ratio that
controls the softness of your
shadow.
So, here's what that looks like.
What I'm showing you here is the
raw direct lighting term ray
traced with one ray per pixel.
So, in this image, all other
effects such as reflections and
global illumination, they're all
disabled so we can focus purely
on the shadow.
And as you can see, the result
is really quite noisy.
Now, to deal with that, we could
just keep firing more and more
rays.
But since that's something we're
really trying to avoid in a real
time application, what we can do
instead is use the Denoiser that
Sean was telling us about
earlier.
And here's the results of that.
Most of the noise is filtered
away and we get these great
looking soft shadows with just
one ray per pixel.
And I'll be showing you this in
action in our live demo later
on.
So now let's talk about Ambient
Occlusion.
So essentially, this is an
approximation of how much
ambient light is able to reach
the surface.
And as you saw in our AR Quick
Look demo earlier, it's a really
great technique for grounding
objects in their environments.
So, let's visualize how this
works with Ray Tracing.
We have a surface point in the
middle of the screen there and
there's a blue block over on the
right that's going to play the
role of our occluder.
We define an imaginary
hemisphere around our surface
points and then we fire some
rays.
If a ray hits something, and we
found that object is blocking
ambient light from reaching the
surface.
Now, as I've mentioned a couple
of times now, in a real time
application, we're really trying
to limit ourselves to just one
or two rays per pixel.
So, we need to use these rays as
efficiently as we can.
Now, one of the ways to do this
is importance sampling.
And the general idea here is to
fire rays in the directions
where we expect they'll
contribute most to our final
image.
Now with Ambient Occlusion the
most important rays are the ones
closer to the normal.
So instead of firing rays evenly
in a hemisphere like you see
here, we instead use cosine
sampling.
Now, this distributes fewer rays
around the horizons and more
rays around the surface normal.
And that's great.
That's exactly where we need
them.
Now in addition to this angular
falloff, Ambient Occlusion also
has a distance term.
So, objects close to the
surface, they tend to block the
most light.
And there's usually a fall off
function in there too,
proportional to the square of
distance.
Now, interesting thing we can do
here is bake that fall off
function right into the ray
distribution itself.
And the way we do this is by
firing rays of different
lengths.
So, as you can see here, because
of that distance squared fall
off function I was telling you
about, the majority of rays end
up being quite short.
Now this is great for
performance.
Short rays are much easier for
Metal to trace through the
acceleration structures.
So, a couple of times now, I've
talked about generating rays in
various shapes and various
distributions, such as the cones
we were using for Soft Shadows
and the hemisphere's that we're
using for Ambient Occlusion.
Now, the way that this works in
practice is we begin by
generating points in 2D
parameter space and then we map
that space with whichever ray
distribution you want to use.
Now the position of these points
in parameter space can have a
big effect on image quality.
If you choose them randomly, you
tend to end up with regions
where sample points clump
together.
Now this causes us to fire rays
in pretty much the same
direction and that's just
wasting rays.
You can also get areas without
any sample points at all.
Now this impacts image quality
because we're undersampling the
scene in these areas.
So, a better approach to
generate sample points is to use
something called a low
discrepancy sequence.
So, the one I'm showing you up
on the screen here is the Halton
2,3 sequence.
You can see that sample points
generated in this way, they
cover the space far more evenly
and we vanish the void by
plumping and undersampling.
So that's how to generate good
rays for a single pixel.
And what we need to do now is
scale that to generate good rays
for all pixels on the screen.
Now, the way that we're doing
this is by taking one of those
low discrepancy sample points I
was just showing you and then we
apply a random delta for each
pixel.
Now, the effect of that is that
each pixel still runs through a
low discrepancy sequence, but
the exact positions of the
sample points are offset from
neighboring pixels on the
screen.
Now, there's a couple of
different ways to generate these
deltas.
One way is just to sample an RG
texture full of random numbers.
What we saw previously that
random numbers aren't always a
great choice for Ray Tracing.
And an alternative that works
really well for Ambient
Occlusion is blue noise.
So, you can see on the right
there that the randomness is the
blue noise texture, it's
distributed far more evenly and
that's great for image quality.
Particularly when we're limited
to just a couple of rays per
pixel.
So, let's look at the effect of
all of this on the Ambient
Occlusion result that we were
trying to generate.
So, here's what we started with.
This is using hemisphere
sampling and random deltas for
all pixels.
And this is what we get with
cosine sampling and the blue
noise that I was telling you
about.
So, I'll flip between these
images so you can see.
Now, both of these images are
generated using just two rays
per pixel.
But you can see by being
selective about how we use those
rays, the amount of noise is
significantly reduced.
And we've managed to capture
much more of the fine surface
detail.
And if we were to keep firing
rays, eventually the two
approaches would converge on
exactly the same image.
But using importance sampling
gets us there much faster.
So that's Shadows and Ambient
Occlusion.
And for these effects we were
really only interested in
whether our rays hit something
or whether they missed.
Now, for many of the other
effects that we typically
associate with Ray Tracing, such
as Global Illumination, you need
to model your rays as they
bounce around the scene.
And to talk some more about that
I'll invite up my colleague
Matt.
[ Applause ]
>> Thanks Wayne.
So, we're going to cover a few
topics in this section, starting
with a brief overview of Global
Illumination.
Then, we'll go into some best
practices for memory and ray
management.
Finally, we'll cover some
strategies for debugging your
Ray Tracing application.
So, what is Global Illumination?
Conceptually it's pretty simple.
Light enters the scene and
directly illuminates the
surfaces that it hits.
And rasterization, that's
typically the end of the
rendering process.
But in the real world, those
objects absorb some of the light
and then the rays bounce off and
keep traveling around the scene.
And as they bounce around, some
interesting visual effects
emerge.
After light has bounced once we
start to see specular
reflections on the mirrored
surfaces like the ball and wall
to the right.
You can also see that objects
and shadows get brighter as they
pick up light that's been
reflected off nearby surfaces.
After the light has bounced
twice, we start to see
reflections between mirrored
surfaces and eventually, some
rays have refracted all the way
through transparent objects, and
they're showing the surfaces
behind them giving us the glass
effect of the box.
Now, if we tried to model all
the light bouncing around the
scene only a small portion of it
would actually make it back to
the camera and that would be
pretty inefficient.
So instead, we'll work backwards
from the camera towards the
light source.
We cast rays from the camera
towards the pixels in our image.
The intersection points of those
rays tell us what objects are
visible.
But we'll need to figure out how
much light is reaching them in
order to figure out what their
color in the final image should
be.
Earlier, Wayne described how to
calculate soft shadows and here
we're going to be performing
exactly the same process.
We cast shadow rays from the
intersection points towards the
lights in the scene in order to
approximate how much light's
reaching them.
That's used as the light
contribution towards the final
image.
Next, from the intersection
points we cast secondary rays in
random directions.
We use Metal to figure out what
those rays hit and then cast
shadow rays to determine their
direct lighting and then use
that to add light to the final
image.
By repeating this process, we
can simulate light bouncing
around the room.
We described this extensively in
last year's talk, so I'll refer
you to that for more details on
how to go through this process.
Our pipeline for this will look
a little bit different than the
hybrid pipelines that we've seen
so far.
First, we set up rays and use
Metal to find their
intersections with the scene.
Then, we write a shader to
process the results of those
intersection tests to tell us
what surfaces we hit.
Then, we generate shadow rays
from those intersection
locations towards the lights in
the scene.
I'll write a shader to figure
out which of those rays hit the
light and then add their light
to the final image.
Finally, we use the hit surfaces
as the starting positions for
our next set of rays.
We repeat this process over and
over again until we've modeled
as many ray bounces as we like.
So that's how Global
Illumination works.
Now we'll discuss some best
practices that come up with
memory for this programing
model.
As any ray bounces around the
scene.
Its state changes depending on
its interactions with the
objects that it hits.
For instance, if a ray hits a
red material that surface
absorbs everything but the red
component of the light.
So, the secondary rays that
reflect off of that surface will
only carry red light.
So, we'll have to keep track of
that information in order to
pass it to the next iteration of
our pipeline.
That means we'll have to
allocate a bunch of resources to
keep track of ray and scene
properties.
To the right, I've listed a few
examples of scene properties you
may want to keep track of.
With all these new buffers
relocated we're going to be
using a lot of memory.
For a 4K image, the ray buffer
alone would be 250 MB.
In one of our demos, we're using
almost 80 bytes per ray.
And this approach can quickly
exceed the amount of available
GPU memory.
One solution to this is just to
batch up your rays into smaller
groups or tiles.
And by restricting the number of
rays that you're launching
simultaneously you can
drastically reduce the memory
footprint of your resources.
Because the data in these
buffers is going to be passed
between pipeline iterations,
storing that data out and then
reading it in, in the next pass
is going to be a major limiting
factor.
For 4K image we're using over 8
million rays.
And for that number of rays,
we're reading and writing almost
5 gigabytes of data per pass.
There's no one solution to every
bandwidth problem, but we can
give you some best practices
that worked well for us.
First, don't index into your
data buffers randomly.
It's much more efficient if you
can index by thread ID, so the
compiler can coalesce all of the
loads and stores since the
memory the threads are accessing
will be in adjacent buffer
positions.
This is really going to improve
your cache coherency.
Next, for variables where you
don't need full precision,
consider using smaller data
types where possible.
Try to use half instead of float
data types for ray and scene and
material properties if you can.
Finally, split up structs if
possible, to avoid loading or
storing data you're not going to
use.
For example, we have a struct
that contains material
properties.
We get a nice performance boost
by stripping out the
transparency variables because
not every ray hits a transparent
surface.
And we didn't want those rays to
have to pay the penalty for
reading and writing that data.
We're going to see a more
concrete example of this later
in the GPU debugging section.
It might be counter intuitive,
but it may be more efficient to
allocate your own buffers to
store origin and direction data
rather than reusing the Metal
ray buffer structs.
This is because the Metal ray
buffers may contain extra data
numbers that you don't want to
have to load and store for every
shader that may access the ray.
To maximize your GPU usage, you
need to be mindful of your
shader occupancy.
Occupancy is a huge topic, so we
won't go into it in depth here.
But if you are getting occupancy
problems the easiest way to
improve it is to reduce your
register pressure.
So, be conscious of the number
of simultaneously live variables
that you have in your shader.
Be careful with loop counters,
function calls, and don't hold
on to full structs if you can
avoid it.
When we process ray intersection
points we need to evaluate the
surface properties of whatever
object a ray hits.
And graphics applications
typically store a lot of the
material properties in textures.
The problem here is that because
a shader may need access to any
texture that an object
references, and we don't know in
advance what object a ray is
going to hit, we may potential
he need access to every -- every
texture that's in the scene.
And this can get out of hand
quickly.
For instance, the commonly used
Sponza Atrium scene has 76
textures, which is over double
our available number of bind
slots.
So, we'll pretty quickly run out
of binding locations.
One way to address this is by
using Metal Argument Buffers.
A Metal Argument Buffer
represents a group of resources
that can be collectively
assigned as a single argument to
a shader.
We gave a talk on this at WWDC
two years ago.
So, I'll refer you to that for
more details on how to use them.
Assuming we have one texture per
primitive, our argument buffer
will be a struct that contains a
reference to a texture.
Here, we've set up a struct that
we called material that contains
a texture reference and any
other information we'd like to
access.
Next, we bind our argument
buffer to a compute kernel.
It will appear as an array of
material structs.
We read from our intersection
buffer to find out what
primitive the ray hit and then
we index into our argument
buffer using that index.
That lets us access a unique
texture for every primitive.
That covers our memory topics.
Now we'll discuss managing the
lifetime of your rays.
As your ray bounces around the
scene it can stop contributing
to the final image for a variety
of reasons.
First, it may leave the scene
entirely.
Unlike the real world, your
scene takes up a finite amount
of space, and if the ray exits
it there's no way for it to make
its way back in.
If that happens, we typically
evaluate an environment map to
get a background color, but that
ray is effectively dead.
Second, as the ray bounces, the
light it carries will be
attenuated by the surfaces is
interacts with.
If it loses enough light it may
not be able to have a measurable
impact on the final image.
And finally, with transparent
surfaces, there are some rays
that can get trapped in
position, such that they can
never get back to the camera.
So how quickly to rays become
inactive?
Well, depending on the kind of
scene it can be quite rapid.
For example, this scene has an
open world and a lot of the rays
will exit quickly by hitting the
environment map.
On the right, we're showing a
simplified representation of a
fully active ray buffer as it
would exist for the first
iteration of our pipeline.
This is the step where we cast
rays from the camera towards the
scene.
Some of those rays will hit the
environment map and become
inactive.
Here, we've colored inactive
rays yellow and we've removed
them from the ray buffer.
After the first pass only 57
percent of our rays are still
active.
When we let the rays continue
traveling, some of the rays that
initially hit the ground bounce
up and hit the environment map.
Now we're down to 43 percent of
rays that -- left active.
Now, some of the rays travel all
the way through the transparent
objects and eventually exit the
scene.
We've only got about a third
remaining active.
And of course, the more we
iterate, the more rays become
inactive.
In this example we know a lot of
the rays in our ray buffer will
be inactive, and anytime we
spend processing them will be
wasted.
But because we don't know in
advance which rays are going to
be inactive, the Metal Ray
Intersector and all of the
shaders that process its results
are still going to have to be
run on all of them.
That means we'll have to
allocate thread group memory,
the compiler may be prefetching
data, and we may have to add
control flow statements to cull
inactive rays.
Our occupancy here stays the
same, but our thread groups have
become sparsely utilized.
And we're wasting all that
processor capacity.
Our solution to this is to the
compact the ray buffers.
For every pass we only add
active rays to the next passes
ray buffer.
This adds some overhead, but it
results in the cache lines and
thread groups being fully
utilized so there's less waste
of processing and less bandwidth
required.
It's also important to note that
this can be used for shadow rays
as well.
Some rays will hit surfaces that
are pointing away from a light
or they may hit the background.
So, we won't want to cache
shadow rays for them.
The downside is that because
we're shuffling ray positions
within the ray buffers, the
indices in our ray buffer no
longer map to constant pixel
locations.
So, we'll need to allocate a
buffer to start tracking pixel
coordinates along with each ray.
Even though we're using extra
buffers, we'll actually be using
much less memory if we factor in
all of the rays that we don't
have to process.
When we're compacting the rays,
we don't want two shaders to try
to add rays to the new ray
buffer at the same location.
Therefore, we need to produce a
unique index into the ray buffer
for every ray that remains
active between passes.
We use an atomic counter to do
this.
Here, the atomic integer
outgoingRayCount contains the
current number of rays in the
new ray buffer.
We use atomic fetch add explicit
to grab the current value of
outgoing ray count and increment
it by one.
Using that value as the index
into the outgoing ray buffer
ensures that we don't get
conflicts.
It has the added benefit of
leaving the number of rays that
remain active in outgoing ray
count.
Now, Ray Compaction doesn't help
you much if you can't restrict
the number of threads that
you're launching.
The outgoing ray count buffer we
just produced contains the total
number of active rays in our
outgoing ray buffer.
We can use that to fill out an
MTLDispatch ThreadGroups
IndirectArguments object.
That just specifies launch
dimensions to be used with the
dispatch.
Then, by using IndirectDispatch
with that indirectBuffer object
we can restrict the number of
threads that we're launching to
only process the rays that
remain active.
There's also a version of the
ray intersection function that
corresponds to this.
The important point here is that
we can pass our ray count via a
buffer, so we can feed the
result of our ray compaction
step in as the number of threads
to launch.
After Ray Compaction we get
about a 15 percent performance
gain in this scene.
But of course, your results will
depend on the complexity of the
scene you're using and the
number of ray bounces.
So that covers ray lifetime and
culling.
Now we're going to discuss
debugging your application with
Xcode.
Debugging on the GPU is
notoriously difficult process.
And this is especially true for
Ray Tracing.
Any change you make may get
invoked multiple times per ray
and you might have to write a
lot of code to dump out buffers
and textures for a bunch of
different stages of your
algorithm to get a clue about
where an error was introduced.
Xcode's frame capture tools are
amazing for debugging exactly
these kinds of issues.
It's incredibly powerful and a
real time saver.
So, I'm going to walk you
through debugging a real-world
issue that we hit when we
implemented super sampling in
our ray tracer.
We implemented the ability to
sample a single pixel multiple
times per frame and all of a
sudden, our ray tracer is
producing a blown-out image.
The first step is just to do a
frame capture as your
application is running.
This records the state of the
GPU for every API call and
shader over the course of a
frame.
By selecting any shader, we can
examine the resources that are
bound to it so we can really
quickly narrow down exactly what
shader's failing by just
selecting all of the shaders
that write to the frame buffer
and examining the frame buffer
contents directly.
So here we can see the first
image is pretty light.
The second image is pretty
washed out.
And the third is almost white.
But here we're going to select
the shader that outputs the
lightest image and we're going
to take a look at the two input
buffers that we used to
calculate the frame buffer.
And the first buffer just
contains the sum of all the
light a ray has accumulated.
The second buffer contains our
new variable.
And that's just the number of
times we've sampled a given
pixel.
Both of these buffers look like
they have valid data in them, so
we'll go directly to the shader
debugger to examine what our
shader is doing with this data.
Our color calculation is just
that some of the luminance for
all the rays that were launched
for a given pixel.
When we only had one ray per
pixel, this worked just fine.
But now, we're failing to
compensate for the fact that
we're sampling multiple times
per pixel.
So, we're going to change that
code in the shader debugger to
divide the total luminance by
the number of input samples.
We reevaluate directly in the
shader debugger and we can
instantly see that our output
image has been fixed.
It's just that easy.
Another issue we hit frequently
was trying to understand the
performance impact of our
changes.
Xcode frame captures tools make
this easy as well.
Here's an example of a struct
that tracks surface
characteristics across ray
bounces.
Not every surface in our scene
uses transparencies.
The final two values,
transmission and index of
refraction, won't be used for
some rays.
But, because we've packed all of
that data into a single struct,
rays that don't hit transparent
surfaces are still going to have
to pay the penalty for reading
and writing those fields out
between passes.
Here, we've refactored the index
of refraction variables into
their own struct.
By separating the structs only
rays that hit transparent
surfaces will have to touch the
refraction data.
But we can still do a bit
better.
Now we've changed all of our
variables to half data types to
save even more space.
We've reduced our memory usage
from 40 to 20 bytes, and rays
that don't hit transparent
objects will only need 12 bytes.
So how do we understand the
performance impact of this?
Here we grab GPU traces using
the frame capture tool both
before and after our change.
The most basic version of
performance analysis we can do
takes place at this phase.
By comparing the shader timings
in our before and after
captures, we can isolate shaders
whose performance has changed.
Here, we can see that the shader
that we've labeled sample
surface has gone from 5.5
milliseconds to 4 milliseconds.
That's almost a 30 percent
savings for one of our more
costly shaders.
If we want to quantify exactly
why we're getting a performance
gain, Xcode helpfully displays
the results of all the
performance counters that it
inserts when it does a frame
capture.
Because we're interested in how
we've affected our memory usage,
we can take a look at the
texture unit statistics and we
see that our average texture
unit stall time has gone down
from 70 percent to 54 percent.
And we've reduced our L2
throughput by almost two-thirds.
Even more helpfully Xcode will
do some analysis of its own and
report potential problems to
you.
Here, it's telling us that our
original version had some real
problems with memory and that
our new version's performing
much better.
One more tip that you may find
useful, is that the compute
pipeline state had some
interesting telemetry as well.
Look at
MaxTotalThreadsForThreadgroup.
This is an indication of the
occupancy of a shader.
You should target 1024 as the
maximum and anything less means
that there may be an occupancy
issue that you can fix.
So that's debugging in Xcode.
It makes developing Ray Tracing
and Global Illumination
algorithms on the Mac platform
incredibly easy.
And now, here's Wayne to give
you a live demo of all of this.
[ Applause ]
>> Thanks Matt.
So, you may recognize this scene
from our platform State of the
Union session earlier this week.
To render it here, I'm using a
MacBook Pro along with four
external GPUs.
And everything you can see on
the screen there is being ray
traced in real time.
So, I can just take hold of the
camera and move around the
scene.
So, let's start over here.
You can see these great shadows
that we're getting with Ray
Tracing.
They're hard at the contact
points and they become softer
and softer as the distance to
the ground increases.
And remember, for these shadows
we're firing just one ray per
pixel and then we're using the
Denoiser that Sean was telling
us about to get this great
filtered result.
And this is all calculated
dynamically.
So, I can just take hold of the
light and move it around.
And I can see the effect of that
pretty much immediately.
There's a really great effect
going on here too.
If we fly over and look in the
window of the tram here, you can
actually see the reflections of
our shadows, and again, you can
see the shadow moving around as
I take control of the light.
And if we head over to this part
of the scene now, there's a
really great reflection effect
going on here.
So, if we look at the left most
tram you can see the reflection
of the tram behind this.
But you can also see reflections
in the windshield of the tram
behind us.
So, there's reflections within
reflections going on here and we
have to simulate a couple of ray
bounces per pixel to achieve
that effect.
So, I'm going to zoom out here a
bit now.
And of course, in this scene
it's not just the camera and the
lights that can move.
Sean was telling us about
Metal's Two-Level Acceleration
structures earlier.
And we're using those here to
enable the trams to move around
the scene.
What I really want to show you
now though is this fantastic
lighting effect we have going on
up on the roof.
So, if we focus on the wall on
the right there you can see that
currently it's being lit
primarily by direct sunlight.
But as I take control of the sun
and I rotate it around you can
see the wall falls into shadow
and now it's being lit by this
really great indirect
illumination.
So, what's going on here is
sunlight is hitting the roof on
the left and it's bouncing and
illuminating the wall on the
right.
Giving us this great color
bleeding effect.
And if I continue to rotate the
sun you can see these really
dramatic shadows start to come
in and they travel across the
surface of the roof there.
If I spin the camera around a
bit you can really see the
reflections as well hitting the
roof on the left.
So, this is -- I really like
this shot.
There's a lot of Ray Tracing
effects all going on at the same
time here.
We have the indirect
illumination, we have the
shadows, we have the
reflections.
And it's all being ray traced in
real time with Metal and multi
GPU.
So, I'm going to switch back to
the keynote now and it's this
multi GPU aspect that I'd like
to talk a bit more about.
[ Applause ]
So, for the demo that we just
saw, the way that we implemented
multi GPU was by dividing the
screen into a set of small tiles
and then we mapped these tiles
onto the different GPUs.
Now in the visualization here
I'm using different colors to
show you how the tiles are
distributed.
So, one GPU renders the tiles in
red.
Another does the tiles in
yellow, and so on.
And after all GPUs are finished,
we just composite those results
together to form our final
image.
So, if we take a step back and
look at what we have here
there's two things that jump
out.
So firstly, in the image on the
left, so the way that we're
assigning tiles to GPUs, it
looks a bit strange.
So why are we doing it like
that?
And secondly, for those small
tiles the implication of that is
that each GPU is going to render
a block of pixels there and then
a block of pixels somewhere
else.
And that just feels like it's
going to be bad for things like
ray coherency, cache hit ratios.
All that kind of stuff.
So, let's deal with these in
turn.
So, imagine we have four GPUs.
The simple way to do multi GPU
here is just to split the screen
into quadrants.
Now the issue with that is that
some parts of a scene will be
much easier to render than
others.
So, if we assume that the street
and the building on the left
there are much easier to render
than the tram on the right.
It stands to reason that the red
and yellow GPUs will finish
before the green and purple
GPUs.
Now we can fix this just by
splitting the screen into
smaller tiles.
Then we can split each of those
into smaller tiles and so on.
Until we reach some minimum tile
size.
Now this has the effect of
distributing work really evenly
across the GPUs.
So, if one part of the screen is
particularly difficult to
render, it just doesn't matter,
because every GPU will be
assigned tiles from that part of
the screen.
Now in practice, this regular
tiling pattern that you see here
probably isn't the way to go.
Because you can get cases where
the tiling aligns with the
geometry in your scene.
And so, we randomize a bit.
And I'll give a few more details
on how we're doing that in a
second.
And one of the really
interesting things about this
approach is that the mapping of
tiles to GPUs, it does not
change.
So, the same GPU will process
the same tiles every frame.
And this is great.
So, you can just calculate that
mapping when your application
initializes or when you resize
the window and that's it.
You don't need to think about
multi GPU load balancing anymore
and there's nothing to monitor.
Nothing to recalculate in your
application's main loop.
So, if we know that small tiles
distribute the work more evenly,
why not just take it to the
extreme and make them a pixel.
So, the problem with that is
that we need to give each GPU
nice coherent blocks of pixels
to be working on.
So, the tradeoff there between
balancing the load evenly and
making sure that each GPU can
run as efficiently as possible.
So, to better understand that
tradeoff we did a simple
experiment.
We took one of the new Mac Pros
with a pair of the Vega II Duo
GPUs, so that's four GPUs in
total, and we tried rendering
the same scene with various
tiles sizes to see how that
effected performance.
Now of course, your mileage may
vary here, but what we found is
that the performance window is
actually really wide.
So, efficiency drops off if you
make your tiles very small or if
you make them very, very large.
But anywhere in the middle keeps
us pretty close to peak
performance.
So now we have our tile size
pinned down, what we need to do
next is assign them to the
various GPUs.
Now to do this, we start by
generating a random number for
each tile and then we compare
those random numbers to a set of
thresholds.
And whichever range the random
number lands in, that gives us
the GPU to use for that tile.
So as an example, here, if the
random number is .4, we assign
it to GPU 1.
If it's .55, it goes on GPU 2.
And so on.
Now, once we've done this for
every tile the output is a list
of tiles that we need each GPU
to render.
As you can see down the bottom
there the ranges that we're
using for each GPU are equal.
So, when assigning tiles to GPUs
they are all equally likely to
be chosen.
But in practice, you almost
certainly don't want this.
For example, you might need to
reserve capacity on one of your
GPUs for non-ray tracing tasks
such as denoising or tone
mapping.
Or you might be using GPUs with
different performance.
In which case you'll want to
send more tiles to the more
powerful GPUs.
And you can account for this
really easily by just adjusting
the ranges.
So now if we go ahead and
reassign the same tiles we used
before, you can see here that
now GPU 2 takes on a much
greater share of the work.
Now, for the actual implantation
of this there was a lot of
really useful information in our
Metal for Pro Apps session
earlier this week.
So, I won't go over that again
here.
But it is definitely useful to
highlight just a couple of areas
that can have a really big
impact on performance.
So firstly, you'll probably want
to composite your tiles together
on the GPU that's driving the
display.
So, it's important to find out
which GPU that is and then work
backwards to figure out how to
get your data there efficiently.
So, if the GPUs are in the same
peer group then you can copy
between them directly using our
new peer group APIs.
Otherwise you'll need to go by
the CPU.
Now secondly, it can often take
a few milliseconds to copy data
between GPUs and we definitely
don't want to block waiting for
those transfers to complete.
So, to give you an example of
how we're dealing with that we
have two GPUs here and we're
using the tiling scheme that I
was just talking about to spread
the rendering across the two
GPUs.
Now in GPU 0 at the top there we
have two queues.
One is just flat out doing
back-to-back Ray Tracing.
And then we have a second queue
that copies the completed tiles
over to GPU 1 asynchronously.
Now, we'll assume that GPU 1 at
the bottom there is the one
that's driving our display.
And here things are a bit
different.
This GPU is also Ray Tracing
part of frame 0, but we can't go
ahead and present that frame
until the rest of our tiles have
been copied over from the other
GPU.
So rather than wait, we just
start work on the next frame.
And then a bit later on when our
tiles arrive from the other GPU,
that's when we go ahead and
composite everything together.
So, I'll show you that one more
time here.
So, you can see that we end up
in this steady state where we
render frame N and then we
composite frame N minus 1.
So essentially, what we're doing
here is latency hiding.
And this together with the
tiling scheme I was showing you
to load balance between the
GPUs, this is enabling us to
achieve really great performance
for our Ray Tracing workloads on
our multi GPU systems.
And with that, we come to the
end of the talk.
We began with a quick refresher
of how Ray Tracing works in
Metal and then we focused on a
few features of the
MPSRayIntersector that are there
to really help with dynamic
scenes.
So that's the Two-Level
Acceleration Structures along
with our GPU Accelerated
Rebuilds and Refitting.
We also introduced the new Metal
Denoiser.
And then we talked through a few
Ray Tracing use cases such as
Shadows, Ambient Occlusion, and
Global Illumination.
When then showed you how to
debug and profile Ray Tracing
workloads using Xcode.
And then, we finished by talking
about how to take advantage of
multiple GPUs in your Ray
Tracing applications.
Now, for more information be
sure to visit
developer.apple.com and there
you'll also find a new sample
demonstrating some of the
features that we've talked about
today.
If you're new to Ray Tracing be
sure to check out our talk from
last year.
And finally, we have our lab
session coming up next at 12.
So, I hope you can join us for
that.
So, thank you all for coming and
I'll see you in the lab shortly.
[ Applause ]