Transcript
[ Music ]
[ Applause ]
>> Lionel Lemarie: Hi, folks.
Welcome to our Metal session.
I'm Lionel.
I'm in the GPU software
performance team here at Apple.
And with my friends Max and
Sarah, we'll be guiding you
through how to bring your OpenGL
app to Metal.
So last year we announced that
OpenGL, OpenGL ES, and OpenCL
are deprecated.
They will continue to be
supported in iOS 13 and macOS
Catalina, but now is the time to
move.
New projects should target Metal
from their inception.
But if you have an OpenGL app
that you want to port to Metal,
you've come to the right place.
So we first introduced Metal in
2014 as our new low-overhead,
high-efficiency,
high-performance GPU programming
API.
Over the past five years,
Apple's core frameworks have
adopted Metal and they're
getting really great results.
If your application is built on
top of layers like SpriteKit,
SceneKit, RealityKit, Core
Image, Core Animation, then
you're already using Metal.
We've also been working closely
with vendors on engines like
Unity, Unreal Engine 4, and
Lumberyard to really take
advantage of Metal.
If you're using one of these
engines, you're already up to
speed.
But if you've built your own
renderer, then Metal gives you a
lot of great benefits.
Metal combines the graphics of
OpenGL and compute of OpenCL
into a unified API.
It allows you to use multithread
rendering in your application.
Whenever there are CPU
operations that need to take
place that are expensive, we
made sure that they happen as
infrequently as possible to
reduce overhead during your
app's execution.
Metal's shading language is C++
based and all the shaders used
in your application can be
precompiled, making it easier to
have a wide variety of material
shaders, for example.
And last but not least, we have
a full suite of debugging and
optimization tools built right
into Xcode.
So once you have ported to
Metal, you have full support to
make your application even
better.
So let's dive in.
In this session we'll take a
look at the different steps
involved in migrating from GL
into Metal, and we'll do that by
comparing a typical GL app to a
Metal app.
As an overview, let's quickly
look through the steps of our GL
app.
First, you set up a window that
you'll use for rendering.
Then you create your resources
like buffers, textures,
samplers.
You implement all your shaders
written in GLSL.
Before you can render anything
in GL, you may have to create
certain object states, such as
GL programs, GL frame buffer
objects, vertex array objects.
So once you've initialized your
resources, the render loop
starts and you draw your frames.
For each frame, you start by
updating your resources, bind a
specific frame buffer, set the
graphic state, and make your
draw calls.
You repeat this process for each
frame buffer you have.
You may have shadow maps, a
lighting pass, some
post-processing.
So potentially quite a few
render passes.
And then finally, you present
the final rendered image.
It's pretty easy.
And as you can see, the Metal
flow looks very similar.
We updated some of the original
concepts and introduced a few
new things.
But overall, the flow is much
the same.
It's not a complete rewrite of
the engine; it works in the same
manner.
So we will reintroduce the new
concepts while drawing parallels
between GL and Metal, comparing
and contrasting the two API's to
help you successfully make the
transition.
When you're walking through any
tutorial on graphics, then the
first thing you learn is how to
create and draw to a window.
So let's start with the window
subsystem.
Both GL and Metal have this
concept, but it's accomplished a
little differently.
The application is required to
set up and present a drawing
surface.
And view and view delegates
manage the interface between the
API and the underlying window
system.
You might be using these
frameworks to manage your GL
views, so we have equivalent
frameworks in Metal.
NSOpenGLView and GLKView map to
MTKView.
And if you are using Core
Animation in your application
with the EAGLLayer, then there's
an equivalent CAMetalLayer.
As an example, let's say you are
using GLKView.
It has a single entry point with
the draw rate.
So you need it to check if the
resolution of your target is
unchanged since the last frame,
update your render target sizes
as needed, right from within the
render loop.
In MetalKit, it's a bit updated.
There's a separate function for
whenever the drawable needs to
change, such as when you're
rotating the screen or resizing
your window.
So you don't need to check if
your resources need to be
reallocated inside your draw
function; it's dedicated to
render code.
If you need additional
flexibility, we provide the
CAMetalLayer, which you use as
the backing layer for your view.
While the CAEAGLLayer defined
the properties of your drawable
such as its color format, the
CAMetalLayer allows you to set
up your drawable size, pixel
format, color space, and more.
Importantly, the CAMetalLayer
maintains a pool of textures and
you call next drawable to get
the drawable to render your
frame to.
It's an important concept that
we'll revisit in a short while
when it's time to present.
So now we have a window.
Next we're going to introduce
some new concepts in Metal.
So the command queues, command
buffers, command encoders.
These objects work together in
Metal to submit work to the GPU.
They're new because the
underlying glContexts managed
the submission for you.
GL is an implicit API, meaning
that there is no code that tells
GL when to schedule the work.
As a developer, you have very
little control about when
graphics work really happens,
such as when shaders are
compiled, when resource storage
is allocated, when validation
occurs, or when work is actually
submitted to the GPU.
The glContext is a big
[state] machine, and a
typical workflow would look like
this.
Your application creates a
glContext, sets it on the
thread, and then calls arbitrary
GL comments.
The comments are recorded by the
context under the hood and would
get executed at some point in
time.
Let's take a closer look to see
what actually goes on.
Say your application just send
GL these calls, a few state
changes, a few draw calls.
In a perfect scenario, the
context would translate this
into GPU comments to fill up an
internal buffer.
And then when it's full, it
would send it to the GPU.
If you insert a glFlush to
enforce execution, you know for
sure they'll be kicked off by
that point.
But actually, the GPU could
start execution at any point
beforehand.
Alright.
So, for example, if we change
one draw call introducing every
dependency, suddenly execution
is kicked off at that point and
you could experience massive
stalls.
So, again, when does work
actually get submitted?
It depends.
And that was one of the
downsides of OpenGL -- wasn't
consistent in performance.
Any one small change could force
you down a bad path.
Metal, on the other hand, is an
explicit API, meaning the
application gets to decide
exactly what work goes to the
GPU and when.
Metal splits the concept of a
glContext into a collection of
internal working objects.
The first object an app creates
is a Metal device object, which
is just an abstract
representation of the GPU.
Then it creates a key object
called a Metal command queue.
The Metal command queue
maintains the order of commands
sent to the GPU by allocating
command buffers to fill.
And a command buffer is simply a
list of GPU commands your app
will fill to send to the GPU for
execution.
So we saw this command buffer
concept in GL -- in the GL
example we just studied.
Let's work with that command
buffer from this point on.
But an app doesn't write the
commands directly to the command
buffer; instead, it creates a
Metal command encoder.
Let's look at the main three
types of encoders.
First one we'll use will be
filled with blit commands that
are used to copy resources
around.
The command encoder translates
API codes into GPU instructions
and then writes them to the
command buffer.
After a series of commands have
been encoded, for example,
series of blits to copy
resources, then your app will
end encoding, which releases the
encoder object.
Additionally, Metal supports a
compute encoder for parallel
work that you would normally
have done in OpenCL before.
You enqueue a number of kernels
that get written to the command
buffer and you run the
encoder to release it.
Lastly, let's use a render
encoder for your familiar
rendering commands.
You enqueue your state changes
and your draw calls and end the
encoder.
So here we have a command buffer
full of different workloads, but
the GPU hasn't done any work
yet.
Metal has created the objects
and encoded commands all within
the CPU.
It's only after your application
has finished encoding comments
and explicitly committed the
command buffer that the GPU
begins to work and executes
those commands.
So now that we have encoded
commands, let's now compare and
contrast GL and Metal's command
submissions.
In GL there's no direct control
of when work gets submitted to
the GPU -- you rely on big
hammers like glFlush and
glFinish to ensure code
execution; glFlush submits the
commands and poses the CPU
threads until they're scheduled,
and glFinish poses the CPU
thread until the GPU is
completely finished.
Work can still get submitted at
any time before these commands
happen, introducing potential
stalls and slowdowns.
And Metal has equivalent
versions of these functions; you
can still explicitly commit and
wait for a command buffer to be
scheduled or completed.
But these wait commands are not
recommended unless you
absolutely need them.
Instead, we suggest that you
simply commit your command
buffer and then add a callback
so that your application can be
notified later when the command
buffer has been completed on the
GPU.
This frees your CPU to continue
doing other work.
So now that we have reviewed
command queue, command buffer,
command encoder, let's move on
and talk about resource
creation.
There are three main types of
resources that any graphic app
is likely to use: Buffers,
textures, and samplers.
Let's take a look at buffers
first.
In GL, you have a buffer object
and the memory associated with
it.
The API codes you use can modify
the object state, the memory, or
both together.
So here, for example,
glBufferData can be used to
modify both the memory and the
state of the object.
The buffer dimensions can be
modified again later by calling
glBufferData, in which case the
old object and its contents will
be discarded internally by
OpenGL.
In Metal, the API to create and
fill a buffer looks very
similar, but the main difference
lies in the fact that the
produced subject is immutable.
If at any point you need to
resize the buffer, you simply
need to create a new one and
discard the old one.
Both OpenGL and Metal have ways
to indicate how you intend to
use an object; however, in GL
the enum is simply a usage hint
about how the data in a buffer
object would be accessed.
The driver uses that hint to
decide where to base the locate
memory for the buffer, but
there's no direct control over
storage.
OpenGL ultimately decides where
to store the objects.
In Metal, the API allows you to
specify a storage mode which
maps to a specific memory
allocation behavior.
Metal gives you control, since
you know best how your objects
are going to be used.
It's an important concept in an
object creation, so we'll come
back to it in a short moment
right after we look at texture
API's.
In GL, each texture has an
internal sampler object, an
app's commonly set up sampling
mode through that sampler.
But you also have the option to
create a separate sampler object
outside of your texture.
Here's an example for creating
and binding your texture,
setting up your sampler, and
then finally filling in the
data.
One thing worth mentioning is
that GL has a lot of API calls
to create initialized textures
with data.
It also has what are called
named resource versions of the
same API.
There's even more API's when it
comes to managing samplers.
The list just goes on and on.
One of the design goals with
Metal was to give a simpler API
that would maintain all of the
flexibility.
So in Metal, texture and sampler
objects are always separate and
immutable after creation.
To create a texture, we create a
descriptor, set various
properties to define texture
dimensions like pixelFormat and
sizes, amongst others.
Again, an important property we
said is the storage mode to
specify where in memory to store
the texture.
And finally, we use that
descriptor to create an
immutable object.
In a similar fashion, you start
with a sampler descriptor, set
its properties, and create the
immutable sampler object.
It's pretty easy.
To fill a texture's image data,
we calculate the bytes per row.
And just like we did in OpenGL,
we specify the region to load.
Then we call the textures
replaceRegion method, which
copies the data into the texture
from a pointer we specify.
Once you load your first
texture, you're likely to
observe that it's upside down.
That's because in Metal the
texture coordinates are flipped on the
y-axis compared to GL.
And it's also worth mentioning
that Metal API's don't perform
any pixelFormat transformation
under the hood.
So you need to upload your
textures in the exact format
that you intend to use.
Now let's get back to storage
modes.
As mentioned, in GL the driver
has to make a best guess on how
you wanted to use your
resources.
As a developer, you can provide
hints in some cases, like when
you created a buffer or by
creating render buffer objects
for frame buffer attachments.
But in all cases, these were
still hints and the
implementation details are
hidden from you.
A few minutes ago, we briefly
saw the additional storage mode
property Metal that you can set
on a texture descriptor and also
when creating a buffer.
Let's look at the main use cases
for those.
Simplest option is to use shared
storage mode, which gives both
the CPU and GPU access to the
resource.
For buffers, this means you get
to point here to the memory
backing of the object.
For textures on iOS, this means
you can call some easy-to-use
functions to set and retrieve
image data.
You can also use a private
storage mode, which gives the
GPU exclusive access to the
data.
It allows Metal to apply some
optimizations that it wouldn't
normally have been able to use
if the CPU had access to it.
But only the GPU can directly
fill the contents of the data.
So you can indirectly fill the
data from the CPU by using a
blitEncoder from a second
intermediate resource that uses
shared storage.
On the voices with dedicated
video memory, setting the
resource to use private storage
allocates it in video memory
only, single copy.
On macOS there's a managed
storage mode which allows both
the CPU and GPU to access an
object's data.
And on systems with dedicated
video memory, Metal may have to
create a second mirrored memory
backing for efficient access by
both processes.
So because of this, explicit
codes are necessary to ensure
that your data is synchronized
for CPU and GPU access, for
example, using didModifyRange.
So to recap, we reviewed some of
the typical uses for each mode.
On macOS you would use the
private storage mode for static
assets and your render targets.
Your small dynamic buffers could
use the shared storage mode.
And your larger buffers with
small updates would use the
managed storage mode.
On iOS, your static data and
rendering targets can use the
private storage mode.
And since our devices use
unified memory, dynamic data of
any size can use the shared
storage mode and still get great
performance.
Next, let's talk about
developing shaders for your
graphics application and what
API's you use to work with
shaders.
When it comes to shader
compilation in GL, you have to
create a shader object, replace
the ShaderSource in the object,
make just in time compilation,
and verify that the compilation
succeeded.
And while this workflow has its
benefits, your application had
to pay the performance costs of
compiling all your shaders every
time.
One of the key ways in which
Metal achieves its efficiency is
by doing work earlier and less
frequently.
At build time, Xcode will
compile all the Metal
ShaderSource files into a
default Metal library file and
place it in your app bundle for
retrieval at runtime.
So this removes the need to
compile a lot of it at runtime
and cuts the compilation time
when your application runs in
half.
All you need to do is create a
Metal library from a file
bundled with your application
and fetch the shader function
from it.
In GL you use GLSL, which is
based on the C programming
language.
The Metal shading language or
MSL is based on C++.
So it should look reasonably
familiar to most GL developers.
Its foundation in C++ means that
you can create classes,
templates, and stretches.
You can define enums and
namespaces.
And like GLSL, there are
built-in vector and matrix
types, numerous built-in
functions and operations come in
and use for graphics.
And there are classes to operate
on textures that specify sampler
state.
Like Metal, MSL is also unified
for graphics and compute.
And finally, since shaders are
pre-compiled, Xcode is able to
give you errors, warning, and
guidance to help you debug at
build time.
So let's take a look at actual
code for MSL and compare it with
GLSL.
We're going to walk through a
simple vertex shader, GLSL on
top, MSL on the bottom.
Let's start defining our
shaders.
These are the prototypes.
In GLSL, void main.
There's nothing in the shader
that specifies the shader stage.
It's purely determined by the
shader type passed into the
glCreateShader call.
In MSL the shader stage is
explicitly specified in the
shader code.
Here the vertex qualifier
indicates that it will be
executed for each vertex
generating perfect examples.
In GLSL, every shader entry
point has to be called main and
accept and return void.
In MSL each entry point has a
distinct name.
And when you're building shaders
with Xcode, the compiler can
resolve include statement in the
preprocessing stage the same it
would do for regular C++ code.
At runtime you can query
functions by their distinct name
from the precompiled Metal
library.
Then let's talk about inputs.
Because each entry point in GLSL
is a main function with no
argument, all of the inputs are
passed as global arguments.
This applies to both vertex
attributes and uniform
variables.
In Metal all the inputs to the
shaded stage are arguments to
the entry function.
The double brackets declare C++
attributes.
We'll look at them in a second.
One of the inputs here that we
have is a model view projection
matrix.
In OpenGL, your application had
to be aware of the GLSL names
within the C++ code in order to
bind data to these variables.
And that made shader development
error-prone.
In MSL the uniform binding
indices are explicitly
controlled by the developer
within the shader, so an
application can bind directly to
a specific slot.
In the example here, slot number
one.
The keyword constant here
indicates that the intention for
the model view projection is to
be uniform for all vertices.
The other input to the shader is
a set of vertex attributes.
In GLSL you typically use
separate attribute inputs.
The main difference here is that
MSL uses a structure of your own
design.
The staging keywords suggest
that each invocation of the
shader will receive its own
arguments.
Once you have all the inputs to
the shaders set up, you can
actually perform all the
calculations.
Then for the outputs, in GLSL
the output is split between
varying attributes like
glTexCoord and predefined
variables, in this case gl
Position.
In MSL, the vertex shader output
is combined into your own
structure.
So we've used a vertex and
vertex output structure.
Let's scroll up in the MSL code
to see what they actually look
like.
As mentioned previously, GLSL
defines the input vertex
attributes separately, and Metal
allows you to define them within
a structure.
In MSL there are a few special
keywords for vertex shader
input.
We mark each structure member
with an attribute keyword and
assign an attribute index to it.
Similar to GLSL, these indices
are used in the Metal API to
assign the vertex buffer streams
to your vertex attributes.
And GLSL predefines special
keywords like GL position to
indicate which variable contains
vertex coordinates that have
been transformed with the model
view projection matrix.
Similarly, for the vertex
output, a structure in MSL, the
special keyword position signals
that the vertex shader output
position is stored in that
structure member.
Similar to GLSL vector type, MSL
defines a number of simd types
via the simd.h header that can
be shared between your CPU and
GPU code.
But there's a few things you
need to remember about them.
Vector and matrix types in your
buffers are aligned to 16 bytes
or 8 bytes for half precision.
So they're not necessarily
packed, for example, a float3
has a size of 12 bytes but is
aligned to 16 bytes.
This is to ensure that the data
is aligned for optimal CPU and
GPU access.
There are specific backed
formats you can use if you need
them.
But you will need to unpack them
in the shader before using them.
So we've just reviewed the main
differences between GLSL and
MSL.
And to make this transition
smooth and easy, my colleague
Max will show you a really cool
tool to help you breeze through
it.
Thank you.
[ Applause ]
>> Good evening.
Metal, it's not just an API and
a shading language, it is also a
powerful collection of tools.
My name is Max, and I'm going to
minimize your hassle porting to
Metal.
Let's take a look at this scene.
This is the very first draw call
from an old OpenGL demo that we
here at Apple also ported to
Metal.
It's drawing a model of a temple
and a tree, both illuminated by
a global light source.
Let's port the fragment shader
together.
So the very first thing I did, I
just copy and pasted my entire
old OpenGL code directly into my
Metal shader file.
Based on this, I've already
created my input structure, as
well as my function prototype.
Let's begin.
So what we are going to do is
just copy and paste the contents
of the main function directly
into our Metal function.
And here we see the very first
powerful thing about Metal.
Because the shader's
precompiled, we are getting
errors instantly.
Let's take a closer look.
Of course, the building vector
types have different names now.
So vec2 becomes a float2; the
vec3 becomes the float3; and the
vec4 becomes a float4.
So we quickly fix that.
The next error we are going to
see is that like all of our
input structures -- all of our
global variables are now coming
from our input structure.
And because I just used a
similar naming scheme, this is
also very easy.
And, of course, we have to do
the exact same thing for our
uniforms.
The next error is a little bit
more complex.
Sampling in Metal is different,
so let's take a look.
We are going to start from
scratch.
So we directly can call a sample
function on our colorMap.
And here we can see how powerful
it is to have full auto
completion.
So this function expects us to
put in a sampler and a texture
coordinate.
We already have the texture
coordinate.
We could pass in the sampler as
an argument to our function or,
conveniently in Metal, we can
just declare one in code like
this.
We need to do the exact same
thing for our normalMap.
The last error that we are
seeing is that we are writing
into, like, one of many OpenGL
magic variables.
Instead, we are just going to
return our final computed color.
We can also see that all the
other functions, like normalize,
dot product, and my favorite
function max, are still exactly
the same.
Our shader now compiled
successfully.
Let's run it.
Something went wrong.
In OpenGL when you're
experiencing an error with your
shader, what you usually do is,
like, you look at your source
code, you look at your output,
and you think really hard.
We're just going to use the
shader debugger instead.
Clicking on the little camera
icon in the debug area will
capture a GPU trace.
This is a recording of every
Metal API call we made.
And we can now navigate to our
draw calls.
Here we are drawing the tree.
And here we are drawing the
temple.
Let me long press on the stairs
of the temple to bring up the
pixel inspector, which allows us
to start the shader debugger.
What we are seeing here now is
the values per line for the code
that we have ported together and
for the pixel we have just
selected.
Let's take a look at our
colorMap first.
We can see this looks like a
reasonable texture.
And we can also see that our
stairs are, like, in the upper
half of this texture; however,
if we were taking a look at our
texture coordinate, we can see
that we are sampling from the
lower half.
Let me quickly verify if this is
the case.
What we are going to do is to
invert the y coordinate of our
texture.
We can now update our shaders --
looks reasonable -- and we can
continue our execution.
There, much better.
This is a pretty common error
that you will experience when
porting from OpenGL to Metal.
And, of course, the real fix is
you go into your texture loading
code and make sure your texture
is loaded at the right origin so
you don't have to do this fix in
every shader.
However, the combination of a
feature-rich editor and mighty
debugging tools will also help
you port in your games to Metal
finally.
Thank you very much.
My colleague Sarah will now
guide you through the rest of
the slides.
[ Applause ]
>> Sarah Clawson: Thanks, Max.
Hi, I'm Sarah Clawson.
And I'm here to take you through
the rest of the port from GL to
Metal.
So far in the life of a graphics
app, we've gone through a lot of
setup.
We've got a window to render to,
a way to get your commands to
the GPU, and a set of resources
and shaders ready to go.
Next up, we're going to talk
about setting up the state for
your render loop.
OpenGL has several key concepts
when it comes to state
management.
The vertex array object defines
both the vertex attribute
layout, as well as the vertex
buffers.
The program is a link
combination of vertex and
fragment shaders.
And the framebuffer is a set of
color and depth stencil
attachments that your
application intends to render
to.
These state objects are created
during initialization and are
used throughout your frames.
Let's walk through an example to
show how OpenGL manages state.
Here we have a sample render
loop where an OpenGL application
binds a framebuffer, sets a
program, and then makes other
state modifications, like
enabling depth, or face culling,
or changing the colorMap before
making a draw call.
If you look at this same API
trace from OpenGL's perspective,
it has to track all these
changes on each API call.
And then when a draw call
happens, it has to stop and
validate to be sure that the
previous changes to primitive
assembly, depth state,
rasterizer, and programmable
stages are all compatible with
each other.
This validation can be super
expensive.
And while OpenGL does try to
minimize its negative impact,
there's limited opportunity to
do so.
It is worth noting that the open
OpenGL state objects were ahead
of the curve when they were
first introduced.
Framebuffer objects combine
attached render targets,
programs linked fragment and
vertex shaders together, and
vertex array objects were larger
objects combining some of the
vertex attribute API's and
vertex buffer setup.
But even with all these changes,
although they yielded positive
results, OpenGL still has to
validate many things on a draw
call, such as will the -- can
the ColorMask help optimize the
fragment shader?
Is the fragment shader output
compatible with the attached
frame buffer?
Is the vertex layout compatible
with the bound program?
Or are the attached render
targets blendable?
So as we redesigned the graphic
state management for Metal, we
took the program shaders
combined with the vertex input
layouts from the VertexArray
objects and added the
information about attachment
pixelFormat and blend state, and
we combined them into one object
called the PipelineDescriptor.
This structure describes all the
relevant states in the graphics
pipeline.
To set up the descriptor, first
you initialize it.
And then you set all the state
we just talked about, like
vertex and fragment shaders,
vertex information, pixel
formats, and blend state.
And then you take that
descriptor and you create what
is called a pipeline state
object or PSO.
This immutable object fully
describes the render state.
And what's great about it is
that you create it once, have it
validated for correctness, and
then use it throughout your
program.
In a similar way, we combined
all the depth and
stencil-related settings into a
depth/stencil state descriptor.
And, again, it is a collection
of all the depth/stencil state.
And you take this descriptor and
you create what's called a
depth/stensil state object.
This object is also immutable
and used throughout your
program.
So the render loop we were
looking at in OpenGL now looks
like this in Metal.
With all of the prevalidated
state objects, there's no longer
any state validation or
tracking.
Let's look through the
comparison.
In Metal, the render encoder is
the start of a render pass,
similar to binding your frame
buffer.
Now that your depth state is
prebaked into an object, you
simply set it on the
renderEncoder.
The PipelineState object
represents and combination of
program shaders, VertexArray
properties, and a pixelFormat.
And it's also set on the
renderEncoder.
And now the renderEncoder
manages your rasterizer state
directly.
And it's important to note here
that there is still flexibility
in your pipeline, as not
everything is prebaked into your
PipelineState object.
Here's the list of state that
we've just been discussing that
you prebake into your PSO: State
like vertex and fragment
functions and pixel formats,
etc.
On the other hand, here's all
the state that you still set
while drawing -- state like
primitive culling mode and
direction, fill mode.
Scissor and viewport areas are
still set just like in OpenGL.
And ultimately, the draw calls
remain the same.
The main difference here is that
instead of enabling new state,
which could incur hidden
validation costs, you simply
swap out a new PipelineState
object that had blending enabled
in its descriptor.
I want to discuss one more
possible optimization that you
may have used in OpenGL in order
to hide certain expensive
operations.
As an OpenGL developer, you may
have seen that your render loop
has an unexpected hiccup on the
first draw call after making a
bunch of state changes.
And if this is the case, you
probably use an optimization to
hide that called shader
pre-warming.
In shader pre-warming, an
application uses dummy draw
calls for the most common GL
programs in order to have OpenGL
create all the state that's
necessary ahead of time.
If you were doing this in your
engine already, then it's going
to be very easy for you to
replace it with PSO creation.
Now shader pre-warming in Metal
is accomplished through creating
separate PSO objects with
different state enabled.
First, you create your
descriptor, and then you set all
of the state up until the first
draw call and create your first
PipelineState object.
Then you can take that same
descriptor, change a bit of
state on it -- like here we're
enabling blending -- and you
create a second PipelineState
object.
Both of these are prevalidated
so that during draw time you can
just swap them out between draw
calls.
Hopefully if you're porting from
OpenGL to Metal, this is a
straightforward change.
Now, as we conclude the setup
stage of our application, I'd
like to bring up one of the main
benefits of porting your app
from OpenGL to Metal, and it is
that it will start doing
expensive operations less often.
In OpenGL, your application
would have to wait until draw
time in order to do things like
compile and link shaders or
validate states, which means
that these expensive operations
happen many times per frame.
Once you port your app to Metal,
your application moves these
operations to different stages
of its lifetime.
With precompiled shaders, shader
compilation has moved out of
initialization and into build
time so it's only done once.
Then with PSO's, state
definition is moved to content
loading.
So that leaves your draw time
free to actually make draw
calls.
So now that we've completed the
setup stage of your application,
let's talk about using all these
resources, shaders, and objects
to render frames.
In order to draw a single frame,
your application needs to first
update textures and buffers,
then establish a render target
to render to, and then make
several render passes before
finally presenting your work.
Let's talk about updating
resources.
Typically, at least some
resources have to be updated
continuously throughout your
render loop.
Such examples are shader
constants, vertex and index
buffers, and textures.
And these modifications can be
accomplished between frames
through synchronization between
the GPU and the CPU.
A typical GL resource update can
be any combination of the
following calls: A buffer can be
updated by the CPU; or you can
update a buffer through the GPU
via buffer-to-buffer copy.
Similarly, a texture can be
updated by the CPU or it can be
updated via texture-to-texture
copy on the GPU.
At a glance, Metal offers
similar functionality.
But as Lionel mentioned earlier,
the containers for buffers and
textures are immutable and are
created during initialization;
however, their contents can be
modified through any combination
of the following.
A buffer with shared or managed
storage mode can be updated
through its contents property on
the CPU.
And on the GPU, the blitEncoder
is in charge of doing all data
copying.
And so you can update a buffer
from the GPU via the
copyFromBuffer methods on the
blitEncoder.
Similarly, a texture with shared
or managed storage mode can be
updated on the CPU through its
replaceRegion method.
Or on the GPU, you can update a
texture through the
copyFromTexture methods on the
blitEncoder.
Note that storage mode matters
here when it comes to these
updates as only buffers and
textures with shared or managed
storage modes can be updated by
the CPU.
OpenGL managed the
synchronization between the GPU
and CPU for you, though
sometimes at exorbitant costs to
your application as it waited
for one or the other to be done.
In Metal, because you control
how the memory is stored, you
also control how and when the
data is synchronized.
And this is true for both
buffers and textures.
If you port your GL app to Metal
and only use a single buffer for
your resource updates, the flow
will look like this.
First, your CPU will update your
resources during the setup of a
render pass.
And then once complete, the
buffer will be available for the
GPU to consume during the
execution of that render pass.
However, while the GPU is
reading from this buffer, the
CPU may begin setting up for the
following render pass and will
need to update the same buffer,
which is a clear race condition.
So let's look at one approach to
solve this problem.
A simple solution would be to
commit this resource to the GPU
with the waitUntilCompleted call
on the commandBuffer it is used
in.
As we discussed earlier, this is
similar to glFinish and it
places a semaphore on all CPU
work until the GPU is done
executing the render pass that
uses that buffer.
After the execution is
completed, a call back is
received from the GPU, and this
way you can ensure that your
single buffer will not be
stomped on by the CPU or the
GPU.
However, as you can see, the CPU
is idle while the GPU is
executing, and the GPU is
starved waiting for the CPU to
commit work.
So while this can be helpful for
you at the beginning while
you're working out these race
conditions, it is not
recommended to use
waitUntilCompleted as it
introduces latency into your
program.
Instead, an efficient way to
synchronize your updates is to
use two or more buffers
depending on your application's
needs so that the CPU can write
to one while the GPU reads from
another.
Let's look at a simple triple
buffering example.
So here we start with the first
resource ready to go for the --
to be consumed by the GPU.
But instead of
waitUntilCompleted, we just add
a completion handler so that
once the corresponding frame is
finished on the GPU, it can let
the CPU know that it is done.
But now we don't have to wait
for it to be done.
While the GPU is executing, with
triple buffering the CPU can
jump two updates ahead because
it's in different buffers.
So here we are with the -- with
the frame done executing on the
GPU, and this is where the
completion handler comes in.
It notifies that GPU work is
done and then returns the buffer
to the buffer pool so that it
can be used by the CPU in the
next frame while the GPU
continues execution.
I think most developers will
find that they'll need to
implement triple buffering to
achieve optimal performance.
As for implementation, for
triple buffering, of course, you
need to start with a queue of
three buffers.
You also need to initialize your
frameBoundarySemaphore with a
starting value of three.
And this semaphore will be
signaled at each frame boundary
when the GPU is done executing,
letting the CPU know that it is
safe to override that buffer.
And finally, we need to
initialize the buffer index to
point at the current frame's
buffer.
Inside the render loop, before
we write to a buffer, we need to
ensure that the GPU is
completely done executing the
corresponding frame.
So at the beginning of each
render pass, we need to wait on
our frameBoundarySemaphore.
And then once the signal has
been received, we know that it's
safe to grab its buffer and
reuse it for new frame data.
And now we encode commands and
bind this resource to the GPU to
be used in the next frame.
But before we commit it, we have
to add our completion handler to
the commandBuffer and then we
commit it.
And once the GPU has finished
executing, our completion
handler will signal our frame
semaphore, allowing the CPU to
know that it is done and it can
reuse the buffer for the next
frame's encoding.
And this is a simple triple
buffer implementation that you
can adopt for any dynamic
resource updates.
Okay.
So now we have our resources
updated, so let's talk about
render targets.
In OpenGL, framebuffer objects
are the destination for
rendering commands.
An FBO collects a number of
textures and render buffer
objects under one umbrella and
facilitates rendering into them.
The state of a framebuffer is
mutable, and the render pass is
loosely outlined by binding a
framebuffer and ultimately
swapping them for display.
This is a typical OpenGL
workflow with framebuffers.
During the application's
initialization stage, a
framebuffer is created.
And then you make it current by
binding it.
And then you attach resources
like textures and then check the
framebuffer status to make sure
it's valid to use.
During draw time, you make a
framebuffer current by binding
it, which is implicit start to a
render pass.
And then you have to clear it
before you make any draw calls
to it.
And then at the end you can
signal that certain attachments
can be discarded to let OpenGL
know that it's not necessary to
store these contents into
memory.
These discard events can serve
as hints to end the render pass,
but it's not a guarantee.
In Metal, the render command
encoder is the destination for
rendering commands.
A render command encoder is
created from a render pass
descriptor, which, similar to an
FBO, collects a number of
rendering destinations for a
render pass and facilitates
rendering into them.
A render command encoder is
directly responsible for
generating the hardware commands
for your GPU, and a render pass
is explicitly delineated by the
starting and ending of encoders.
Here's a render pass in Metal.
You start by creating your
renderPassDescriptor.
And the renderPassDescriptor
describes all the attached
resources and also specifies the
operations that happen at the
beginning and end of a render
pass -- these are called load
and store actions.
In contrast to GL, in Metal you
do not clear a resource
directly; instead, you specify a
load action to clear it and also
the color.
Here, it is black.
The store action here is don't
care, which is similar to GL
discard framebuffer in our GL
example.
If you want to store the results
to memory, you would use the
store action here instead.
And at render time, you use your
descriptor to create your
encoder so the state is set.
You make all your draw calls and
then explicitly end encoding.
But before discarding
framebuffers or ending encoding,
let's actually draw something.
A series of render commands is
often referred to as a render
pass.
Inside the render pass, you set
up state and draw call inputs
like textures and buffers and
then issue your draw commands.
This is a typical OpenGL draw
sequence.
A well-behaved OpenGL app tries
to set all of its state ahead of
time, and then it binds its
target and a GL program to link
shaders.
Then it will bind resources such
as vertex buffers, uniforms, and
textures to different stages in
the program.
And finally, it will draw.
As we've discussed a few moments
ago, OpenGL state changes can
cause hidden validation checks.
And if you're already grouping
your state changes together in
OpenGL to avoid these
performance hits, then you'll
get the most out of Metal's
pre-validated state objects.
In Metal, because validation
only happens when you create
your PipelineState object and
because shaders are precompiled,
your render loop becomes much
smaller.
But for a programmer, there's
not that many changes to do.
Here is the same code that we
looked at in OpenGL but now in
Metal.
You start with your render
command encoder, which is an
equivalent to setting the GL
framebuffer.
And then you set your prebuilt
PipelineState object, which is
equivalent to GL use program.
And after that, we assign
resources for our Metal program,
starting with the VertexBuffer
and uniforms.
And you can note here that you
have to set your uniforms per
shader stage instead of like in
GL you set it for the GL
program.
And here, because we ported it
directly from OpenGL, we're
sending the same set of
uniforms; but in Metal you can
send different ones if you want.
And then you set your textures
and issue the draw call.
And finally, once you've done
all the draw calls, you can end
your render pass.
And now, once the work is
submitted, there's still the
matter of presenting.
As the GPU renders the scene, it
writes out to a framebuffer to
display.
In OpenGL, in order to present a
rendered frame, when you return
from drawInRect, the context
calls the presetRenderBuffer for
you.
Metal, on the other hand,
accomplishes this directly
through Core Animations pool of
drawables.
And drawables are textures for
on-screen display.
And you can encode a render pass
to encode to drawables.
You fetch the current drawable,
and then after your render loop
tell the command buffer to
present it.
Remember our code from the very,
very beginning of this talk when
we were talking about the
windows subsystem.
Here we're going to dive into
glkView and drawInMTKView to see
how you can present what you've
rendered.
So here it is.
In glkView you bind your
framebuffer; perform your render
commands; and then when you
return from drawInRect, the
present is managed for you.
In Metal it's much the same: You
create your commandBuffer,
perform your render commands by
creating ending encoders, and
then the one extra step you have
to take is to call
presentDrawable yourself before
finally committing your
commandBuffer.
And if your render loop is very
simple with a single encoder,
then this is all you have to do;
however, if you do have a more
complex app, you may want to
check out the talk we have on
delivering optimized Metal apps
and games for how to handle your
drawables.
And that concludes our frame.
So we've shown how the window
subsystem can be migrated
easily.
We've gone over the resource
creation steps.
We've ported our shaders and
used the great tools to quickly
find issues.
We created our render command
queue, command buffers, and
command encoders to set up our
render passes.
And we created our prevalidated
state objects.
Then to render each frame, we
used triple buffering to update
our resources.
We used the render command
encoders for our command -- for
our render passes where we drew
our geometry before ultimately
presenting the rendered frame.
We've walked through the life of
a graphics app and showed how
Metal is a natural evolution.
Many of OpenGL's established
concepts have migrated into
Metal to work alongside new
concepts that we've added to
address specific problems raised
in the graphics community.
If you can take one thing away
from this session, we hope it's
that porting your applications
from OpenGL to Metal is not
intimidating and that your
application will actually
benefit from it.
But if you have room for two
things, it's that Metal also
offers an awesome set of tools
to enhance your developing
experience.
Max already demoed Xcode's
built-in frame capture and
shader debugger to offer deeper
insight into subtle issues
within your code.
But Xcode also offers the new
GPU memory viewer to understand
and optimize how to use memory
in your application.
In instruments we have a game
performance template that
includes the Metal system trace
to visualize submission issues
which might cause frame drops.
And new this year we also have
support for Metal in the
simulator.
Yay, you can get excited.
[laughs]
New with Xcode 11 on macOS
Catalina, we have full hardware
acceleration to run your games
and apps for iOS and tvOS
simulator using Metal.
The simulator supports the
MTLGPUFamilyApple2 feature set
and should meet the majority of
your needs to run all of your
apps and games in all available
screen resolutions.
For a deeper dive into the
simulator and how it achieves
hardware acceleration, please
check out the simulator talk
tomorrow morning.
If you're looking to solve a
specific issue with Metal, you
can see our many, many sessions
online.
For more information, you can
check out our documentation on
our website or you can visit us
in the Metal lab tomorrow
morning.
And with that, thank you all for
coming, and I hope to see you at
the bash.
[ Applause ]