Transcript
[ Music ]
[ Applause ]
>> Good afternoon, and welcome
to Adopting Metal, Part I.
I'm Warren Moore from the GPU
Software Team, and I'm joined
by my colleague, Matt Collins,
who will be driving
the demos today.
I want to start off by
asking a deceptively,
simple question: what is Metal?
You've heard us say that Metal
is Apple's low overhead API
for GPUs, that it has a unified
graphics compute language,
and that it's built for
efficient multithreading,
and is designed for
our platforms.
And all of this is true,
but Metal is a lot more
than Metal.framework.
Metal is supported by additional
frameworks and tools and so on.
Metal is supported by additional
frameworks and tools and so on.
And they make it a lot more than
just the metal framework API.
In particular, last year
we introduced MetalKit,
which includes utilities
for doing common tasks
like interacting UIKit, and
AppKit, and loading textures,
as well as Metal Performance
Shaders, which allow you
to do common tasks such
as imaging processing,
and contain hand-tuned,
highly optimized Shaders
that you can drop right into
your app to do these tasks.
Metal is also tightly integrated
with our developer tools,
Xcode and Instruments.
When you have Shaders
in your app,
they're actually compiled
right along with your app,
including your app bundle to do
Metal's integration with Xcode.
And the GPU Frame Debugger,
allows you to take a snapshot
of your app at any given point,
and see exactly what's going on.
Metal System Trace in
Instruments allows you
Metal System Trace in
Instruments allows you
to get an ongoing view
of the performance
and behavior of your Metal apps.
So two years ago, we introduced
Metal on iOS, and since then,
we've brought Metal
to Mac OS and tvOS.
So it really has broad
support across our platforms.
And it's also widely
supported by our hardware.
It's supported on our desktop
and mobile architectures
from Apple, AMD,
Intel, and NVIDIA,
and this includes all Apple
Macs introduced since 2012,
and all iOS devices since 2013,
as well as the new Apple TV.
So Metal gives your applications
access to the performance
and power of the GPU in
literally hundreds of millions
of our most popular products.
And Metal is also a foundational
technology on these platforms.
It powers Core Graphics, Core
Animation, as well as our Games
and Graphics Libraries
such as SpriteKit,
SceneKit, and Model I/O.
And it's also an
important component
in key system applications
like Preview and Safari.
in key system applications
like Preview and Safari.
And Metal has been widely
adopted by developers
of all sizes, from AAA
Studios, game engine providers,
independent developers, and
creators of professional tools,
and they've built
amazing games and apps
across all of our platforms.
These are just a few examples,
but I'd like to highlight
a couple.
For instance, Fancy
Guo used Metal
to dramatically improve
performance
and bring amazing visual effects
to their highly popular
MORPG, Furious Wings.
And Metal has also been used
to build inspiring professional
content creation tools,
like the upcoming version
of Affinity Photos for iPad.
And I'd like to show you just a
quick preview of what's coming.
This is Affinity Photos
built by Serif Labs.
And they're building a fully
featured, photo editing app
for the iPad Pro, allowing them
to achieve truly
stunning results.
And this year at WWDC, we
want to give you the tools
And this year at WWDC, we
want to give you the tools
to help you start using Metal
to build amazing experiences
in your apps as well.
We have a lot of phenomenal
content this year at WWDC,
five sessions dedicated
to Metal.
Of course, this is the first
session, Adopting Metal, Part I.
And during this session,
we'll talk a little bit
about some foundational
concepts in Metal, go on to talk
about doing 2D drawing and
then actually add lighting,
texturing, and animation
as well as we move into 3D.
In Part II of this session,
happening in this room
after this session, we'll talk
about dynamic data management
and go on and talk about
some of the finer points
of synchronizing
the GPU and CPU,
and really taking your
performance to the next level
with multi-threaded encoding.
Of course, we're also going to
talk about what's new in Metal.
And there's really a
tremendous list of new features
that you probably saw teased
during the Platform State
of Union yesterday.
I won't go through all
of these in detail,
but if you're interested in
implementing any of these
but if you're interested in
implementing any of these
in your apps, you
should definitely check
out the What's New sessions.
And finally, we have
an awesome talk
on advanced Shader optimization.
And this is really a hardcore
talk for the people who want
to get the absolute most
out of their Metal Shaders.
We'll talk specifically
about how the hardware works
and how you can use Metal to
really drive it to the max,
and tune your Shader code.
Throughout the course
of these sessions,
we'll build a sample project,
starting with just a simple
Hello Triangle, and Hello world
of graphics programming.
And then as I mentioned, we'll
move to animation and texturing.
And in Part II, we'll take
it to the next level and talk
about updating object data
in real time and also,
performing draw calls
across multiple threads.
Now of course, we have
to make some assumptions
about who you are.
We assume that you're
familiar with the fundamentals
of graphics programming, ideally
with a programmable pipeline.
So you're familiar
with Shaders and so on.
And also of course
that you're interested
And also of course
that you're interested
in actually using Metal
to make your games
and apps even more awesome
than they already are.
I assume that everybody here
is on the same page with that.
That's why you're here, right?
So just to go through
the agenda.
We'll kick things off
with a conceptual overview
that will sort of introduce
the philosophy of Metal
and why Metal is
shaped the way it is.
Then we'll actually get right
down to the nitty gritty
and talk about creating
a Metal device.
We'll go on to talk
about loading data
into memory that's
accessible by the GPU.
And we'll talk about the Metal
shading language briefly.
We'll talk about creating
pre-validated pipeline states.
And then talk about issuing GPU
commands, including draw calls.
And then we'll finish up
with a discussion of how
to perform animation
and texturing in Metal.
Part II will take
things even further,
and I've already mentioned
what we'll discuss there.
So let's just forward ahead.
Starting off with the
conceptual overview.
There are a few things that
I want to emphasize here.
Use and API that matches
the hardware and driver.
Favor explicit over implicit.
And do expensive
work less often.
Let's start with using an API
that matches the hardware.
Metal is a thoroughly
modern API.
And by that, I mean
that it integrates with
and exposes the latest
hardware features,
and it matches very closely to
how the hardware actually works.
And being a comparatively
new API, it's very thin
and has no historical cruft that
you get with other legacy APIs.
So there are no fancy
tricks required
for low overhead operation.
It's baked in to how Metal
is shaped and how it operates
at the most fundamental level.
And fortunately,
it's unique by design
across all -- across
our platforms.
When we say that we
want to favor explicit
over implicit operation, we
mean that we put in your hands,
the responsibility to
perform some explicit control
over how commands are
submitted to the GPU,
as well as how you manage
and synchronize your data.
And this puts on you, a
lot of responsibility,
but with great responsibility
comes great performance.
but with great responsibility
comes great performance.
So, just to illustrate
what we mean when we say
"to do expensive work less
often," there are kind
of three regimes of time
that we can think about.
The time that your app is built,
the time that your app is
loading, loading assets
and so on, and then draw time,
the things that happen
60 times per second.
So with a legacy API, like
OpenGL, you pay the cost of work
like state validation every
time you issue a draw call.
You take the hit for
recompiling Shaders
on the fly in the worst case.
And all this adds overhead
on top of the necessary work
of encoding the actual work for
the GPU, like your draw calls.
With Metal, we push some of this
work earlier in the process.
So, as I alluded to earlier,
Shader compilation
can actually happen
at application build
time with Metal.
And additionally, we allow
you to validate the state
that you're going to be using
for your draw calls in advance
at load time, so you don't pay
that cost every time
you issue a draw call.
that cost every time
you issue a draw call.
Instead, the only work
that remains to be done
when you're issuing your draw
calls, is to do your draw calls.
So with that conceptual
overview,
let's talk about where
the rubber hits the road,
and that's, the Metal device.
So there's a Class MTL device,
and it's an abstract
representation of your GPU.
And it functions as the root
object in your Metal app,
meaning that you'll
use it to create things
like command queues, resources,
pipeline state objects,
and other objects
that you'll be using.
It's very easy to
create a Metal device.
You'll just call
MTLCreateSystemDefaultDevice.
Now devices are persistent
objects, so you'll probably want
to create one at the
beginning of your application
and then hold onto reference
to it throughout your
application life cycle.
It's just that easy.
So now let's talk a little
bit about how to get data
into a place where
the GPU can access it,
so that you can issue
your draw calls.
And in Metal, we'll store
our data in buffers.
So buffers are just allocations
of memory that can store data
in any format that you choose.
These might be vertex data,
index data, constant data.
And you write data
into these buffers
and then access them
later on in your vertex
and fragmentFunctions.
Let's take a look at what
that might look like.
So here is an example
of a couple of buffers
that you might create, as
you're loading your data.
We have a vertexBuffer,
containing some vertices.
And an indexBuffer, containing
some contiguous indices.
To get a little bit more
concrete, each instance
of this vertex type
might be a Swift struct
that contains a position
vector for the vertex,
as well as a color
for the vertex.
You can just lay them out
contiguously in memory.
Now let's talk about how
you actually create buffers.
So the API for this
is on the device
that you've already created.
And you simply call
newBufferWithLength
And you simply call
newBufferWithLength
to get a buffer of
a particular size
that doesn't have any
data loaded into it.
Or you can call
newBufferWithBytes
and pass a pointer to data
that already lives in memory.
And Metal will then
copy that data
into the newly created
Metal buffer,
and it will be ready
for your use.
You can also memcpy into
the contents pointer
of the buffer if you choose.
So, since we're going to
be showing a 2D triangle
as the first part of our demo,
let's talk about defining the
geometry for this triangle here.
So, since we want to
keep the vertex shader
and fragment shader
as simple as possible,
we'll actually provide these
coordinates in Clip Space.
And Metal's Clip
Space is interesting.
It differs from some APIs and
it's similar to some APIs.
This is like the
DirectX Clip Space.
It runs from negative 1
to 1 in X, negative 1 to 1
in Y, and zero to 1 in Z.
So this is the coordinate space
that we'll specify
our vertices in.
So in code, it looks like this.
We create a Swift
array of vertices,
and then we just
append vertices each
with a position and a color.
Now, we don't strictly
need to use index drawing
to do the simple of a used
case, but we'll go ahead
and create an indexBuffer and
append the indices 0, 1, and 2,
which correspond to of
course, the first, second,
and third vertices
of our triangle.
And then we'll create a couple
of buffers with our device.
So we'll create the vertexBuffer
by calling newBuffer(withBytes,
which loads our vertex data
into this Metal buffer,
and we'll call a
newBuffer(withBytes again
and pass the index data and
get back the indexBuffer.
So, now that we have
our data and memory,
let's talk a little bit about
Metal's unique shading language.
The Metal shading language is
an extended subset of C++ 14,
and it's a unified language
for graphics and compute,
meaning that you can
do a whole lot more
than just 3D graphics with it.
It really just lets you
write programs for the GPU.
It really just lets you
write programs for the GPU.
So here's a block diagram of
the stages of the pipeline.
And what we're really talking
about now is the vertex
and fragment processing stages.
Each of these has an associated
function that you'll write,
that's used to either process
the vertices or the fragments
that will wind up on screen.
Syntax-wise, it looks
a little bit like this.
And we're not actually going to
go through this in any detail.
I just want to call
your attention
to these function qualifiers:
vertex and fragment.
You'll notice that right out
in front of these functions,
unlike in say a regular
C++ program,
we actually have these
qualifiers that denote
which stage this function
is associated with.
So we have a vertex
function up top,
and a fragmentFunction
down below.
And I'll show you shortly how
to actually associate these
with your pipeline so that
you can use them to draw.
And we'll also look at the
internals of these functions
for our 2D demo and
later on for our 3D demo.
Now, I've mentioned a couple
of times that Metal allows you
Now, I've mentioned a couple
of times that Metal allows you
to compile your Shaders
directly into your app bundle.
And the way that happens is,
if you have even a
single .Metal file
in your Compile Sources
Phase of your project,
the Metal will automatically
generate what's called a Metal
Lib file, the default.Metallib
file,
and copy it into your bundle
at the time your
application is built
with no further effort
on your part.
So there's the insides
of your app bundle.
There's your default.Metallib.
So just to recap.
You can build Metal
Shaders at runtime.
Again, if you have a .Metal file
in your app, it will be compiled
by Xcode using the
Metal toolchain.
And then produce
default.Metallib which will wind
up in your app bundle.
And the natural question you
have at this point is, "Well,
how do you actually get
these functions at runtime?"
And the answer is that
you'll use a class called
Metal Library.
So Metal Library is a collection
of these compiled
function objects,
produced by the compiler.
And there are multiple
ways to create it.
You can go through the flow
that we just discussed,
which is to build
a default.Metallib
which is to build
a default.Metallib
into your app bundle, and
then load it at runtime.
You can also build .metallibs
with a command line
toolchain that we provide.
And you can also build directly
from a source string at runtime
if you're for example,
building Shaders
through string concatenation.
So in code, it looks like this.
In order to load up
the default.Metallib,
you simply call
newDefaultLibrary
on your existing Metal device.
And there's other API for
loading from, for example,
an offline compiled
.Metallib, or from source.
And you can consult
the API docs for that.
So, you have a Metal Library.
What do you get from
Metal Library?
You get a Metal function.
Now a Metal function
is simply an object
that represents a
single function.
And it's associated with a
particular pipeline stage.
Remember that we saw
the diagram earlier,
the vertex or fragment stage.
And we also have an additional
function qualifier called
kernel, that signifies a data
parallel of compute function.
kernel, that signifies a data
parallel of compute function.
So here's that code snippet
again, and you can see
that the function named
here is vertex transform,
and the fragmentFunction
name is fragment lighting.
And I rehash this so that
I can show you the API
for loading these
functions from your library,
which looks like this.
We simply call
NewFunctionWithName
and pass a string that
represents the name
of your function, and get
back a Metal function object,
and then hold onto it.
Now, I'll show you how
to actually use all these
objects in a moment.
But that was just a
brief introduction
to the Metal Shading language.
So let's talk about building
pre-validated pipeline states.
But first, let's
motivate it a little.
So, with an API like OpenGL,
you're often setting
a lot of state.
And then you issue draw calls.
And in between those, the
driver is obligated to validate
that the safety you've set, is
in fact a valid state and again,
in the worst case, you
can even pay the cost
of recompiling Shaders
at runtime.
of recompiling Shaders
at runtime.
This is what we want to avoid.
So with Metal, it
looks more like this.
You set a pre-validated
pipeline state object,
and maybe set a few other
bits of ancillary state,
and then issue your draw call.
Now what we're trying to do
here is to reduce the overhead
of draw calls by again,
pushing more work earlier
into the process.
So here are a few examples of
state that you'll want to set
on your pipeline state object,
which we'll talk
about in a moment.
First, let's state that you
can set pretty much anytime
when you're drawing.
You'll notice that in
the left hand column,
the state that you'll set
on the pipeline state,
includes the vertex
and fragmentFunction
that will be used to draw.
And it also includes things
like your alpha blending state.
On the right hand side,
instead we see the state
that you can set before
issuing any given draw call,
including the front face
winding and the call mode.
So let's talk about how
you actually create objects
that contain this
pre-validated state.
The chief object is the
Metal RenderPipelineState.
It represents the
sort of configuration
It represents the
sort of configuration
of the GPU pipeline, and
it contains a set of --
of validated state that you'll
create during load time.
Like devices,
RenderPipelineStates
are persistent objects
that you'll want to keep
alive throughout the lifetime
of your application,
though if you have a lot
of different functions, you can
create pipeline state objects
asynchronously while
your app is running.
To actually create a
RenderPipelineState,
we don't create one directly.
Instead, we use an object
called a Descriptor that bundles
up all the parameters
that we're going to use
to create this
RenderPipelineState.
Often in Metal, we'll
create Descriptor objects
that really just bring together
all of the different parameters
that we need to create
yet another object.
And so for the
RenderPipelineState object,
that's called a Render
Pipeline Descriptor.
You'll notice that it contains
pointers to the vertex function
and fragmentFunction,
as I mentioned earlier.
And it also contains a
collection of attachments.
And attachments signify the type
of texture that will be rendered
into when we actually
do our drawing.
into when we actually
do our drawing.
Now in Metal, all rendering
is rendered to texture,
but we don't need pointers to
those textures right up front.
Instead we just need you
to supply the pixel formats
that you'll be rendering into so
that we can optimize the
pipeline state for them.
Additionally, if you're using
a Depth or Stencil Buffer,
then you can also specify the
pixel format of those targets.
So once you've constructed a
render pipeline descriptor,
you can pass it off
to your Metal device
and get back a
MTLRenderPipelineState object.
Let's take a look
at that in code.
Here's a minimal configuration
for a RenderPipelineState.
You'll notice that we're
setting our vertex function
and fragmentFunction
properties to the vertex
and fragmentFunction objects
that we created earlier
from our library.
And we're also configuring
the pixel format
of the primary color attachment
to be .bgra8Unorm, which is one
of our renderable and
displayable pixel formats.
This represents basically
the texture that will --
ultimately be drawn to
when we do our drawing.
And finally, once
we've constructed
And finally, once
we've constructed
that pipeline descriptor,
we can use the new
RenderPipelineState function
on the device to actually get
back this pre-validated object.
I want to emphasize once more
that PipelineStates
are persistent objects,
and you should create them
during load time and keep them
around as you do your
device and resources.
You can switch among them
when doing drawing in order
to achieve different effects.
You'll generally have about
one per pair of vertex
and fragmentFunctions.
So now that we've
talked about how
to construct pre-validated
state and how to load some
of your resources into memory,
let's talk about actually
issuing GPU commands,
including draw calls.
We'll go through this
in several stages.
We'll talk about interfacing
with UIKit and AppKit,
talk about -- a bit about the
Metal command submission model,
and then get into render passes
and draw calls, and finally,
how to present your
content on the screen.
So in terms of interacting
with UIKit and AppKit,
we're going to use a utility
for MetalKit called MTKView.
And MTKView is a
cross-platform, view class.
And MTKView is a
cross-platform, view class.
It inherits from
NSView on Mac OS
and from UIView on iOS and tvOS.
And it reduces the amount of
code that you have to write
in order to get up
and running on Metal.
For example, it creates and
manages a CA Metal Layer
for you, which is a specialized
CALayer subclass that interacts
with the Windows server or
with the display loop in order
to get your content on screen.
It can also, by use of
CV or CA display link,
manage the draw callback
cycle for you
by issuing periodic callbacks
in which you'll actually
do your drawing.
And it also manages the textures
that you'll be rendering into.
I want to emphasize
on particular aspect
of what this does for
you, and that's drawables.
So, inside of the CA Metal Layer
that's managed by your MTKView,
there is a collection
of drawables.
And drawables wrap a texture
that will ultimately
be displayed on screen.
And these are kept in an
internal queue, and then reused
across frames because they're
comparatively expensive
across frames because they're
comparatively expensive
and they need to be
managed by the system
because they actually
are very tightly bound
to how things actually get
displayed on the screen.
So we manage them for you and
we hand you the drawable object
that wraps up one of these
textures that you can draw into.
So here's how you can --
there are numerous properties
that you can configure
on an MTKView
to determine how it
manages the textures
that you're going
to be drawing into.
In particular, you can set a
clear color that will determine
which color, the primary
color target is clear to.
You can specify the
color pixel format,
which should match the color
format that you specified
on your pipeline state object,
as well as specifying a depth
and/or stencil pixel format.
And this last property is
probably the most important.
This is where we
set the delegate.
So, MTKView doesn't actually do
any drawing in and of itself.
You can either subclass it,
or you can implement a
delegate that's responsible
for doing the drawing.
And we'll talk through
the later use case.
So let's take a look at
what you have to do in order
to become an MTKView delegate.
It really boils down to
implementing two methods:
drawable sizeable
change, and draw.
So in drawable sizeable
change, you are responsible
for responding to things
like the window resizing
or the device rotating.
So for example, if your
projection matrix is dependent
on the window size, then
this gives you an opportunity
to respond to that instead
of rebuilding it every frame.
So the draw method will
be called periodically,
in order for you to
actually encode the commands
that you want to have executed,
including your draw calls.
And we're not showing
the complete internals
of that method here, but this is
just a taste of what's to come
when we talk about
command submission.
So you'll create a
commandBuffer, do some things
with it, and then commit it.
We'll talk a lot more about that
in a moment, but this is sort
of your hook for doing the
drawing if you're using MTKView.
And we recommend using MTKView,
especially as you're getting
started, because it takes care
of a lot of things for you.
So let's talk about Metal's
command submission model.
This is the picture that
we're going to be building
up over the next several slides.
And it's not important for you
to memorize everything
that's going on here.
We're going to be
building this up.
This is just sort
of an overview.
The objects that we're going to
be constructing as we go along.
So, Metals Command Submission
Model is fairly explicit,
meaning that your
obligated to construct
and submit commandBuffers
yourself.
And you can think of a
commandBuffer as a parcel
of work to be executed
by the GPU,
in contrast to what we're
calling a Metal buffer,
which stores data.
Command buffers store work
to be done by the GPU.
And commandBuffer submission
is under your control,
meaning that when you
have a commandBuffer
that you've constructed, you're
obligated to tell the GPU
when it's ready to be executed.
We'll talk all about
this in a moment.
Additionally, we'll talk
about command encoders
which are objects that are used
to translate from API calls
into work for the GPU.
It's important to realize
that these command encoders
perform no deferred state
that these command encoders
perform no deferred state
of validation.
So all of the pre-validated
state that bundled
up in your pipeline,
state objects,
we assume that that's valid
because we validated
it in advance.
And so there's no additional
work to be done by the encoder
or the driver at the point
that you're issuing
commands to be rendered.
Additionally, Metal's Command
Submission Model is inherently
multi-threaded, which allows you
to construct multiple
Command Buffers in parallel,
and have your app decide
the execution order.
This allows you to scale to,
and beyond, tens of thousands
of draw calls per frame.
Adopting Metal Part II will
talk about this in depth,
but I wanted to mention it
now to hint at what's to come.
So let's talk a little bit more
about these objects in depth.
The first thing we'll talk
about is the Command Queue.
And the Command Queue,
which corresponds
to a Metal class called Metal
Command Queue, manages the work
that has been queued up
for the device to execute.
Like the device and
resources and pipeline states,
queues are persistent objects
that you'll want to create
queues are persistent objects
that you'll want to create
up front and then keep a handle
on for the lifetime of your app.
You'll often only
need to create one.
And this is how thread safety
is introduced into the Metal API
in the sense that you can create
Command Buffers and render
into -- and use them
on multiple threads.
And the queue allows you
to create and commit them
in a thread safe fashion,
without you having
to do your own locking.
It's really simple to
create a Command Queue.
You simply call a new
Command Queue on your device,
and you'll get back a
Metal Command Queue.
Of course, a queue can't do much
unless you actually put work
into to, so let's
talk about that.
So I've already mentioned
Command Buffers.
And Command Buffers are the
parcels of work to be executed
by the GPU, and in
Metal, they're represented
by a class called
Metal Command Buffer.
So a Metal Command Buffer
contains a set of commands
to be executed by the GPU,
and these are each enqueued
onto a Command Queue for
scheduling by the driver.
onto a Command Queue for
scheduling by the driver.
These, in contrast to almost
everything we're talked
about thus far, are
transient objects,
meaning that you'll create
one or more of them per frame,
and then encode commands
into them,
and then let them
go off to the GPU.
You won't reuse them.
You won't hold onto
a reference to them.
They're just fire and forget.
To create a Command
Buffer, you simply call
up the Command Buffer
on a Command Queue.
So we've talked a bit
about Buffers, and Queues,
and now let's actually
talk about how we get data
and commands into a
commandBuffer, and that's done
with a special class of
objects called Command Encoders.
And there are several
types of Command Encoders,
including Render,
Blit, and Compute.
And these each allow you
to do different things.
And they all have this common
thread though allowing you
to encode work into
a Command Buffer.
So for example, a Render
Command Encoder will allow you
to set state and
perform draw calls.
A Compute Command Encoder
will allow you to enqueue work
A Compute Command Encoder
will allow you to enqueue work
for the GPU to execute in a
data parallel fashion that's not
rendering work.
That's your GP, GPU, and
other stuff like that.
And the Blit Command Encoder
allows you to copy data
between buffers and
textures, and vice versa.
We're going to look in detail
at the Render Command
Encoder in this session.
And as I mentioned, it
has the responsibility
of encoding commands.
And each Render Command
Encoder, encodes the work
of a single pass into
a Command Buffer.
So you'll issue some
state changes
and then you'll issue some
draws and it manages a set
of Render target attachments,
that represent the textures
that are going to be drawn into
by this one particular pass.
So schematically,
what we're talking
about here is sort
of the last stage.
You can see we have
these attachments sort
of hanging off the frame buffer
right stage of the pipeline.
And if we were doing multi pass
rendering, then one or more
of the render targets in
this pass might become inputs
of the render targets in
this pass might become inputs
for a subsequent pass.
But this is sort of the
single pass, simple use case.
So the again, the attachments
represent the textures
that we're going to be drawing
into at the end of this pass.
So in terms of actually creating
a render command encoder,
we use another type
of descriptor object,
called a RenderPassDescriptor.
So a RenderPassDescriptor
contains a collection
of attachments, each of which
has an associated load store
action, a clear color,
clear value,
and an associated Metal
Texture to be rendered into.
And we'll talk a little bit more
about load and store actions
in a couple of slides.
But the important
thing to realize here,
is that you'll be constructing
a RenderPassDescriptor
at the beginning of your frame,
and actually associating it
with the textures that
are going to be drawn.
So in contrast to the
renderPipelineState object
that only needs to know the
pixel format, this is sort
of where the rubber hits the
road and you actually have
to give us the textures that
we're going to be drawing into.
So again, a RenderPassDescriptor
contains a collection
of render pass attachments,
each of which might be a color,
depth, or stencil target,
and refers to a texture
to render into.
And it also specifies
these things called load
and store actions.
Let's talk in more depth about
what that actually means.
So, at the beginning of a pass,
you have your color buffer
and your depth buffer and
they contain unknown content.
And in order to actually
do any meaningful work,
we'll need to clear them.
And we do this by setting
their associated load action
on the RenderPassDescriptor.
So we set a load action of clear
on the color and depth targets,
and that clears them to their
corresponding clear color
or clear value, as
the case may be.
Then we'll do some drawing,
which will actually
put the results
of our draw calls
into these textures.
And then the store
action will be performed.
And the store action here is
going to be one of two things.
The store action of
store, signifies the result
of rendering should
actually be written back
to memory and stored.
And in the case of the color
buffer, we're actually going
to present it potentially
on screen.
In the case of the depth buffer,
we're really only using it
when we're actually drawing and
rendering, and so we don't care
about where the results of
that go at the end of the pass.
So we can set a store action
of "Don't Care" in order
to save some bandwidth.
This is an optimization
that you can do
if you don't actually need
to write back the results
of rendering into
the render target.
So to go in a little bit more
depth on load and store actions,
these determine how
texture contents are handled
at the start and
end of your pass.
In addition to the clear
load action that we just saw,
there's also a Load-Load
action that allows you
to load pixel contents
of your textures
with the results
of a previous pass.
There's also "Don't Care."
For example, if you're going to
be rendering across all pixels
of a given target, then you
don't actually care what was
in the texture previously,
nor do you need to clear it
because you know
that you're going
to actually be setting every
single pixel to some value.
to actually be setting every
single pixel to some value.
So that's another way
that you can optimize,
if you know that in
fact you are going
to be hitting every
single pixel in this pass.
Now, I could walk
you through how
to create a RenderPassDescriptor
and then create a Render Command
Encoder, but fortunately,
MTKView makes this
really easy on you.
You saw earlier that we
configured the MTKView,
with a couple of properties
that by now I hope you become
familiar, like the clear color
and the texture formats
of your render targets.
So you can actually
just ask the view
for its current
RenderPassDescriptor
and you'll get back a
configured RenderPassDescriptor
that you can then go on to use
to create a Render
Command Encoder.
And this is how you do that.
You simply call Render Command
Encoder on your Command Buffer.
Now it's important to note here
that current
RenderPassDescriptor is
potentially a blocking call.
And the reason for that is
that it will actually call
into the CA Metal Layers
next drawable function,
which we won't talk about in
detail now, but which is used
which we won't talk about in
detail now, but which is used
to obtain the drawable
that wraps the texture
that can be presented on screen.
And because that is
a finite resource,
if there is not a drawable
currently available,
if all of them are in flight,
then this call will block.
So it' something to be aware of.
So we've talked about
loading resources into memory,
and we've talked about
creating pre-validated state
and we've talked about
now, created Render Passes
and Render Command Encoders.
So how do we actually get
data into our Shaders?
First, we need to talk a little
bit about argument tables.
So argument tables are mappings
from Metal resources
to Shader parameters.
And each type of resource that
you're going to be working with,
such as a buffer or a texture,
has its own separate
buffer argument table.
So you can see here
on the right,
that we have the
buffer argument table
and the texture argument table,
each of which contain a couple
of buffers that are maps
to particular indices
in the argument table.
Now the number of slots
that are available
in any given argument table
are actually dependent upon
in any given argument table
are actually dependent upon
the device.
So you should query for them.
Let's make that a
little bit more concrete.
So, on the Render Command
Encoder there's a function
called Set Over Text
Buffer, and you'll notice
that it has three parameters.
It takes a buffer, and
offset, and an index.
So this last parameter is
what we care about the most
because it's our
argument table index.
So this is sort of the host
side of setting resources
that are going to be
used in your Shader.
And there's a corresponding
Shader side
which looks like this.
So this is in the
middle shading language.
Inside your Shader file,
you'll specify that the --
that each given parameter
that corresponds to a resource
that you want to access, has an
attribute that looks like this.
So, this is the first
buffer index.
Buffer Index Zero in
the Argument Table,
corresponds to the buffer
that we just set back
on our Render Command Encoder.
And we'll look at a little
bit more about this in detail
when we actually talk
about doing drawing in 2D.
We've already created
a renderPipelineState,
but we actually need to tell
our Render Command Encoder
which pipeline state to use
before doing any drawing.
So this is API for that.
We simple call
setRenderPipelineState
with the previously created
PipelineState object,
and that configures the pipeline
with the Shaders that we've --
that we created earlier that
we're going to be using to draw.
Now of course, the
RenderPipelineState,
has associated -- an associated
vertex and fragments function.
So let's take a look at the
vertex and fragmentFunction
that we're actually going
to be using to draw in 2D.
Back in Metal Shading
language, it looks like this.
So, this is basically a
pass through vertex function
which means that we're not going
to be doing any fancy
math in here.
It's really just going to
copy all these attributes,
straight on through.
So, the first parameter to this
function is a list of vertices
which is the buffer
that we just bound.
And the second parameter is
this thing that's attributed
with the vertex ID attribute,
which is something that's going
to be populated by Metal
with the index of the vertex
that we're currently
operating on.
And the reason that's
important is
because the vertexBuffer
contains all the vertices,
and we can access it at random.
But what we actually want to do
in our vertex function
is operate
on one particular
vertex at a time.
So this tells us which
vertex we're operating on.
So, we create an instance
of this struct VertexOut,
which represents all the
varying properties of the vertex
that we want to pass
through to the rasterizer.
So we create an instance of
this and set its position
to the position vector of the
vertex indexed at vertexId.
And similarly for the color.
And this just passes that data
on through from the vertexBuffer
to the struct that will be
interpolated by the rasterizer.
And then we return that
structure back on out.
Now let's look at
the fragmentFunction.
It's even simpler.
So we take in the interpolated
struct, using the stage
in attribute, and that signifies
that this is the data that's
coming in from the rasterizer.
And we just extract
that color --
the color from the
incoming structure,
and then pass it back on out.
And so, what's happened in
this process is the vertices,
which were already specified
in Clip Space in this example,
are being interpolated and
then rasterized and then
for each fragment
that we're processing,
we simple return the
interpolated color
that was created for
us by the rasterizer.
So once we've specified
the RenderPipelineState
which contains our vertex
and fragmentFunction,
we can also set additional
state, kind of like the stuff
that I mentioned earlier,
including the front
facing state.
So if you want to specify a
different front facing winding
order, than Metal's
default of clockwise,
then you can do that here.
It's a lot of configuration,
but we're actually
about to see some draw
calls happen, right now.
about to see some draw
calls happen, right now.
So Metal has numerous functions
for drawing geometry including
indexed, instance, and indirect,
but we'll just look at
basic index drawing.
Let's say that we want to draw
that triangle, at long last.
So, here we call
drawIndexedPrimitives,
and we specify that the
prototype is triangle
because we want to
draw a triangle.
We pass an index count of
three to signify that we want
to draw a single triangle,
and then we also specify
the type of indices.
We made our Swift array a
collection of UN 16s earlier,
so we mirror that here.
And we also pass
in the indexBuffer
that we created earlier
that signifies
which vertices should be drawn,
and then we pass in
an offset of zero.
And this is actually
going to result
in a single triangle
being drawn to the screen.
We might also additionally
set some more state
and issue other draw
calls, but for the purposes
of this first demo, this
is all there is to it.
So in order to conclude
a render pass,
we simple call it endEncoding
on the Render Command Encoder.
To recap all of that, you
will create a request,
a RenderPassDescriptor, at
the beginning of your frame.
Then you'll create a
Render Command Encoder
with that Descriptor.
Set the RenderPipelineState.
Set any other necessary state.
Issue draw calls, and
finally end encoding.
So here's a recap of all the
code that we've seen thus far.
Nothing new here.
Exactly what we've seen and
exactly what I just said.
Create a Render Command
Encoder, set state, set state,
bind some buffers, and draw.
So you've rendered all
this great content,
but how do you actually
get it on the screen?
It's pretty straightforward.
So, first color attachment
of your render pass is
usually a drawables texture
that you've either gotten
from a CA Metal Layer
or from and MKTView.
So in order to request that that
texture actually get presented
on screen, you can
actually just call present
on the commandBuffer, and
pass in that drawable,
and that will be displayed
to the screen once all the
preceding passes are complete.
Then to finally actually
finish up the frame,
since we've been encoding
into this commandBuffer,
we need to signify
that we're done
with the commandBuffer
by calling commit.
Committing tells the driver
that the commandBuffer's ready
to be executed by the GPU.
So to recap that, we created
a command queue at start-up,
and since it's a
persistent object,
we hold onto reference to it.
Each frame we create a
commandBuffer, encode one
or more rendering passes into it
with a render command encoder.
Present the drawable
to the screen
and then commit the
commandBuffer.
And now I'm going to hand
things off to my colleague Matt,
to walk us through the
demo of drawing in 2D.
>> Thanks Warren.
[ Applause ]
>> So here's the proof.
A 2D triangle, this is the Metal
triangle demo as you can tell
by our awesome title,
is very simple.
by our awesome title,
is very simple.
Just a triangle, three colors
on the ends interpolated
nicely over the edges.
Let's take a look at the code.
Now first I want to
show you what it takes
to become a delegate of MTKView,
and Warren mentioned we have
two functions to implement.
So here we have MTKView,
drawable, sizeable change.
And this is what is called
when you need to respond
to changes in your window.
This sample is very simple so
we didn't actually implement it.
We'll leave that up to you
guys for your own applications.
And the other thing
is simple the draw.
We chose to put this
into a render function.
So, when our draw gets
called, we go into our render.
Render's also quite simple.
I just wanted to show you.
When we take MTKView's
current RenderPassDescriptor,
you just grab it out
like Warren said,
and then you create the
RenderPassDescriptor
and your encoder with it.
And I'd like to draw
your attention here,
"push debug group."
And this is how you talk
to the awesome Metal tools.
So when you do a frame
capture, this will then sort all
So when you do a frame
capture, this will then sort all
of your draws by whatever
debug group you've had.
So here, we have one
draw and a draw triangle
and then we pop the debug
group after we've drawn,
and so this draw will show up
labeled as "Draw Triangle."
Let's take a look at the Shader.
Now Warren mentioned,
we had structs.
We have a vertex end
struct, which is the format
of the data we're putting
into the Shader, as you see.
That's just a position
and a color.
And we have the vertex
out struct,
which is what we're passing
down to the rasterizer.
And you see here the
position has been tagged
with this position attribute.
And this represents the
Clip Space position.
And every vertex Shader that you
have or vertex function, sorry,
must have one of these.
And as you saw, these
should look kind
of familiar, they're
very simple.
Vertices come in.
We have a pass through.
And you write them out.
And in the fragmentFunction, we
take in the vertices that came
out of the rasterizer
and we read the color,
and send that down.
So that's the simple
triangle demo.
I'll send it back
to your Warren.
>> Thanks Matt.
So, we've shown how to
actually draw 2D content.
And 2D is cool, but you
know what's even cooler?
Three-D. So let's talk a
little bit about animation
and texturing in Metal.
In order to actually
get into 3D --
alright well, we'll go through
this in a couple stages.
We'll talk about how to
actually get into 3D.
And we'll talk about animating
with a constant buffer,
and then we'll talk a little
bit about texturing a sampling.
In order to move into 3D,
whereas we've been specifying
our vertices in Clip Space,
we now need to specify them
in a model local space.
And then multiply them
by a suitable model
view projection matrix,
in order to move them
back into Clip Space.
And we'll also add properties
for a vertex normal as well
as texture coordinates so
that we can actually use those
in our fragmentFunction,
to determine lighting
and to determine how to
apply the texture map.
So, here's our extended vertex.
We have removed the color
attribute and we've added
in a normal vector as well as
a set of texture coordinates.
And similarly to how we had
in 2D, we'll just be adding
on a new buffer that will store
all the constants that we need
to reference from our various
vertex and fragmentFunctions
in order to actually transform
those vertices appropriately.
Now, you'll notice
that the outline
of this buffer is dashed, and
there's a good reason for that.
Because I don't want to create
another Metal buffer in order
to manage this tiny
amount of data.
This is only a couple
of matrices.
And it turns out that Metal
actually has an awesome API
for binding very small buffers
and managing them for you.
So again, for small
bits of data,
less than about 4 kilobytes,
you can use this API set
of vertex bytes and pass it --
a pointer directly to your data.
And of course, tell
us what size it is.
And Metal will create
and or reuse a buffer
And Metal will create
and or reuse a buffer
that contains that data.
And again, you can actually
specify the argument table index
here, specifying it as 1,
because our vertices are
already bound at Index 0,
so we bind at Index 1 so that
we can then read from that,
inside of our functions.
So let's take a look at how
our functions actually change
and respond to this.
Before that, we'll
see an example of how
to actually call setForTextBytes
inside your application code.
So, we'll create
this constant struct
that again creates these --
contains these two matrices that
we're going to be multiplying
by the Model View Projection
Matrix, and the normal matrix,
which is the matrix that
transforms the normal
from local space into iSpace.
We'll construct them using
whatever matrix utilities we're
comfortable with, and then
multiply them together.
And finally use setVertexBytes,
passing a reference
to that structure and then
Metal will copy that into again,
this implicit buffer that's
going to be used for drawing
this implicit buffer that's
going to be used for drawing
in our subsequent draw call.
Now, last year at
WWDC, we introduced
and awesome framework
called Model I/O,
and Model I/O contains a
lot of awesome utilities.
But one of the great
things about Model I/O is
that it also allows you
to generate common shapes.
And because of MetalKit,
it actually has very tight
integration with Metal
so that you can create
vertex data
that can be rendered
directly by Metal.
So, instead of actually
specifying all these vertices
by hand, I can for example,
draw my model in some sort
of content creation
package, export it,
and load it with Model I/O.
Or in this case,
generate it procedurally.
So let's take a look
at that in code.
So I want to generate
some vertexBuffers
that represent this cube.
Well, in order to actually get
Model I/O to speak in Metal,
I'll create this thing
called a MeshBufferAllocator.
So MTKMeshBufferAllocator
is the glue
So MTKMeshBufferAllocator
is the glue
between Model I/O and Metal.
By passing a device to
a Mesh Buffer Allocator,
we allow Model I/O to create
Metal buffers directly
and then hand them back to us.
So we create an MDLMesh
using this utility method
boxWithExtent, etcetera,
pass in our allocator,
and this will create an
MDLMesh - a Model I/O Mesh -
that contains the relevant data.
We then need to extract it
by using MetalKit's utilities
that are provided
for this purpose.
And that looks like this.
So first, we generate
and MTKMesh that takes
in the MDLMesh that we just
generated, as well as a device.
And then in order to extract
the vertexBuffer, we just index
into the mesh and pull it out.
Similarly, for the indexBuffer.
And there are also a
couple of parameters here
that we've already
seen that we'll need
to supply to our draw call.
But the emphasis here is on
the fact that it's very easy
to use Model I/O to
generate procedural geometry
and subsequently
pull out buffers
that you can use
directly in Metal.
And now let's talk a
little bit about textures.
We have our vertex data.
We want to apply a
texture map to it
to add a little bit more detail.
Well, as you know, textures
are blocks of memory
in some pre-specified,
pixel format.
And they predominantly are
used to store image data.
In Metal, it's no great surprise
that you create textures
with a descriptor object,
specifically a Metal
Texture Descriptor.
And texture descriptors
are parameter objects
that brings together texture
properties like height and width
and pixel format, and
are used by the device
to actually generate the
texture object: Metal texture.
Let's take a look at that.
So we have these
convenience functions
on Metal Texture Descriptor,
that allow you to ask
for the descriptor that
corresponds to a 2D texture,
supplying on the necessary
parameters: height, width,
pixel format, and whether or
not you want it to be mipmapped.
You can then ask
for a new texture
You can then ask
for a new texture
by calling newTexture
on the device.
Now, this texture doesn't
actually have any image content
in it, so you'll
need to use a method
like Replace Region or similar.
You can consult the docs for
that, but we're going to use
yet another utility to make
that a little bit easier to day.
And that's called
MTKTextureLoader.
So this is a utility provided by
MetalKit, and it can load images
from a number of sources,
including your asset catalogs
or from a file URL,
or from CG images
that you have already
sitting in memory,
in the form of an MS
image or a UI image.
And this generates and
populates Metal textures
of the appropriate size
and format that correspond
to the image data
that you already have.
Now let's take a
look at that in code.
So you can create
an MTKTextureLoader
by simply passing
your Metal device.
You'll get back a TextureLoader,
and you can subsequently fetch a
data asset or whatever have you
from your asset catalog.
And as long as you
get the data back,
then you can call
texture Loader.newTexture,
and hand it to data, and it will
hand you back a Metal texture.
and hand it to data, and it will
hand you back a Metal texture.
You might also be acquainted
with the notion called Samplers.
Now, Samplers and Metal are
distinct objects from textures.
They're not bound together.
And Samplers simply
contain the state related
to texture sampling.
So parameters such as
filtering modes, address modes,
as well as level of detail.
And so we support
all those shown here.
In order to get a Sampler
state that we'll bind later
on in our Render Command
encoder, to do textured drawing,
we'll create a Metal
Sampler Descriptor,
and that looks like this.
So we create an empty
Metal Sampler Descriptor
that has default properties,
and we specify whichever
properties we want.
Here, I'm specifying that we
want the texture to repeat
in both axes, and
that when minifying,
we want to use the
nearest filtering
and when magnifying
we linear filtering.
So once we've created
this descriptor object,
we call newSamplerState,
and we get back a Metal
Sampler State Object,
that we can subsequently use to
bind and sample from a texture.
that we can subsequently use to
bind and sample from a texture.
In the Render Command Encoder,
the API looks like this.
We create a texture so
we set it at Slot Zero
of the Fragment Texture
Argument Table.
And then we bind our
Sampler State at Index Zero
of the Sampler State Argument
Table for the fragmentFunction.
And let's look at those
functions in turn.
So the vertex function this
time around, will multiply
by the MVP Matrix that
we're going to get
out of -- a constant buffer.
It will then transform
the vertex positions
from all the local
space into Clip Space,
which is what we're obligated to
return from the vertex function.
And it will also transform
those vertex normal
from Models Local
Space into Eye Space,
so that we can do our lighting.
Here's what it looks
like in code.
So notice that we've added
a parameter attributed
with Buffer 1, and like
I mentioned earlier,
this corresponds to
the constants buffer.
So we've created a struct type
in our metal shaving
language code that corresponds
to the constant struct that
we created in our SWF code,
to the constant struct that
we created in our SWF code,
that allows us to fetch out
the Model View Projection
in normal matrices.
And again, this is bound
at Argument Table Index 1.
So that corresponds to the
attribute that you see there.
So, to actually move into Clip
Space, we index once again
into the vertexBuffer
at Vertex ID.
Get up a position vector.
Multiply it by the MVP
matrix and assign it
to the outgoing struct.
Similarly, for the normal.
And also, we just copy
through the texture coordinates
to the outgoing struct as well.
And all of these of
course will be interpolated
by the rasterizer.
So we just go ahead
and return that struct.
The fragmentFunction is a
little bit more involved
than previously.
We want to actually
compute some basic lighting,
so we'll include two terms of
ambient and diffuse lighting,
and also sample from the texture
[inaudible] you just bound
to apply the texture
to the surface.
It looks like this.
We're not going to talk
through this in exacting detail,
but the important
thing to note here is
that we've added a parameter
that corresponds to the texture
that we've created and bound,
we've given it an access
qualifier of sample
which allows us to
sample from it.
It's sitting at Argument
Table Index Zero.
The Sampler State that
we created is sitting
at Argument Slot Zero for the
sampler, and all we need to do
to actually read a
[inaudible] from the --
a filtered value
from the texture,
is call Sample, on the texture.
So Text2D.Sample, actually
it takes the sampler state,
as well as the texture
coordinates
and gives us back
the color vector.
We'll also go ahead and do
all of our fancy lighting,
but I won't talk
through any detail.
But it's just dependent upon the
dot product between the normal
and the lighting direction.
And we specified some constants
related to the light earlier
in our Shader file that
we'll see during the demo.
And that's pretty much it.
So we constructed the color
for this particular fragment
by multiplying through the value
that we sample from the texture,
by the lighting intensity,
to result in an animated
textured lit cube.
to result in an animated
textured lit cube.
And I will now let Matt
show you exactly that.
>> Alright, let's take
a look at this demo.
Here's the Metal texture mesh.
You can see, it's a
very complicated cube.
Some simple lighting,
and texturing,
on a nice colored background.
Go ahead and admire
it in all its glory,
and now we'll take a
look at the Shader.
So you can see some new stuff
in our Shader compared
to last time.
The first thing we take a look
at is this constant struct.
This corresponds
the Swift's direct
that has a 4 X 4 Model View
Projection Matrix, and a 3 X 3,
normal matrix, and
those are used
for the appropriate transforms.
As Warren mentioned, we
have some light data here.
Ambient light intensity,
which is quite low.
And the diffused light
intensity, which is quite high,
and the direction of
the light that we'll use
to actually compute
the dot product.
Our input and output structs
are slightly different.
We've got a little
more information
that we need to pass down now.
We have position.
We have the normal, which
we needed for the lighting,
and the texture coordinates
which we need to
apply the texture.
which we need to
apply the texture.
And similarly, when we output
from our vertex function,
we need that same data again.
So let's take a look
at the vertex function.
Just as Warren said,
it's basically just a couple
simple matrix, multiplies,
and then a pass through for
the texture coordinates.
And a quick look at
our fragmentFunction,
which is exactly what
Warren just showed you.
Now let's see how
the renderer looks.
A little more going on now.
So we have a little
bit of animation.
So we need to update a little
time step to know how much
to rotate our cube by.
So here we have a
little helper function
to update my time step
-- update with Time Step.
And that will change
our constants.
Just like Warren said, we
don't have much data that we'd
like to send over to the GPU,
so when you set vertex bytes,
send a small structure over,
which was the two
matrices before.
And that's what we'll use
to compute the animated
positions of our vertices.
Put the texture, the
samplers, and issue your Draws.
Put the texture, the
samplers, and issue your Draws.
Highly recommend you
guys always remember
to push your debug
groups so you know,
exactly what you're looking
at if you're going to look
at a frame capture later on.
Present your drawable and
commit, and then you're done.
>> Cool. Thanks again, Matt.
So with these adopting Metal
sessions, we really wanted
to take advantage of the
fact that we've had a couple
of years now, teaching Metal,
and introducing awesome
new utilities
that make Metal easy to use.
And so we hope that these --
this two-part session
is useful for that.
You've seen that
Metal is a powerful
and low overhead GPU programming
technology, and fortunately now,
you've become acquainted
with some of the APIs
that are available inside of it.
Metal is a very closely --
is very much informed by how
the GPU actually operates
and is you know, philosophically
of course, we want you to push
as much expensive work
up front as possible.
as much expensive work
up front as possible.
And so you've seen sort
of some of the ways
that that informs
the API as well.
And the emphasis of course
is not on the restrictions
that that entails,
but of course,
the power that it
imbues you with.
So you've seen how
explicit memory management
and command submission can let
you work a little bit smarter,
in a sense that if you know
how your application is shaped
and you know what it's
doing, then you can actually,
you can take the reins and
control the GPU directly.
And of course, over the next
few sessions on Metal here
at WWDC this year,
we'll show you even more
that Metal has in store.
And then of course, it
will be your turn to go
and build awesome
new experiences.
So for more information on this
session, Session Number 602,
you can go to this
URL, and of course,
there are some related sessions.
Part II will be happening in
this very room, very shortly.
And tomorrow we have, What's
New in Metal, Parts I and II.
And the Advanced Metal
Shader Optimization talk
that I mentioned.
So thank you, and
have a wonderful WWDC.
[ Applause ]