Transcript
>> Good morning, everyone, and
welcome to the SceneKit, What's
New session.
As you already know, SceneKit is
Apple's high-level API for 3D.
It's built on top of Metal and
it's available on all our
platforms.
This session will be about the
new feature we are adding to
SceneKit.
We will not cover the basics
today, so if you are new to
SceneKit, I encourage you to
watch our previous presentations
from the past years.
So here's our agenda for today.
I will first present some camera
improvements that includes some
new camera effects and some new
APIs to simplify the control of
cameras.
Then, Amaury will come on stage
and talk about tessellation and
subdivision, some improvements
and new APIs on the animation
front, and to finish, we'll
present some new developer tools
and talk about some related
technologies, including ARKit.
Now, to start, I would like to
show you a little demo that
illustrates some of the new API
I'm going to detail later during
the session.
So this demo is a simple game
example with a character that I
can manipulate with a virtual
D-pad.
I can attack, jump, and we are
on the camera with the virtual
D-pad.
And the first thing I would like
to highlight in the demo is the
behavior of the camera.
As you can see, the camera
follows the character smoothly.
And by smoothly, I mean it
doesn't strictly reproduce the
movement of the character, but
it tries to keep a constant
distance an elevation with the
character, but always move with
a smooth acceleration and
deceleration.
And you will see that the
behavior of the camera will
adapt, depending on the
different zones of the game.
For example, here as I approach
this fight area, the camera
moved down a little bit and
adjust the depth of field to
focus on the character and the
enemies.
Speaking of enemies, we have two
enemies here with very basic
behavior.
One is chasing me while the
other is moving away, and I will
explain briefly how this was
implemented using GameplayKit.
For now, let's just kill them,
and the other one.
And collect that gem here.
Here the camera smoothly
transitioned to a cinematic view
with a strong depth of field to
emphasize the key, and you can
notice also some nice
bokeh in the background.
So let's collect that key.
And as I get close to this new
zone here, where I have to
carefully jump on platforms, the
camera smoothly transitioned to
a new behavior where it now
stops rotating and stay aligned
with the platform.
To simplify my jumps to the
platforms, so let's jump.
That was intentional.
And let's collect the key.
Again, a new zone where the
camera will automatically
reorient itself and stay aligned
with the platform.
And it looks like there are some
friends to free here, and I have
the key, so let's do it.
And to finish, the new cinematic
view with many characters
running around.
And here we have 3200 bots to
animate, and we are going to
achieve that in half a millisecond on
this new iPad Pro.
So this is really fast.
We have been focusing on
character animation performance
in this release.
And implementing all this has
been much simpler with the new
APIs, with the new animation
APIs I will present later.
So that's all the demo.
[ Applause ]
Other ways we are really happy
to share this demo with you as a
sample code, and the sample code
runs on iOS, tvOS, and macOS,
and is available in Swift and
Objective-C.
During the demo, I insisted on
the camera behavior because this
is something that is really
difficult to write.
We are seeing many questions and
related requests on developer
forums about this, and so we
are, we improved our camera API
in this release to both simplify
this problem and also improve
the ongoing quality.
So to do this, we are
transitioning to a physically
based camera API, and we
leveraged this new API to
implement the physically
plausible depth of field.
We also improved the motion blur
and added built-in support
for screen space ambient
occlusion.
And I will then, I will talk
next about the new APIs, what I
did to simplify the control of
cameras.
So transitioning to physically
based API, camera API.
First, we are deprecating our
legacy projection model, so
will, of course, ensure the
backward compatibility, but be aware that
we are moving away from the x4
and y4 properties to adopt some
things that closer match real photo cameras.
For example, if you want to
configure your perspective projection
projection, you can now either
set the fieldOfView property or
configure the focalLength and
sensorHeight.
These properties are linked
together, so if you configure
the focalLength for instance,
it will update the fieldOfView
accordingly, and vice versa.
Then, SceneKit is modeling
a real photo camera depth of
field.
To achieve at the depth of
field, set the wantsDepthOfField
property to true, and then
configure it by setting the
focusDistance and fStop
properties.
SceneKit will approximate
real photo camera depth
of field, and it will blur the
scene in a way that is
consistent with these parameters
that come from the photography
world.
So new depth of field will also
simulate the bokeh you would get
with real photo cameras.
Bokeh appear on bright objects
that are out of focus, and
because they are generated by
pixels with very high intensity,
this feature works best if you
render your scene with an HDR
camera.
And to configure this, just set
wantsHDR property to true on
SCNCamera.
Then, the shape of the bokeh
depends on the number of blade
of the aperture.
And you can also configure that
on SCNCamera.
Here are a few examples with a
different value for this
property.
Then, we improved our motion
blur in this release.
We already presented a first
report of motion blur last year
that was able to blur the scene
based on the motion of the
camera.
That means that if your camera
was moving fast in a scene, you
would get the motion blur.
But if your camera was static
and the objects were moving
around, you would not get any
motion blur.
So in this release, we are
adding support for object motion
blur, and you get per-object
blur automatically when you
activate the motion blur on your
camera.
One more effect provided by
SCNCamera is now the
ambient occlusion.
The principle of ambient
occlusion is simple.
The idea is that a point on a
flat surface receives all the
environment incoming light,
whereas points in cavities will
just receive part of it because
some of this light will be
occluded by the surface.
Simply to put, screen space
ambient occlusion, which means
that this occlusion factor will
be computed in screen space for
every pixels.
This is done by analyzing the
depth buffer and normal buffer,
and SceneKit will determine if a
point is in a cavity or not by
comparing its depth and normal
with the neighbor fragments.
So here's an example of an
object with no ambient
occlusion, and here with a very
strong ambient occlusion to make
the effect of use on the slide.
To activate the screen space
ambient occlusion, just set the
screenSpaceAmbient
OcclusionIntensity to a value
greater than 0, and then you
have a few parameters you can
tune that will depend on the
look you want to achieve and on
the size and the topology of
your scene.
But let's see all these effects
live in a demo, and for this,
please welcome Anatole on stage.
[ Applause ]
>> Thank you, Thomas.
Good morning, everyone.
So let's come back to the first
demo and bring up some debug UI
to help me show you our new
depth of field effect.
Here we have -- sorry.
Here we have a beautiful
golden key in the foreground.
The focus distance is set to a
small distance because the key
is close to the camera.
And the effect number is set to
a small value because we want a
strong blur.
As you can see, nice work here
on the background due to the
illuminance of the particle.
Now, with the second camera, I
now have a far-focused distance.
So the objects in the background
are sharp and the flowers in the
foreground are blurred.
Yet another point of view, and
here I can, for example, play
with the FStop number to get
more or less blur.
And I can wait for the camera.
And then with the second slider,
we define where we want to focus
in this scene.
So this new depth of field
effect can be incredibly useful
for instance to produce
cinematic effect in video game.
Now, let's open another app to
show you of our new per
object motion blur.
Here we have a scene with a
tower of blocks.
If I press the Shoot button, I
throw some spheres on the tower.
And this is how it looks by
default without any motion blur.
Now, I reset the scene, enable
the Motion Blur, and toss
in some more spheres.
You can see the effect of the
motion blur on those spheres
because they move really fast.
And you can also see the effect
applied to the blocks when the
tower explodes.
But now, let's take a closer
look.
I can for that freeze the scene
and move closer a little bit.
Our brain interprets the blur's
movement, and you can actually
see the objects in motion
even if the image is static.
We can even change the point of
view, and we still have a good
idea of the direction of each
object.
So this really improves the
perception of motion in the
scene, and the result looks more
realistic.
And now, let's see the screen
space ambient occlusion in
demo.
You can see some spheres
illuminated by the sky and the
direction of light.
With the first slider, I can add
some ambient occlusion to the
scene.
As you can see, some ambient
shadow is added, and I can
change the intensity to give the
sphere more or less shadows.
The amount of occlusion depend
on the curvature of the surface.
To know if a pixel is in the
cavity or not, we inspect the
neighbor pixels.
And so we have this radius
parameter that let us define how
far we look from neighboring
pixels.
The visual result is sharper
occlusion with small radius and
more spread out shadows with
large radius.
This is computed in real time,
so it is perfect if you work
with dynamic objects when
pre-baked ambient occlusion
maps or [inaudible].
That adds some detail to the
perception of the deeps -- of
the depths, sorry -- and take a
look of global illumination to
your scene.
That's it for the demo.
Let's come back to the slide,
and I'll hand it back to Thomas.
[ Applause ]
Thank you, Anatole.
So we have talked about some new
camera effects.
Now, let's talk about camera
control.
I did say earlier this is a
difficult problem, and we see
many questions about it.
And we identified two
main use cases -- people will want
to inspect a 3D object by
rotating a 3D object or rotating
around the 3D object.
For example, developers who are
building a simple 3D viewer or
an editor and developers who
need a, some more sophisticated
camera behavior.
For example, for games or a more
advanced app.
So let's start with the first
use case for now.
Until now, if you wanted to
manipulate a 3D object, you had
to implement your own management
of events and move the camera
position and orientation based
on gestures or mouse events.
For convenience, we are
providing an allowCameraControl
API on SCNView, but these were
just providing you a default
camera behavior that was not
configurable and that was
essentially there for debugging
properties.
So in this new release, we are
introducing a new class named
SCNCameraController, and the
SCNCameraController allows you
to manipulate a camera with the
most common camera behavior you
would find in 3D software.
So the, this behavior had built
in in the camera controller, and
the SCNView has a built-in
default camera controller that
you can directly configure for
the need of your application.
Now, if you need something more
specific, you can still
instantiate your own
SCNCameraController and drive it
programmatically if you want.
So the SCNCameraController
provide out of the box most of
the common camera operation
tools.
And to give some examples, the
Orbit Turntable allows you to
orbit your camera around a 3D
object and will prevent roll.
That means that the horizon will
always remain level, regardless
of the rotations you are doing.
The Orbit Arcball will orbit the
camera using the vertical and
horizontal axes in screen space.
So this mode can be more
intuitive in some cases, but it
doesn't prevent roll, so it
really depends on your
application.
[inaudible] the Fly mode is more
suitable for lap scenes you want
to maybe get into.
And in that case, the center of
rotation of the camera is a
camera itself, which means that
you rotate the camera to look
around in a position to orbit
around an object.
So again, we believe that the
scene camera controller will
provide most of the common
camera operation tools.
And if you need something very
specific, you will be able, you
can still try your camera
controller and your camera
programmatically.
Now, let's see the second class
of programs.
Developing, we need a more
sophisticated camera behavior.
For example, for games.
And we address this problem by
chaining constraints to define a
camera behavior.
SceneKit was already providing a
bunch of built-in constraints,
and we're adding a few new this
year that will work on any
arbitrary node but work well in
particular for camera.
And to illustrate some of them,
the SCNDistanceConstraint forces
a node to keep a minimum and
maximum distance with another
specified target node.
So replicate our constraints.
We replicate a node position and
orientation with an optional
offset.
And the acceleration constraint
will ensure that the node won't
move or accelerate faster than a
given maximum velocity and
acceleration.
So these are just examples.
Let's see what we can do with
this constraint.
So here we have a character
moving around in a scene, and
the camera has no constraint
yet, and he's therefore static.
If I add the look at constraint
to the camera with the character
as a target node, I now have my
camera that rotates to satisfy
the look at constraint and I
have the camera that look in the
direction of the character.
If I add a replicator constraint
and I have basic camera
behavior, that replicate the
movement of the character with
some offset and continue to look
in the direction of the
character.
If I add into that an
acceleration constraint, I now
have the same behavior, but that
is most thanks to the
acceleration constraint that is
applied after the other
constraint.
And if I replace the replicator
constraint by a distance
constraint, I now have a new
camera behavior that now follows
a character to satisfy the
distance constraint.
It continued to look into the
direction of the character, of
course, and all this will be
always moves, regardless of the
movement of the character,
thanks to the acceleration
constraint.
And so that's how easy it is to
define a camera behavior, and
that's what we did in our Fox 2
example.
And just by defining a different
set of constraints, depending on
the different zones of the game,
we were able to define the
camera behavior for the entire
game.
One more note about camera
control.
We extended SCNNode with the
categories that provide much
utilities for you to convert and
access vectors in different
spaces.
But most of all, all the node
transform properties,
access position, rotation,
scale, and matrix, and
transform, are now directly
available as SIMD properties
for, to ease math operations.
So thanks to SIMD types,
operations on quaternions
vectors, and matrices are much
simpler to write and they are
much more performant anyway.
Just be aware of a few
limitations with SMD types, as
they are not KVO and KVC
compliant and they cannot be
included as an NSValue.
So that's for camera controller.
Now, I'll hand it over to Amaury
to talk about tessellation and
subdivision.
[ Applause ]
>> Thank you, Thomas.
So we know that great graphics
are essential to your
application to build engagement
and delight your users.
And there are many aspects to
great graphics.
For instance, there is added
realism.
And that's why over the years
we've introduced new rendering
capabilities, such as
physically-based shading and
this year's push for more realistic
camera optics.
But high-resolution assets is
another very important aspect.
In your application, you want to
be able to have both very smooth
surfaces as well as very rich
and detailed ones.
Now, the issue is that when you
deal with high-resolution
assets, they require more
memory, both on disk and at
runtime, and they would require
more processing time.
So in this section, we will have
a look at proper techniques that
allow artists and you developers
to work with low-resolution
models that can become of high
quality when rendered on the
screen.
So I will start by explaining
what tessellation is and how it
works, then I will show you how
you can leverage this in
SceneKit.
And finally, we will have a look
at something a little different,
subdivision surfaces.
So tessellation.
Tessellation is a feature
applied last year in Metal, and
the idea behind this feature is
that you provide the GPU with a
low-resolution mesh, or coarse
mesh, and then you let the GPU
generate model memory that has
more vertices on the fly when
the model is rendered.
So subdivision surfaces really
are a powerful tool.
They are what you use in the
industry to easily create,
store, and animate
low-resolution models that can
become of very high quality when
rendered.
So let's have a look.
So this is a triangle, and this
is a tessellated version of it.
So SceneKit gets to decide how
much an edge can be split.
It can decide how much vertices
can be created on the first edge
and on the second edge.
And of course, it can do that
for the third edge.
And what's nice is that SceneKit
also gets to generate more
vertices inside the triangle.
So these are called tessellation
factors.
SceneKit is a high-level API,
and we made it super easy for
you to perform tessellation.
We are adding the new
SCNGeometryTessellator class as
well as the tessellator property
on SCNGeometry.
And the tessellator exposes a
few properties that allow for
different modes.
So let's have a look at the
simplest example first.
In this mode, you provide
SceneKit with constant edge and
inside tessellation factors that
will be used for all the
triangles in the coarse mesh.
So with this mode, you will have
a uniform tessellation, and you
will add the same amount of
geometry everywhere across the
coarse mesh.
So let's take a look at the more
complex example now.
Here you can ask SceneKit to
come up with special
tessellation factors so that no
edge is too long.
So here you provide a maximum
edge length in local space.
Even more powerful, you can ask
SceneKit to constantly evaluate
at each frame the tessellation
factors, depending on the
project inside of the object.
So in this mode, you will
provide the maximum edge length
in screen space that is in
pixels.
So that's it for tessellation.
And now, if you have a look at
the original triangle and the
tessellated version, you might
be a little disappointed, and
that's because all the new
geometry data actually lies in
the original triangle.
So for your highly-detailed
mesh, you want to do something
with this extra geometry.
And that leads us to the new
tessellation-based geometry
APIs.
So first of all, let me remind
you of shader modifiers.
Shader modifiers are completely
supported with the new
tessellation pipeline.
And with a few lines of code,
you can create really custom
effects.
So for instance, if you have an
application, and there's water,
and you want to simulate an
ocean with waves or really any
idea, an effect of your own,
shader modifiers are the right
tools, are really powerful for
that.
But of course, we are also
adding out-of-the-box effects,
such as geometry smoothing.
This is a new feature.
And for instance, if you specify
the pnTriangles smoothingMode,
SceneKit will take into account
the position in normal of each
vertex as well as the position
in normal of its neighbors to
project them on a smooth
surface.
Another effect that you might
have heard about -- displacement
maps and height maps.
So what's a height map?
Well, height map is a
gray-scaled image that stores
the elevation or altitude of any
point on the surface.
This technique is commonly used
for effects such as terrain
rendering, so let's take that as
an example.
We start with a plane that we
tessellate and that we deform
using the height map.
So it's a really simple example,
but it's highly effective.
And the API is really simple
too.
We are adding the new
displacement material property
on SCNMaterial, and so I bet you
already know how to use it.
You just specify its contents,
and then if you modify its
intensity, you can come up with
the animation I just showed.
Now, let's get one step further
with vector displacement maps.
Vector displacement maps are the
extension of height maps, but
instead of only storing the
elevation, you can store a
displacement is all three
directions, and that's why you
have a cut-out image.
For instance, green is the
displacement along the normal,
and red and blue along the
tangent and bitangent.
So will you guess what that
does?
Let's have a look.
Okay, so this is a, see, the
example, but in your
application, you can use
displacement maps to add the
details, fine details to your
geometry.
So for instance, if you have a
demo with a chameleon, you can
add detail to its skin.
Or if you have an application
and there are rocks you can get
really close to, vector
displacement maps are the right
tool.
The API is the same, except that
instead of just red, you specify
"all" for the texture components
to indicate that you are
interested in more than one
color option of the input image.
So that's it for tessellation
and tessellation-based effects.
Now, let's have a look at
subdivision surfaces.
So you might already have heard
of our subdivision surfaces and
[inaudible] class subdivision.
It's a standardized algorithm
that starts with a coarse mesh
and that iteratively refines it.
And so you see how quickly we
get from a coarse mesh to a very
smooth and detailed one.
Now, not everything is perfectly
round, so with subdivision
surfaces, you can specify
creases and corners to have
distinct sharpnesses for your
edges and vertices.
So subdivision surfaces, they
are extensively used in the
industry to easily create,
store, and animate your
resolution models that can
become of very high quality when
rendered on the screen.
And it turns out that we added
support for subdivision surfaces
a few years ago in SceneKit, but
we used to run the subdivision
code on the CPU.
And so that takes some time and
a huge amount of memory,
especially when you go to higher
level of subdivision, as the
number of vertices generated
grows exponentially.
So we have great news this year.
You might have heard about the
OpenSubdiv project from Pixar,
which is an open tools
implementation for efficiently
evaluation of subdivision
surfaces.
And last year at WWDC, Apple
announced that we would be
contributing to this project
with a Metal-based
implementation so that you can
run the subdivision code on the
GPU using Metal.
And so this year, you can
leverage all these amazing
technologies very easily.
And with the Metal-based
implementation come many
advantages.
First, we leverage tessellation,
and it comes with all the memory
benefits I've talked about
earlier.
And with the tessellation, we'll
have very smooth surfaces even
for low subdivision levels.
Now, in addition to uniform
subdivision, we support
feature-adaptive subdivision,
which I will explain in a
minute.
And last, we have an all-GPU
pipeline for efficient animation
of subdivided meshes.
So let's take a look at this
example.
It's the key you saw in the
demo, and as you see, it's a
very coarse mesh.
Not much detail.
And this is a tessellation
version, subdivided one.
So with asset, you can see that
you have hard edges but also
nice curves.
So how did we do that?
Using creases.
So with subdivision surfaces,
artists can really come up with
great designs and they can
easily create them and then tune
them to have the desired look
they want.
Now, feature-adaptive
subdivision.
When uniform subdivision
converges towards a [inaudible]
face that this moves by
exponentially increasing the
number of polygons,
feature-adaptive subdivision can
isolate irregular parts of your
mesh and create busy patches as
well.
So then, with tessellation, we
can create new vertices on these
perfect mathematical curves, and
so it leads to very smooth
surfaces with a lower memory
footprint.
Now, the API is really simple.
You just specify a subdivision
level and then you opt into
tessellation.
And then for feature-adaptive
subdivision, it's really easy to
configure too.
Now, last, animation of
subdivision surfaces.
This year, we have an all-GPU
pipeline that is really
efficient.
You can have a coarse mesh that
you can deform using morphing.
And then if you want, you can
add skeletal with skinning.
And finally, as the last step,
we run the refinement code on
the GPU.
And this is very performant
because we are working on the
GPU on the low-resolution mesh,
and then the very detailed one
is generated on the fly by the
GPU using tessellation.
So now that you want to give
subdivision surfaces a try, just
remember two things.
First, if you are loading a set
from files, specify the
preserveOriginalTopology option.
And if you are creating
geometries programmatically,
remember to use the polygon
primitive type.
This is because with
subdivision, working with
triangles is not the same as
working with quads.
And with that, let's have a
quick demo.
OK. So this is a simple
pottery application I'm
going to be working on.
So I can easily pinch to zoom,
and I can drag to rotate.
It's very easy.
Now, if I pinch, then pay
attention to the silhouette of
the object.
As you can see, it's very
smooth, and you can use a normal
map to add detail on the
surface.
And now, the purpose of this
application is very simple.
I can take my finger and simply
sketch on the mesh.
So let's kill that and let's
write something.
So as I draw, pay a look at the
silhouette of the object.
Here I'm actually modifying the
geometry.
I'm not just adding surface
details like we do with a normal
map.
OK, so how was that done?
Well, with subdivision surfaces,
tessellation, and height maps.
So let's have a look.
What we do is that we start with
a very coarse mesh.
As you can see, very low
polygon.
It doesn't come with normals, so
we have flat shading here, but
it has texture coordinates.
So we can simply map a normal on
it and later a height map.
So now, let's subdivide it.
Let's see the smooth normals
that are generated, and then
let's have a look at the wire
frame and see how much vertices
are created, how many vertices
are created when I enabled
subdivision on it.
When I take my finger, I simply
draw in the height map, and all
the vertices are displaced.
Now, let's clear that.
And just for fun, let's enable
screen space tessellation.
Now, I will pinch and take a
look at when I get close or
farther from the object.
SceneKit will come up with new
tessellation factors, and it
will create new vertices on the
fly.
And that's it for the demo.
[ Applause ]
Thank you.
So as a wrap up, tessellation
and features relying on
tessellation are available with
Metal on all Macs and available
on iOS devices with the A9 chip
or later, so that includes
iPhone 6S and all of the iPad
Pro models.
Now, let's have a look at
something completely different
-- our enhancements to animation
APIs.
This year, we are introducing
the new SCNAnimation protocol as
well as the SCNAnimationPlayer
class.
They make it easier to start
animations and to mutate it,
mutate them while they are
running.
So for instance, now you can
easily change the speed of an
animation and you can blend
animations on the fly.
Of course, we still fully
support CA animation APIs.
CA animation conforms to our new
protocol.
But with the new APIs, it's much
easier.
You can work dynamically on
animations while they are
running, and these APIs are
available on all platforms,
including watchOS.
So let's take a look at the old
way.
So let's say you had a character
that could walk and jump.
You would first start by adding
the walk animation, and then
when you wanted the character to
jump, you would add the jump
animation to replace the other
one.
Now, with the new API, you start
by creating and configuring
animation players, and then when
you want a character to jump,
you manipulate the player
instead of the animation
directly.
So it's a very similar API.
The difference is that now you
can mutate animations while they
are running.
So you can change their speed
and you can mix animations.
And animation blending is
actually new this year.
So let's take the example of
Max, who can walk, run, but also
step.
We have different animation
files for each kind of motion.
With the new blending API, you
can easily transition from the
step animation to the walk
animation, and so you can bring
fluidity and be more expressive
in your applications.
And after you blend animation,
you can also play with the
speed.
So Max can run slower on it.
Finally, let me mention
enhancements to our animation
evaluation code.
So we have a new implementation
that makes it faster to start
animations on arbitrary objects
in the scene.
And we made the evaluation of
skeletal animation much more
performant.
So if you have a lot of
characters, you have a lot of
bones in your scene, such as in
the Fox demo you just saw.
We have this new implementation
that makes things much faster.
And with that, let me hand over
to Sebastien for updates on our
developer tools.
[ Applause ]
>> Thank you, Amaury.
So last year, we introduced FPS
Gauges.
It's a great way to have an
overview of the performance of
your application regarding
SceneKit.
And they split the categories so
that you can see exactly where
the time is taken for the CPU
and the GPU.
So you can know if it's going in
the rendering and physics or
particles.
It's integrated in Xcode, and
you can have a look at exactly
how it behaves all the time.
And it's cool to exactly know
what's taking time so you can
reuse your meshes or animations,
for example.
But what happens with, when you
skip a frame?
How do you know exactly what
happened?
What caused the frame skipping?
This year, we are introducing a
new instrument.
It's a template for SceneKit
that you can use to record the
trace of your application and
exactly know what happens frame
by frame.
It's very simple to use.
You just create a template as
you would for any other trace,
and it'll record your
application performance, and you
will get this view.
It's full length that detail
what's happening in your
application.
The first one is the frame.
It gives you the time that is
taken to render one frame for
your application, and you can
see exactly how long it takes.
The second one gives you the
rendering time.
It's the time that is taken by
SceneKit to actually gather all
the data and send it to the GPU.
The third one gives you the
updating stage.
It's the time that is taken to
update the physics, particles,
and your custom delegate, if you
have one.
And the last one, but not the
least, is the time that is taken
to upload texture to the GPU as
well as compile shaders.
And let's see how it looks when
we have a skipped frame.
This is a very simple example.
You can see that all the frames
are very slow to render and that
at one point we have one frame
that is taking the time that it
would take normally for four
frames to render.
And when we dig down, we dig
what exactly is happening, we
can see that a new shader was
compiled, and it takes a lot of
time.
In this case, we can know, maybe
try to find a strategy to load
the shader at the start of the
application.
We have also added a way to
combine the SceneKit instrument
with the Metal instrument trace.
So you can see the combination
of both at the same time and see
exactly what happens behind the
screen to understand what's
happening in your application.
This year, we have also added a
new debug tool.
It's an enhancement to the view
debugger in Xcode.
It's very simple to use.
You just use the regular view
debugger as you would normally.
And it will capture the view
hierarchy as well as the scene
automatically.
So if there is a SceneKit scene,
SceneKit view in your
application, it will capture the
scene.
And if you select the scene in
the view hierarchy, it will
automatically send it to the
SceneKit editor, where you can
inspect all your objects, move
the camera around, and see
exactly how, what is in your
application and what is in your
scene.
We have also added support for
the new features this year.
So we have a new way to handle
the cameras in Xcode.
You can see that there is a new
way to choose the behavior that
you want.
We have added perspective and
autographic cameras so that you
can actually inspect your scene
much easier, and you'll still
have access to all your regular
cameras.
We have also added the new
behavior, so you can fly around
on turn and use the arcball.
It's much easier to inspect very
big scenes.
We have also completely revamped
the Shader Modifier Editor.
So now, you can edit your shader
modifiers as well as your
material at the same time in one
screen.
You don't have to go back and
forth in selecting objects.
It's a completely new
implementation, and it also
supports custom material
properties.
So if there is no, not enough
property slots in your material,
you can add colors, floats, or
vectors to add your own
implementation in your shader
modifier.
Very easy to use.
We have added support for more
features that we have added this
year.
So first is there is a new
displacement material slot that
you can use for the
tessellation.
Of course, we have tessellation.
We have added support for the
new constraints.
So you can add them into your
nodes, and test them in real
time in Xcode, and edit them in
the inspector.
We have support for cascaded
shadows that we will tell you
about more later.
And we also have a new
procedural sky that is very easy
to use as a background or the
lighting environment.
For example, to test your
[inaudible] material when you
don't have a proper [inaudible]
map set.
And it's completely
configurable, so you can get day
or night sky, for example.
And last, but not least, we have
added the possibility to
override the material for
reference nodes.
And what are reference nodes?
Reference nodes are actually
nodes that refer to one, only
one, scene file but used more
than once in your scene.
Until now, you could only have
exactly the same rendering for
both, for all the nodes.
It's not only two nodes.
But now, you can overwrite some
or all of the materials of, that
you are using in the scene and
change the look of some of, or
all the instance.
And with that, I hand it over to
Thomas to tell you about related
technologies.
[ Applause ]
>> Thank you.
OK, so let's talk about related
technologies and start with
ARKit.
So you have [applause] -- yeah.
[laughs] So you have all seen
the introduction of ARKit on
Monday during the keynote and
State of the Union, and you may
have noticed that ARKit provides
an ARSCNView that provides an
easy solution for AR that just
work out of the box.
And in fact, the ARSCNView is a
subclass of SCNView.
And that means that you have
full access to SceneKit via
ARKit.
You have access to the scene
graph.
You can add post processes,
particles, physics, force fields,
custom shaders.
If you want, you can do
basically everything.
And so with ARKit and SceneKit,
it's really easy to set up a
scene like in this example here.
I would let you guess where this
video is shot.
There is a hint in the
background, and my accent is
another hint.
And so here, to set up a scene
like this, we just needed to
load a SceneKit object, as you
usually do, and set up an ARView,
and run an AR session.
And then, from the ARSCNView
delegate, you just need to
attach your 3D objects
to the anchor detected by ARKit.
It's that simple.
And to support ARSCNView, we
extended our material property
to support AVCaptureDevice and
AVPlayer as a natively supported
type of content.
So that means that it's very
easy.
With just one line of code, you
can now directly connect.
So you do a feed off your iPhone
or iPad directly to a texture in
SceneKit or to the scene
background.
Yeah.
[ Applause ]
Now, I would like to give you a
little trick regarding augmented
reality and shadows.
So the fact is that your object
will look better integrated if
your object casts some shadows
on the ground like in this
example here.
And so the trick is that to
achieve this, we editing our
object in Xcode, we added a
directional light that casts
some shadows, and then we added
a plane to receive the shadows.
Now, the goal is to hide the
plane because we don't want to
see it in our scene, but we
cannot simply make the plane
hidden.
Otherwise, the shadow would
disappear.
So the trick is to configure the
plane to not write in the color
buffer.
This can be done
programmatically or using the
inspector in Xcode.
And if I do this, the plane will
disappear, but the shadow as
well.
But the plane still writes in
the depth buffer, which means
that I cannot configure my
light, and change the shadow
technique, and move from forward
here to default shadows.
And now, the shadow will come
back because default shadows
work as a second pass full
screen, and it will compose
shadows over the image based on
the depth buffer of the scene
and the depth buffer of the
light map.
And so for default shadows, the
plane still exists because the
plane still rendered into the
depth buffer.
So that's for the trick for
shadows in ARKit.
Now, I would like to talk about
GameplayKit.
GameplayKit entity and
components now support driving
SceneKit objects.
The typical use case is when you
want to implement character
behaviors, and that's what we
did in our Fox 2 example for the
enemies.
In our case, we have GKScene
with two entities, one for each
enemy.
And we implemented two behaviors
as GKComponent.
Now, the main interest resides
in the Xcode integration that
allows us to directly assign the
behavior to our enemies directly
in Xcode.
And we can also use the Xcode
inspector to directly edit the
properties of our behaviors.
Next, Model I/O.
Model I/O improved their support
for USD in the 3D.
As a reminder, USD stands for
Universal Scene Description, and
it is a 3D file format developed
by Pixar.
So SceneKit and Model I/O
improved the support for USD
and, in particular, for Metal
I/O's and for animations.
If you want more information
about USD, you can check our
previous presentation from last
year, and that will also be in
our session on Friday afternoon,
From Art to Engine with Model
I/O.
Next is UIFocus.
And so UIFocus engine is part of
UIKit, and it allows you to
select and focus objects on your
Apple TV using the Siri Remote.
Now, SCN, it now conforms to
UIFocusItem, which means that
you can decide which object in
your scene, which object would
be focusable on it.
And then, the focus engine will
call you back and will tell you
what object should be now
focused in response to gestures
on the Apple Siri Remote.
And then, it's up to you to
decide what action you want to
take and what visual feedback
you want to provide.
To briefly explain how this
works, let's say here I
configure the white pieces to be
focusable.
SceneKit will automatically
compute the projected area of
your object and give that to the
focus engine.
Then, the focus engine will take
care of selecting the right
object based on the gestures on
the remote.
And SceneKit will take care of
keeping the projected areas on
the screen updated if you move
your objects or if you move the
camera around.
So the only thing you have to do
here is just to define what
object should be focusable on
it, and that's all.
[ Applause ]
Now, to conclude this
presentation, I would like to
mention a few rendering
additions that we added to our
renderer.
And the first one is a support
for point cloud rendering.
We improved our [inaudible]
object and added these
properties that will allow you
to configure the appearance of
point cloud.
So these properties were
[inaudible] point cloud, which
means that they were created
with an area of point and with a
primitive type point.
And with these properties, you
can configure the size in screen
space and in world space, and
SceneKit will render your point
cloud, and texture them, and
light them according to the
materials that are attached to
your geometry.
Then, we added two new
transparency modes to our
material to address the problems
of double-sided objects or
concave objects that are
semitransparent.
So if you take the object on the
left, you can see a double-sided
sphere, semitransparent, and you
will notice the artifact because
the polygons are not rendered
from back to front.
So SceneKit will render your
objects from back to front for
great transparency, but it won't
sort the individual polygons of
a geometry.
And so to address that problem,
we introduced two new
transparency mode.
So first one, single layer, will
render your object in two passes
to only render the front-most
faces in the second pass.
So that fixes the artifact, but,
as you can see, object doesn't
look double sided.
And the typical use case for
this mode is when you want to
fade out an object and avoid the
artifacts due to the overlapping
polygon during the fade-out.
So second mode, dual layer, will
also render your object in two
passes.
So first pass with render the
back faces, and the second pass
will render the front faces.
And it allows us to render this
double-sided sphere correctly
here and also the gem from our
Fox demo.
And this gem, that comes from
the Swift Playground Learn to
Code curriculum.
One last addition is a support
for cascaded shadow maps.
So cascaded shadow maps is an
optimization of shadow maps.
The idea is that it splits your
shadow maps into multiple
textures or multiple cascades to
allocate more precision to areas
that are closer to the camera
and less precision for areas
that are far away.
To configure cascaded shadow
maps, just tell us how many
cascades you want, you can
configure the size of your
cascade, and then you have a
parameter named
shadowCascadeSplittingFactor
that you can adjust to control
the distribution of your
cascades, depending on the
distance from the point of view.
For example, this is a recording
on, from our Fox 2 example, and
in our case, we're using full
cascade, represented by
different color tint here, and
here I'm playing with the
splitting factor so that you can
get an idea on how this impacts
distribution on the cascade
depending on the distance from
the point of view.
As you can see, the right area
represents the first cascade,
and so it represents a smaller
area of the world, and so we
have higher precision there,
whereas the green cascade in the
back represents a much larger
area, and so we're fewer
precision for the green cascade.
Note that you can, you are also
able to visualize cascaded
shadow maps with your own scene
using Xcode.
Okay, to wrap up, so this new
version of SceneKit introduced
new camera APIs to simplify the
control of cameras with some new
great effects like depths of
field, motion blur, and screen
space ambient occlusion.
Also, a great GPU support for
tessellation and subdivision.
Some new APIs.
Much more performant for
animations and with the
animation blending.
Some new, great tools to
capture, trace, and edit your
scenes.
And of course, a really great
story for augmented reality
thanks to ARKit.
For more information, please
check our developer website.
You can access our sample code
from there.
Some related sessions.
So [inaudible], graphic
technologies that SceneKit
integrate with, like Metal,
SpriteKit, and Model I/O, of
course.
There will be, there is a
session about tvOS and UIFocus
support.
And there will be a great
session tomorrow about SceneKit,
using SceneKit in Swift
Playground tomorrow morning.
And that's about it.
Thank you.
[ Applause ]