Transcript
[ Music ]
[ Applause ]
>> Hi, I'm Alex.
I work for a group at Apple
called Tools Foundation.
Normally we get to do fun stuff
like operating systems and
compilers.
But this year we got to do
something a little different.
We built a game called
SwiftShot.
Some of you may have seen it
earlier today and you might have
played it downstairs.
But the important part is that
SwiftShot is a showcase of some
of the new functionality in
ARKit.
ARKit 2 is now available on
hundreds of millions of devices,
providing a platform for
engaging AR experiences.
And you are able to reach the
widest possible audience with
that.
There is no special setup, just
point the device's camera at a
surface and start playing.
It's integrated into iOS.
First-party and third-party
engines like SceneKit and
SpriteKit as well as third-party
ones like Unreal and Unity have
a full integration at this
point.
A little agenda for you.
First we're going to talk some
game design principles for
augmented reality, a few of the
things we learned along the way.
We are going to go deep into the
internals of the game and in
particular, we are going to
cover WorldMap sharing which is
a new feature in ARKit 2.
And we will also talk about
networking and physics and how
we made that work.
First, let's, you know, take a
deep look at the game.
[ Music ]
So, let's talk a little bit
about designing games for
augmented reality.
Above all else, gameplay comes
first.
You should ask yourself if you
are designing a game, would this
game be fun and enjoyable if it
were just 1970s graphics or
plain, flat-shaded grey cubes.
It is worth prototyping with
those kinds of artwork and get
that gameplay down.
Because if it's fun with those
boring grey boxes, it's going to
be fun when you add all the
graphics and sound later.
You should spend time refining
that and don't convince yourself
that if I just add another 5%
better graphics, or that one
feature, that the game is
suddenly going to be fun.
Because, you know, there's a
wasteland of games out there
that were never fun from the
get-go.
So, try not to fool yourself.
Let's start with the gameplay.
Keep games short.
You are looking for a typical
mobile experience still - easy
in, easy out.
You want to keep a variety of
content so that it is fresh,
avoid mental fatigue on the part
of the player of repeating the
same thing over and over again.
One of the things we learned is
that spectating the game turned
out to be just as fun as playing
it.
Sitting there on the sidelines
and watching like it is a
sporting match going side to
side, that is just a really
enjoyable experience.
It's something to think about.
Games are a key form of social
and personal interaction.
Augmented reality can offer a
kind of personal touch that you
might have had before playing
like a traditional card game
around the table with older
relatives.
But now you have technology to
help along the way.
It isn't enough to just take a
3D game and put it on a table in
front of you.
With augmented reality, you know
how the device is positioned.
You also know a little bit about
the user's environment and you
should try to take advantage of
that in the game and make
experiences that are really for
augmented reality first.
Your device can be used as a
camera to look inward at an
object of focus.
In this case, this is a 3D
puzzle game where we're looking
to repair a broken vase.
We can look all around it,
figure out what piece goes
where, and do our best on the
repairs.
In SwiftShot, we took a similar
concept.
The focus is the table you're
playing on and you can walk
around it.
But the table isn't just a
tracking surface for augmented
reality.
It's an integral part of the
gameplay.
The height of the table is
actually significant and as a
result, you'll see in the game
that there are slingshots at
different heights on tops of
stacks of blocks in order to
give you better shots or take
advantage of the player dodging
and weaving a little bit.
Another possible principle is
your device is a camera you use
to look around you.
In this case, we're looking for
unicorns hiding out in the
wilderness and we're taking
pictures of them.
It's just around you, not
inward.
The device can also be a portal
into an alternate universe.
You don't need to see what the
camera sees directly.
The environment can be entirely
replaced.
Laws of physics can be bent or
completely changed.
Whatever you need to do to make
it fun.
In this case, we're able to see
the stars, even though it's
bright daylight.
Also, your device can be the
controller itself.
You're able to fuse yourself
with the virtual world using the
device as the controller.
In this example, we're sort of
magnetically levitating blocks
and placing them in the sorting
cube.
That's the focus of the
interaction in SwiftShot.
You want to encourage slow
movement of the device.
That gives the best images to
the camera without motion blur
and it can do the best job at
tracking.
And despite how thin and light
these devices are, waving them
all around at arm's length turns
out to be a little bit tiring.
So, you're looking for slow and
deliberate movements.
You want to encourage the player
to move around the play field In
this case, our shot of the enemy
slingshot is blocked by those
blocks.
So, we have to move over to
another slingshot to clear the
obstruction.
Control feedback is important
for immersion.
In SwiftShot, we give feedback
using both audio and haptics.
There's a variety of dynamic
behavior in the stretching band
sound and haptics on the phones
to give you that feel that
you're doing it.
We'll talk a lot more later
about the dynamic audio.
So, next I'd like to bring up
David Paschich, who will go deep
into the details of SwiftShot.
Thank you.
David?
>> Thank you Alex, and hello,
everybody.
I just want to echo what Alex
said.
The response that we've seen
from people here at the show to
SwiftShot has been really
amazing and it's been gratifying
to see some people already
downloading it, building it and
altering it from the sample
code.
So, I thank you for that.
We're really excited about that.
I want to talk by talking first
about the technologies we used
in building SwiftShot.
The first and foremost is ARKit,
which lets us render the content
into the physical world around
the players, immersing them in
the experience.
We use SceneKit to manage and
draw that content, using
advanced 3D rendering and
realistic physics for fun
gameplay.
Metal lets us harness the power
of the GPU devices.
It came into play both within
SceneKit for the shading and
rendering and also for the flag
simulation, which I'll talk
about a little later on.
GameplayKit provides an entity
component architecture for our
game object.
It let us easily share behaviors
between objects in the game.
Multi-peer connectivity provides
the networking layer, including
discovery of nearby devices and
synchronization, and encryption
as well.
AV Foundation controls both the
music for atmosphere and the
sound effects for the devices,
really giving you that immersive
experience.
And lastly, we built the entire
application in Swift.
Swift's type safety, performance
and advanced features like
protocol extensions let us focus
more on the gameplay and worry
less about crashes and
mismatched interfaces between
code layers.
Those are the iOS technologies
we use.
I'll talk about how we use those
as we implemented several of the
features of the game.
Establishing the shared
coordinate space.
Networking.
Physics. Asset important and
management.
Flag simulation.
And the dynamic audio.
We'll start by talking about
setting up a shared coordinate
space.
The key in the experience is
having the player see the same
object in the same places on
both devices.
To do that, we have to have a
shared coordinate space,
allowing them to talk about
locations in the world in the
same way.
ARKit provides a number of
features you can use to set this
up.
In iOS 11.3, we introduced image
recognition, allowing your apps
to recognize images in the world
around you.
Now in iOS 12, we're adding two
additional technologies - object
detection and world map sharing.
Both image detection and object
detection let you add content to
things the user sees in the real
world but they require you to
have pre-recorded those objects
for later recognition.
You saw that in the keynote
during the Lego demo,
recognizing built models and
adding content.
For this game, we wanted to
enable users to play anywhere
with a table such as a café,
their kitchen and so forth.
WorldMap sharing is how we did
that.
You can also apply this
technique to applications
besides games, like a fixed
installation in a retail
environment or a museum.
In the game room downstairs, we
use iBeacons so devices know
which table they're next to and
can load the correct WorldMap
for that area.
That really makes the experience
magical.
One of the features of SwiftShot
you may have used if you built
your app yourself is the ability
to, ability for players to place
the game board in the virtual
world.
At the tables downstairs, we're
using preloaded maps.
But here's an example of
building your own board and
placing it in the virtual world.
This is how that works.
As you saw in the video, you
start by scanning the surface,
letting ARKit build up a map of
the area.
You can then serialize that map
out as data and transfer it to
another device.
The target device then loads the
map into ARKit and uses it to
recognize the same surface.
At that point, we now have a
shared reference point in the
real world, and both devices can
render the game board into the
same place in that world.
The first step in the
implementation is getting the
World Map from the ARSession on
the first device.
That's the call to a new API in
iOS 12 in ARSession,
getCurrentWorldMap.
It builds an ARWorldMap object
from the session's current
understanding of the world
around you and then returns it
in an asynchronous callback.
We then use NSKeyedArchiver to
serialize that out to a data
object.
You can then save the data or
send it over the network.
Once you have that data object,
you next have to decide how to
get it from one device to
another.
For ad hoc gaming like you saw
in the videoing, SwiftShot uses
a peer-to-peer network
connection which we'll get into
more detail on shortly.
When the second device joins the
network session, the first
device serialized the WorldMap
and sent it over the network.
This is great for casual gaming
situations, allowing users to
set up anywhere they can find a
surface to play on.
For the gaming tables
downstairs, we used a different
approach.
We spent some time during setup
for the conference recording
WorldMaps for each of the
tables, ensuring that we could
localize that shared coordinate
space from multiple angles.
Each table has its own unique
characteristics as well as
slightly different lighting and
positioning.
We then saved the files to local
sstorage on each device.
Since the devices in use are
managed by our conference team,
we're able to use mobile device
management to make sure that the
same files are present on every
device in the game.
To make the solution even more
seamless, you can use iBeacons
on each table.
By correlating the identifier of
the iBeacon with particular
WorldMaps, each instance of the
SwiftShot application can load
the correct WorldMap
automatically.
Now, if you're building a
consumer application, you can
also use things like iOS's
on-demand resources or your own
cloud-sharing solution to share
WorldMaps between devices.
This would allow you to for
instance select the correct
WorldMap for a particular retail
location somewhere out in the
world.
There's really a lot
possibilities here to tailor
users' experience and really
build something great.
So, those are a couple of the
ways to get that WorldMap data
from one device to another.
Let's talk about how you then
load it on the second device.
In this case, we use
NSKeyUnarchiveder to blow up
that WorldMap again from the
data that we received.
We then build an ARWorldTracking
configuration and add the
WorldMap to that configuration
object, setting up the way we
want.
And then lastly, we ask the
ARSession to run that
configuration, resetting any
existing anchors and tracking.
ARKit on the target device then
starts scanning the world around
you, correlating those feature
points from the original map
with those that it sees there.
Once it's able to do that,
you've got that shared
coordinate space.
Both devices have 000 in the
same place in the real world.
So, a quick word about privacy
with WorldMaps.
In the process of recording the
WorldMap, we take into account
features of the world around
you, physical arrangements of
objects and so forth.
While it does include geographic
information like latitude and
longitude and thus your
application doesn't need to ask
for location permission to use
ARKit, it may include personally
identifiable information about
the user's environment.
So, we recommend that you treat
a serialized WorldMap the same
way that you would any other
user-created private data.
This means that you want to make
sure that you're encrypting it
both at rest and when moving
across the network.
You may also want to let your
users know if you're planning to
save that WorldMap information
for an extended period of time,
past a single session of your
application.
In SwiftShot, we're able to take
advantage of iOS's built-in
encryption for encrypting the
data while at rest.
I'll talk next about how we did
the networking for encryption,
on the networking.
Now, in addition to setting up
shared coordinate space for
SwiftShot, we needed to tell the
other device where the user has
chosen to locate the board.
We use an ARAnchor to do this.
When you create an ARAnchor, you
provide a name as well as
position and rotation
information as a 4 x 4
transform.
ARKit can then include the
Anchor in the ARWorldMap we
generate and serialize out, and
then, so we can transfer that
board information to the other
device.
Now, the system ARAnchor class
just has the name and the
orientation we created.
We can look up the anchor that
we're interested in by name on
the other side.
For our application though, we
need to include some additional
information for the other
device, and that's the size that
the user chose for that board,
deciding whether they're playing
on a, you know, a small table
top and surface, or they want to
blow the board up to be the size
of a basketball court.
We thought about, you know,
adding that to our network
protocol alongside the WorldMap,
but then we came up with a
better solution.
We created a custom subclass of
ARAnchor that we called board
anchor and added that
information to that class, the
size of the board.
We then made sure that we
implemented the NSCoding
required classes or override
them to include that information
when the object is serialized
out.
Now, the information is included
directly within the WorldMap
when we transfer it over to the
other device.
It makes it very easy and
straightforward.
One thing to keep in mind, and
this bit us for a little bit.
When you use Swift to make a
subclass like this, when you
serialize it out, the name of
the module or the name of your
application is included in the
class name.
This is something to be aware of
if you're planning to move
WorldMaps between different
applications.
NSKeyedArchiver can help you
accommodate that.
So, that's WorldMap sharing.
It's a new feature in iOS 12.
We're really looking forward to
seeing what everyone can build
with that.
Next, let's talk about the
networking we built into the
game.
We used iOS's multi-peer
connectivity API which has been
in the system since iOS 7 in
order to do this.
Multi-peer connectivity.
Allows us to set up a
peer-to-peer session on a local
network, allowing devices in the
session to communicate without
going through a dedicated
server.
Now, in our application, we
designate one of the devices as
the server but that's something
that we did for our application.
It's not inherent in the
protocol.
Encryption and authentication
are built into multi-peer
connectivity.
In our case, we didn't use
authentication because we wanted
a very quick in-and-out
experience but we did use
encryption.
We found that turning on
encryption really provided no
performance penalty, so there's
either in network data size or
computation.
So there's really no reason not
to use it.
Multi-peer connectivity also
provides APIs for advertisements
and discovery.
We use this to broadcast
available games and allow
players to select a game to
join.
So, here's how we get that
session set up.
First, on one device, the user
decides to set themselves up as
hosts for the application.
They scan the world, place the
gameboard within that world, and
then the device starts a new
session, a multi-peer
connectivity session, and starts
advertising it to other devices
on the local network.
A user on the other device sees
a list of available games.
When he selects one, his device
sends a request to join the
existing session.
Once the first device accepts
the request, multi-peer
connectivity sets up a true
peer-to-peer network.
Any device in the network can
send a message to any other
device in the network.
In SwiftShot, we designate the
device that started the session
as the source of truth for the
game state.
But again, that's the decision
we layered on top of the
networking protocol; it's not
inherent in multi-peer
connectivity.
Once the session is set up,
multi-peer connectivity lets us
send data between peers in three
ways.
As data packets.
As resources, file URLs on the
local storage.
And as streams.
Data objects can be sent,
broadcast to all peers in the
network whereas resources and
streams are device to device.
In SwiftShot, we use the data
packets primarily as a way to
share game events and also the
physics state.
We'll talk about that later on.
And then we used the resources
to transfer the WorldMap.
It ended up we didn't need
streams for our application.
Under the covers, multi-peer
connectivity relies on UDP for
the transfer between devices.
This gives a low latency for,
great for applications like
games.
Now, UDP inherently doesn't
guarantee delivery, so
multi-peer connectivity lets you
make that decision and specify
whether a particular data packet
is to be sent reliably or
unreliably.
If you choose reliably,
multi-peer connectivity takes
care of the retries for you, so
you don't have to worry about
that in your code.
Even when you're broadcasting to
all members of the session.
Now that we have a networking
layer, we need to build our
application protocol on top of
it.
SwiftEnums with associated types
make this very easy.
Each case has a data structure
around it, ensuring type safety
as information moves around the
system.
Some of those can be further
enums.
So, for instance, in this
example, gameAction includes
things like a player grabbed a
catapult.
A projectile launched, and so
forth.
The PhysicsSyncData is a strut
and we'll talk more about how we
encoded that later on.
Again, Swift makes this very
easy.
For struts, if all the members
of the struct are codable, then
all you need to do is mark that
struct as codable and the Swift
compiler takes care of the rest,
building all the infrastructure
needed for the serialization.
Swift doesn't do that for enums
and so we ended up implementing
that ourselves, implementing the
init and then coding method from
the codable protocol to make
that work.
Serialization then is very easy.
Just build a property listing
coder and have it encode the
object out for you.
We can then send a data packet
within the multi-peer
connectivity session.
Now, a reasonable question here
might be how's this going to do
in size and performance?
Property-- binary property lists
are pretty compact and the
encoder's pretty fast.
But sometimes, you know, the
soft implementation in many ways
is optimized for developer time,
which is sometimes your most
precious resource on a project.
Now, we ran up against some of
those limitations as we started
to build the next feature, and
we'll talk about how we overcame
this.
So, let's talk next about the
physics simulation in the game.
For a game like SwiftShot,
physics is really key to create
a fun interaction that comes
from the realistic interaction
between objects and the game.
It's a really great experience
to take that shot and bounce it
off an object in a game and take
out the opponent's slingshot.
And that really comes from the
physics simulation.
We use SceneKit's built-in
physics engine.
It's integrated with the
rendering engine, updating
positions of the object and
scene automatically, and
informing us of collisions using
delegation.
In our implementation, we
decided that the best approach
was for one device in the
session to act as a source of
truth or server.
It sends periodic updates about
the physics state to the other
devices in the network using
that multi-peer connectivity
broadcast method.
Now, the other devices also have
the physics simulation on.
That's because we don't send
information about every object
in the game, only those objects
that are relevant to the
gameplay such as the box,
projectile and catapult.
Things like simulating the
swinging of the rope and the
sling, particles and so forth,
those are just done locally on
each device since it's not
critical to the game that they
be in the same place on every
device.
Now, one of the things that we
discovered was when we were
doing this was that the physics
engine responded very
differently depending on the
scale of the objects.
And so the physics simulation
thinks the objects are about 10
times the size as you would see
them in the real world.
We found that gave the best
gameplay experience and the best
performance.
We had to tweak some of the laws
of physics to make that look
right but, you know, when you're
building a game, if it looks
right and feels right and it's
fun, then it is right.
Now, to share that physics state
and make sure everything looked
right, we need to share four
pieces of information.
The position.
The velocity.
The angular velocity.
And the orientation.
That's a lot of information
about every object in the game,
so it was vital that we minimize
the number of bits actually
used.
I'll walk you through that using
position as an example.
SceneKit represents position as
a vector of three floating point
values.
This is the native format and
gives the best performance for
calculations at run time.
However, there are really more
bits than necessary to specify
the object's location.
A 30-bit float has 8-bits of
exponent and 23 bits of
mantissa.
For a range of plus or minus 10
to the 38th meters.
It's way more than we need for
this game.
So, because the physics
simulation thinks our table is
28 meters long, we said you
know, 80 meters is going to give
us plenty of buffer space around
that on either side.
When we're coding that then,
we're able to eliminate the sign
bit by normalizing that between
0 and 80 meters, even though our
origin is at the center of the
table.
Now all values are positive.
We then scale that value to be
in a range of 0 to 1.
That way we don't need the
exponent information that's
inherent in the protocol.
And then lastly, we take that
and we scale it to the number of
bits available so that all 1s is
a floating point 1 and all 0s is
the floating point 0.
This gave us millimeter scale
precision which, as we
discovered, was really enough to
achieve that smooth synchronous
appearance in the game.
Now, we did a similar technique
for all the other values that
you saw.
The velocity, angular velocity
and orientation.
Tailing the ranges and the
number of bits for each to
really make sure that we
transmit the information using
the minimal amount of data.
Overall, we reduce the number of
bits for each object by more
than half.
Now, even though we've
compressed the numbers, property
lists still have a fair amount
of overhead for the metadata
around it, sending each field by
name.
We said there's no reason for
that.
We all know what these objects
are.
That's not information we need.
So, to do this, we implemented a
new serialization strategy which
we call a BitStream.
BitStreams are designed to pack
the information into as few
bytes as possible by providing
fast serialization and
deserialization.
Now, our implementation is
purpose-built for communicating
binary data with low latency in
an application like this.
Strategies like this wouldn't
work well for data that needs to
persist or data that, where you
need to keep track of the schema
and watch it changing over time.
But for an ephemeral application
like this, it was just the
thing.
To help implement this, we
created two protocols, BitStream
Encodable and BitStream
Decodable.
Combine those and you get
BitStream Codable.
Then we took that and marked all
the objects that we needed to
serialize, using that protocol,
helping us to get the
implementation.
That includes both our own data
objects and the object we use
from the system such as the simD
floating point vector type.
So, here's the implementation of
compressing floats.
The compressors, configured with
the minimum and maximum range,
and the number of bits we wanted
to use.
It clamps the value to the range
and then converts it to an
integer value for encoding using
the specified number of bits.
Each component for each object
in the scene is compressed in
this way.
We also use an additional bit at
the front to tell if an object
has moved since the last update.
If it hasn't moved, we don't
resend that information.
So, let's go back to our action
enum, with the three different
actions to talk about how we
apply BitStream to do this.
For regular codable, if you're
doing your own serialization,
you specify encoding keys for
enums for the different cases in
the enum.
For BitStream, we used integer
values for this rather than
string values.
And then in our encoding method,
we're able to then append that
value first followed by the data
structure associated with that
case of the enum.
Now, if you look at this code
though, there's kind of a pit
fall here.
We know that this one has, this
case has three different cases.
And so we only need two bits to
encode it.
But what happens when we add
another case, 4 bits with 4
cases, we'll still find.
We add that fifth case and now
we need to go through and change
that so that every time we do
this, we're using three bits
instead of two.
Now, that's kind of tedious.
This code's a little bit
repetitive and, you know,
there's stuff that could go
wrong there.
We really, if we don't remember
this, we're just going to end up
in a bad place.
So, we took a look at this and
figured out that there was a way
that Swift can help us do this.
So, we used a new feature in
Swift 4.2, which is case
iterable.
We added that protocol
compliance to our enum type.
When you do that, Swift adds a
new static member of the type
called all cases, containing
each of the cases in the enum.
That lets us automatically get a
count of the number of cases.
We then added another extension,
this time on the raw
representable type which all
enums with number types like
that conform to.
Where it's case iterable and
where that number is affixed
with integer.
And to this, we get to
automatically take those number
of cases and figure out how many
bits it takes to represent all
those cases on the wire.
Lastly, we added a generic
method on the writable BitStream
type allowing us to encode that
enum.
It appends things of that type
and it uses that new static
property to figure out the
number of bits that are needed
to use.
Now, our encode method is much
simpler.
We just used append enum on the
proper coding key for each and
Swift takes care of the rest.
When we add more cases to the
enum, the BitField expands
automatically.
If we remove cases, it contracts
automatically.
We don't have to worry about it.
So, how much faster and more
compact is BitStreamCodable?
We ran some tests using XE test
support for performance testing
using a representative message
in which we send information
about object movement.
The results were pretty
impressive - 1/10 the size,
twice as fast to code, 10 times
as fast to decode.
Now when we talk about going
from 75 microseconds down to 6
microseconds, that seems like
small potatoes.
But there's around 200 objects
in the game and we want to do
this very frequently to make
sure the game remains smooth for
all participants.
By using this encoding format,
we were able to do those physics
updates at 60 fps, ensuring that
you get a smooth experience for
everyone in the game.
Now, I've talked about this.
We did some things with codable
and some things with BitStream
Codable that, you could have a
problem there because we're
encoding things two different
ways.
And that means now we need to
have two different code paths
through our application.
Swift helps us out again and
lets us figure out how to
combine them.
We then added constrained
extensions so that anything that
is codable in BitStream Codable,
we provide default
implementation of the BitStream
encoding.
And then we just go ahead and
use a binary [inaudible] encoder
to encode the data and stuff it
into BitStream.
And then anything, any struct
that is codable, we just add
that by marking it BitStream
Codable.
Now, this implementation then is
not as fast and compact as if we
went forward and made everything
BitStream Codable directly.
But we discovered we didn't need
to do that for every object in
the game, only the most frequent
messages.
This let us really move quickly
and keep better rna on the game.
So, that's how we did the
physics.
Next I want to talk about how we
dealt with the assets on the
game levels and this is the
question that a lot of people
asked us downstairs.
You know, the assets include the
3D modules, the textures, the
animations and so forth.
So, we have some text angle
artists here in Apple and they
used some commercial tools to
build the visuals for the games.
The blocks, the catapults and so
forth.
They then exported those assets
in the common DAE file format.
We're looking forward to the
commercial tools supporting USDZ
but for this game they weren't
quite there yet.
We then built a command line
tool in Swift that converts the
object from DAE into SceneKit
files using the SceneKit API.
Because SceneKit provides the
same APIs on both iOS and macOS,
we're able to run this tool as
part of our build process on
macOS and include the SceneKit
files directly in our iOS build
in the application.
We structured the data so that
each individual type of block is
its own file and then for each
levels, we combine those blocks
together.
This let us iterate on the
appearance and physics behavior
of each individual block and
then pull them all together for
those levels and iterate on
gameplay design.
Try out some of the different
levels that you'll see if you
look in the source code to the
application.
To optimize, further optimize
for different distances,
SceneKit supports varying the
assets used based on the level
of detail required.
Nearby objects use more polygons
and more detailed textures while
far away objects use fewer
polygons and less detailed
textures.
This really optimizes the
rendering of the scene.
However, we still want the
gameplay to stay consistent.
And so we specified the physics
body separately.
SceneKit provides a number of
built-in physics body types such
as cube, sphere, cylinder.
And if you use those, you really
get the best performance.
If you don't specify one,
SceneKit will build a convex
hull automatically for you and
that works.
But it is a lower, can be a
lower performance implementation
by adding these objects where
they were available and where
they made sense, we really sped
up the performance of the game.
So, here's some examples of the
physics finished product.
First one is one of the blocks
from the game.
In this case, a cylinder with
textures for a great wood grain
look.
Next is the slingshot with the
sling head idle.
We add the [inaudible] colors at
RunTime using shaders and built
some custom animation for the
sling's motion during gameplay.
Lastly, we included some extra
assets that didn't get included
in the gameplay.
Even though we had to sacrifice
them, we want you to have them
and use them in your own sample
code.
So, one of the other fun things
we included is this flag
animation.
It really improves the immersion
in the game environment.
We wanted a realistic wind
effect on this.
Now, we could've used a cloth
simulation out of the physics
engine.
But instead, we decided to use
the GPU and do it with Metal.
We started with a SceneKit asset
built by our technical artist.
To get the Apple logo on the
flag, we applied a texture at
RunTime.
Then we built a Swift class
around the Metal device.
Swift code builds a metal
command queue and inserts
information from the state of
the game, such as the direction
the wind is blowing.
That command queue is running a
custom Metal compute shader.
That comes from a legacy code
built in C.
But because Metal is based on
modern C++, it was a very easy
conversion to make.
We then also run another compute
shader to compute normal for the
surface, so we can get a great,
smooth flag look without a huge
number of polygons in the scene.
And it really makes the flag
look amazing.
Each frame, the shader updates
the geometry of the match to its
new position.
By taking advantage of the GPU
in this way, we get a great
effect without it impacting the
main CPU.
So, lastly I'd like to talk
about the audio implementation
in SwiftShot.
Audio can make any game even
more immersive and engaging.
We knew we wanted to provide
realistic sound effects
positioned properly in the world
for that really immersive
experience.
And giving the user great
feedback on how they're
interacting with that world.
We also wanted to make sure it
was fast and pay attention to
how much adding the audio would
add to the size of our app.
So, we came up with what we
think is a great solution.
We created a few representative
sound samples using some toys we
borrowed from children of people
on the team.
We then recorded those and used
those to combine them into an AU
preset file and use those to
build a custom Midi instrument
in AV Foundation using AV Audio
Unit Midi Instrument.
That made it easy to quickly
play the right sound at the
right time in response to user
inputs and collisions in the
game.
We didn't just play the sounds
as is.
To give good feedback to the
user, we pull back on the
slingshot.
We vary the sound in a couple of
ways.
We change the pitch based on how
far back they've pulled the
slingshot.
And we vary the volume based on
the speed as you pull back.
And we do that at RunTime by
selecting the right Midi note
and then using some additional
Midi commands to alter that
sound before we play it.
So, let's take a listen and this
is, we'll play it.
[ Sound effects ]
Now, we also wanted to make sure
that when you're using the
slingshot, we also give users
some audio feedback as to
whether or not they're within
range of the slingshot and
whether or not they've grabbed
that.
And those are the little beeps
you heard at the start.
Because those are UI feedback
for the users, those sounds only
come out of the device that the
user is using to interact with
the slingshot.
However, we also want everybody
else in the game to know what's
going on with the slingshot,
whether someone else is pulling
something or something like
that.
But we want one of those to be
quieter.
So, we use positional audio so
that if my opponent across the
table is pulling their
slingshot, I still hear that
sound from my device but it's
quieter and positioned correctly
in the world.
For colliding blocks, we took a
similar approach but slightly
different.
We really wanted a cacophonous
effect.
And the blocks are generally not
near any one player so again,
using the positional support
from SceneKit really made this
sound great.
Each device makes sounds
separately without worrying
about synchronizing across
devices because we want it to be
cacophonous, blocks smashing
about.
Again, we use a custom Midi
instrument to take a small
number of sounds and vary them.
In this case, varying the attack
rate based on the strength of
the collision impulse coming
from the SceneKit physics
engine.
These sounds again are localized
in 3D coordinates based on the
device's position in the scene.
So, collisions in the far end of
the table are quieter than those
at your end.
Let's take a listen to this.
[ Sound effects ]
One more shot.
There we go.
Right.
So we wanted to share one more
little trick that we discovered
as we were working on this.
In the process of setting up the
sounds, we discovered that we
needed to have a script run at
RunTime to do some file name
path conversions on the property
list for the DAU preset.
We found that we're able to
build that tool using Swift but
set it up as a command line
tool.
Do you notice at the top of
this, the traditional Unix
shebang-style statement at the
top of the script.
That tells your shell to fire up
Swift to run this.
By doing this, we can then treat
Swift as a scripting language.
You can develop one of these by
using a Swift playground to work
with your code interactively and
make sure that you've gotten it
right.
Once it's ready, just save it
out to a file, add the shebang
line to the top and make the
file executable in the file
system.
Now you've got a command line
tool that you can use either,
you know, outside the
application or in Xcode using a
RunScript phase.
It's very easy and it really
gives you access to all the
system frameworks.
In this case, we're able to edit
the P list directly.
It's a really great technique
and we hope that you'll be able
to take advantage of it.
So, today I hope you've seen how
AR provides really new
opportunities for engaging games
and other experiences.
We encourage you to design with
AR in mind from the start.
And remember that for games, the
play is the thing.
You can't sprinkle fun on top at
the end.
We really hope that you'll
download the SwiftShot available
as sample code and use it to
guide you as you build your own
apps and we're planning to
update that with each subsequent
seed of iOS 12 as we go to the
release.
And finally, if you haven't had
a chance yet, we hope you'll
play SwiftShot with us
downstairs in the game room.
For more information, there's an
ARKit lab immediately after this
session and the get together
this evening.
I'm also happy to announce that
for those of you here at the
conference, we're going to have
a SwiftShot tournament this
Friday from noon to 2, so we
hope you'll join us for that.
Thank you very much.
[ Applause ]