Transcript
[ Music ]
[ Applause ]
>> Thanks, Mike.
Hey everyone.
How many of you're interested in
AR?
All right.
Cool. Now how many of you have
an AR app in the App Store
already?
Okay. A few less.
I get it. AR can be pretty
intimidating.
Well, today we're going to look
at a few techniques that we use
which will hopefully help you in
two ways.
First, they're going to
hopefully help make AR more
approachable and second, we're
going to show you how you can
save time and money by making
sure you build the right things
before you write a single line
of code.
So let's get started.
If you've seen previous talks by
my team here at WWDC, you may
already be familiar with our
revolutionary app Toast Modern,
the best app for finding and
rating the toast around you.
I know it's amazing.
Well, we've been building up our
database of toast, and now we've
been tasked with adding AR to
this app.
I'm sure you're all aware that
the artisanal toast scene in the
Bay Area it's pretty big right
now.
Well, today it's going to get a
whole lot bigger.
Introducing ToastAR, the world's
first AR toast recognition and
rating app.
I know. Thank you.
Thank you.
Through the use of computer
vision and AR, ToastAR lets you
check the ratings on a piece of
toast and get this, where it's
from.
Revolutionary I know.
I mean, you're all developers
for Apple platforms so you
expect the best APIs and the
best hardware.
Why not also expect only the
best toast too.
So how did we prototype this
beautiful, revolutionary app?
Well, you may think we used a
programming language like Swift
or a framework like ARKit or
maybe even a sophisticated
application such as Unity.
These are all valid ways to
prototype, but they're not the
only solution.
Sometimes you can use low-tech
physical prototyping techniques
to work through your design
problems before you write a
single line of code.
So today I'm going to show you
how you can use just the camera
app and the world around you to
prototype for AR just like we
did for ToastAR.
So before we get started,
there's two terms you should be
familiar with.
First world space.
When we talk about world space,
we're talking about design
elements in the world around you
and your device.
And when we talk about screen
space, we're talking about these
elements on the screen of your
device.
So let's take a look at the
original design we got for
ToastAR and work through our
process.
In this design, we have a label
in world space which shows us
where the toast is from and its
current rating.
Now, you may want to jump in and
start coding this immediately.
It looks pretty simple, but
instead we're going to save time
and money by making sure we are
building only the right things
before we write a single line of
code.
AR is about the world around
you.
So testing out your designs is
difficult through a single still
or even video.
Physical prototyping, however,
allows us to do this in an
approachable, low-tech and
highly-iterative manner.
So how did we get started here?
Well, we printed out our labels
and stuck them to some
toothpicks.
I know. I know.
It may seem odd, but this
low-tech process is going to
hopefully help us in three ways.
First, we're going to be testing
our designs in context.
Remember, AR is about the world
around you.
Second, we're going to hopefully
figure out any issues with our
design before we write a single
line of code.
And third, through this process,
hopefully we'll get new ideas
that will help make ToastAR even
better.
But who even has a printer these
days, right?
You can even just draw something
and cut it out.
So now that we have our labels
attached to our toothpicks,
well, we stick them in a piece
of toast.
Then, just using the camera app,
we can test this design out in
world space.
I know. Revolutionary here,
right?
[ Applause ]
When we tried this, we learned
one thing really quickly.
These labels, at an angle, are
pretty difficult to read.
So the label is hard to read at
an angle.
Maybe we should have it always
face the camera.
So let's try this out.
So with some help from a
co-worker and some extremely
sophisticated hardware, okay,
it's a clear ruler but
beautiful.
We tested this out.
And we found that yes, the label
is easier to read when it's
always facing the camera.
Well, this behavior is commonly
referred to as billboarding.
And it seems like it'd be pretty
easy to implement in SceneKit
using an SCNBillboardConstraint.
But before we start coding this,
let's continue to test this
design and make sure we're only
building the right things.
So how does the label work when
there's a lot more text or when
it's at a distance?
Well, we can try this out pretty
easily too.
Again, with the camera app, we
can test this out and we find
that yes, it's a little
difficult to read when these
labels are at a distance.
Maybe we should bring these
labels on the screen.
So how do we prototype this?
Well, we're talking about screen
space, and we want to do this
quickly.
So we're going to go ahead and
use Keynote.
First, we're going to record a
video of our toast.
We'll record a slow pan.
And notice that we're going to
pause on the toast to give our
upcoming animation some time.
Next, we're going to bring this
into Keynote.
So you'll create a custom
document size which matches the
device you're eventually going
to put this on and then simply
import your video and bring it
into Keynote.
Third, we're going to animate
our labeling, import our design,
add a move action, give it an
appropriate duration, and move
it to the desired destination.
Now here's the important part.
We're going to check the build
order and make sure our video
and our animation begin at the
same time but we're giving the
animation a delay so it's on
screen when the toast is in the
field of view.
Finally, we're going to put this
on a device and test it out in
context.
So we'll put it on a device
using Keynote for iOS.
And we'll test this out in
context.
Great. Looks like we could fit
all the text we need on this
label.
And putting this on screen space
will hopefully make it easier to
expand the scope of this label
at a future date.
Now let's see how this scales,
though.
What happens when there's
multiple pieces of toast in the
field of view?
So using the same techniques,
we'll try this out too.
Now the problem here is when
there's multiple pieces of toast
in the field of view, I have no
idea which piece of toast I'm
currently interacting with.
Maybe I should try putting an
indicator in world space so that
I know which piece of toast I'm
currently interacting with.
Well, we can try this out too,
quickly.
So, again, with some help from
our co-worker and our trusty
ruler, we can try this out too.
Print out our design, attach it
to the ruler, and try this out.
It's the camera app world space.
This seems to help us.
We know which piece of toast
we're currently interacting
with.
So let's put it all together.
Using the techniques we just
reviewed, we can put together a
rough demo of our prototype.
And when we know we're building
the right things, we now know we
have some high-fidelity
prototypes to base our code off
of.
In just a few hours, we worked
through problems that could have
taken us days to code through.
So what did we learn?
Well, we learned these labels in
world space are difficult to
read at an angle and at a
distance, so we brought them on
screen.
But we also learned that we
needed indicators in world space
to let the user know what
they're currently interacting
with.
Physical prototyping allowed us
to test these AR experiences out
quickly.
We used the camera app to test
our world space interactions.
And we used Keynote to test out
our screen space interactions.
So how did prototyping help us?
Well, we tested our ideas, and
now we're going to be saving
time and money by making sure we
build the right things before we
write a single line of code.
We also got some new ideas along
the process.
These ideas helped make ToastAR
even better.
And take it from me, it's pretty
amazing.
Hopefully when you start working
on your next project, you now
have a few new tools when you
tackle your next AR app.
Thank you so much.
[ Applause ]