Transcript
[ Music ]
>> Hey, guys.
[ Applause ]
Thank you.
Thanks for coming by, guys.
Welcome to Designing Fluid
Interfaces.
My name is Chan.
And, I work on the human
interface team here at Apple.
And, most recently, I worked on
this, the fluid gestural
interface for iPhone 10.
So, me, Marcos, and Nathan, we
want to share a little bit about
what we learned working on this,
and other projects like this in
the past.
So, the question we ask
ourselves a lot is, what
actually makes an interface feel
fluid?
And, we've noticed that a lot of
people actually describe it
differently.
You know, sometimes, when people
actually try this stuff, when we
show them a demo, and they try
it, and they hold it in their
hands, they sometimes say it
feels fast.
Or, other people sometimes say
it feels smooth.
And, when it's feeling really
good, sometimes people even say
it feels natural, or magical.
But, when it comes down to it,
it really feels like, it's one
of those things where you just
know it when you feel it.
It just feels right.
And, you can have a gestural UI,
and we've seen lots of gestural
UI's out there, but if it's not
done right, something just feels
off about it.
And, it's oftentimes hard to put
your finger on why.
And, it's more than just about
frame rates, you know.
You can have something chugging
along at a nice 60 frames per
second, but it just feels off.
So, what gives us this feeling?
Well, we think it boils down to
when the tool feels like an
extension of your mind.
An extension of your mind.
So, why is this important?
Well, if you look at it, the
iPhone is a tool, right?
It's a hand tool for information
and communication.
And, it works by marrying our
tactile senses with our sense of
vision.
But, if you think about it, it's
actually part of a long line of
hand tools extending back
thousands of years.
The tool on the left here was
used to extract bone marrow
150,000 years ago, extending the
sharpness of what our fingers
could do.
So, we've been making hand tools
for some time now.
And, the most amazing thing is
that our hands have actually
evolved and adapted alongside
our tools.
We've evolved a huge
concentration of muscles,
nerves, blood vessels that can
perform the most delicate
gestures, and sense the lightest
of touches.
So, we're extremely adapted to
this tactile world we all live
in.
But, if you look at the history
of computers, we started in a
place where there was a lot of
layers of extraction between you
and the interface.
There was so much you had to
know just to operate it.
And, that made it out of reach
for a lot of people.
But, over the last few decades
or so, you've sort of been
stripping those layers back you
know, starting with indirect
manipulation, where things were
a little bit more one-to-one.
A little bit more direct all the
way to now, where we're finally
stripping away all those layers
back, to where you're directly
interacting with the content.
This to us is the magical
element.
It's when it stops feeling like
a computer, and starts feeling
more like an extension of the
natural world.
This means the interface is now
communicating with us at a much
more ancient level than
interfaces have ever done.
And, we have really high
standards for it.
You know, if the slightest thing
feels wrong, boom, the illusion
is just shattered.
But, when it feels right, it
feels like an extension of
yourself, an extension of your
physical body.
It's a tool that's in sync with
your thought.
It feels delightful to use, and
it feels really low-friction,
and even playful.
So, what gives us this feeling?
And, when it feels off, how do
we make it feel right?
That's what this presentation's
all about.
We're going to talk about four
things today.
And, we're going to start with
designing some principles,
talking about how we build
interfaces that feel like an
extension of us.
How to design motion that feels
in tune with the motion of our
own bodies, and the world around
us.
And, also designing gestures
that feel elegant and
intelligent.
We're also going to talk about
that, now that we've built this
kind of stuff, how do we build
interactions on top of it that
feel native to the medium of
touch, as a medium?
So, let's get started.
How do we design an interface
that actually extends our mind?
How do we do this?
Well, we think the way to do it,
is to align the interface to the
way we think and the way we
move.
So, the most important part of
that is that our minds are
constantly responding to changes
and stimulus and thought, you
know?
Our minds and bodies are
constantly in a state of dynamic
change.
So, it's not that our interfaces
should be fluid, it's that we're
fluid, and our interfaces need
to be able to respond to that.
So, that begins with response.
You know, our tools depend on
the latency.
Think about how hard it would be
to use any tool, or play an
instrument, or do anything in
the physical world, if there was
a delay to using it?
And, we found that people are
really, really sensitive to
latency.
You know? If you introduce any
amount of lag, things all of a
sudden just kind of fall off a
cliff in terms of how they
respond to you.
There's all this additional
mental burden.
It feels super disconnected.
It doesn't feel like an
extension of you anymore.
So, we work so hard to reduce
latency.
Where, we actually engineered
the latest iPhone to respond
quicker to your finger, so we
can detect all the nuances of
your gestures as instantly as
possible.
So, we really care about this
stuff, and we think you should
too.
And, that means look for delays
everywhere.
It's not just swipes.
It's taps, it's presses, it's
every interaction with the
object.
Everything needs to respond
instantly.
And, during the process of
designing this stuff, you know,
oftentimes the delays kind of
tend to seep in a little bit.
You know? So, it's really
important to keep an eye out for
delays.
Be vigilant and mindful of all
the latencies or timers that we
could introduce into the
interface so that it always
feels responsive.
So, that's the topic of
response.
It's really simple, but it makes
an interface feel lively and
dynamic.
Next, we want to allow for
constant redirection and
interruption.
This one's big.
So, our bodies and minds are
constantly in a state of
redirecting in response to
change in thought, like we
talked about.
So, if I was walking to the end
of this stage here, and I
realize I forgot something back
there, I could just turn back
immediately.
And, I wouldn't have to wait for
my body to reach the end before
I did that, right?
So, it's important for our
interfaces to be able to reflect
that ability of constantly
redirecting.
And, it makes it feel connected
to you.
That's why for iPhone 10, we
built a fully redirectable
interface.
So, what's that?
So, the iPhone 10's an
actually-- it's pretty simple
two-axis gesture.
You go horizontally between
apps.
And, you go vertically to go
home.
But you can also mix the two
axes, so you can be on your way
home, and peek at multitasking
and decide whether or not to go
there.
Or, you can go to multitasking
and decide, actually, no, I want
to go home.
So, this might not seem that
important, but what if we didn't
do this?
What if it wasn't redirectable?
So, what if the only gestures
you could do was this horizontal
gesture between apps, and then a
vertical gesture to go home, and
that's it.
You couldn't do any of that
in-between stuff I just
mentioned.
Well, what would happen is that
you would have to think before
what you did, before you
performed the gesture, you'd
have to think what you want to
do.
And so, the series of events
would be very linear, right?
So, you'd have to think, do I
want to go home?
Do I want to go to multitasking?
Then you make your decision,
then you perform the gesture,
and then you release.
But, the cool thing is when it's
redirectable, the thought and
gesture happen in parallel.
And, you sort of think it with
the gesture, and it turns out
this is way faster than thinking
before doing.
You know? Because it's a
multi-axis gestural space.
It's not separate gestures.
It's one gesture that does all
this stuff.
Home, multitasking, quick app
switching, so you don't have to
think about it as a separate
gesture.
And, helps with discovery.
Because you can discover a new
gesture along the path of an
existing gesture.
And, it allows you to layer
gestures at the speed of
thought.
So, what does that last one
mean?
So, let me show you some
examples.
And, we've slowed down the
physics on the simulation, so
you can actually see a little
bit what I'm talking about.
So, I can swipe to go home, and
then swipe to the next page, or
springboard while I'm going
home.
I can layer these two gestures
once I've internalized them.
Another example is that I can
launch an app and realize, oh,
actually I need to go to
multitasking, and I can
interrupt the app and go
straight to multitasking, while
the app is launching.
Or, I can launch an app and
realize, oh, that was the wrong
app.
And, I can shoot it back home,
while I'm launching it.
Now, there's one other one where
I can actually just launch an
app, and if I'm in a hurry, I
can start interacting with the
app as it's launching.
So, this stuff might not seem
really important, but we've
found it's super important for
the interface to be always
responding, always understanding
you.
It always feels alive.
And, that's really important for
your expectation and
understanding of the interface,
to be comfortable with it.
To realize that it's always
going to respond to you when you
need it.
And, that applies as well to
changes in motion, not just to
the start of an interaction, but
when you're in the middle of an
interaction, and you're
changing.
It's important for us to be
responsive to interruption as
well.
So, a good example is
multitasking on iPhone 10.
So, we have this pause gesture
where you slide your finger up
halfway up the screen, and
pause, and so we need to figure
out how to detect this change in
motion.
And so, how do we do this?
How do we detect this change in
motion?
Should we use a timer?
Should we wait until your finger
has come below a certain
velocity for a few amount of
time, and then [inaudible] bring
in the multitasking cards?
Well, it turns out that's too
slow.
People expect to be able to get
to multitasking instantly.
And, we need a way that can
respond as fast as them.
So, instead we look at your
finger's acceleration.
It turns out there's a huge
spike in the acceleration of
your finger when you pause.
And, actually the faster you
stop, the faster we can detect
it.
So, it's actually responding to
the change in motion, as fast as
we know how, instead of waiting
for some timer.
So, this is a good example of
responding to redirection as
fast as possible.
So, this is the concept of
interruption and redirection.
This stuff makes the interface
feel really, really connected to
you.
Next, we want to talk a little
bit about the architecture of
the interface.
How you lay it out,
conceptually.
And, we think when you're doing
that, it's important to maintain
spatial consistency throughout
movement.
What does that mean?
This kind of mimics the way our
object persistence memory works
in the real world.
So, things smoothly leave and
enter our perception in
symmetric paths.
So, if something disappears one
way, we expect it to emerge from
where it came?
Right? So, if I walked off this
stage this way, and then emerged
that way, you'd be pretty
amazed, right?
Because that's impossible.
So, we wanted to play into this
consistent sense of space that
we all have in the world.
And so, what that means is, if
something is going out of view
in your interface, and coming
back into view, it should do so
in symmetric paths.
It should have a consistent
offscreen path as it enters and
leaves.
A good example of this is
actually iOS navigation.
When I tap on an element in this
list here, it slides in from the
right.
When I tap the back button, it
goes back to the right.
It's a symmetric path.
Each element has a consistent
place where it lives at both
states.
This also reinforces the
gesture.
If I choose to slide it myself
to the right, because I know
that's where it lives, I can do
that.
It's expected.
So, what if we didn't do this.
Here's an example, where when I
tap on something, it slides in,
and then when I hit back it goes
down.
And, it feels disconnected and
confusing, right?
It feels like I'm sending it
somewhere.
In fact, if I wanted to
communicate that I was sending
it somewhere, this is how I
could do it, right?
So, that's the topic of spatial
consistency.
It helps the gesture feel
aligned with our spatial
understanding of the world.
Now, the next one is to hint in
the direction of the gesture.
You know, we humans are always,
kind of, predicting the next few
steps of our experience.
We're always using the, kind of,
trajectories of everything
that's happening in the world to
predict the next few steps of
emotion.
So, we think it's great when an
interface plays into that
prediction.
So, if you have two states here,
initial state and a final state.
The object-- and you have an
intermediate transition.
The object should transition
smoothly between these two
states in a way that it grows
from the initial state to the
final state, whether it's
through a gesture or an
animation.
So, good example is Control
Center actually.
We have these modules here in
Control Center, where as you
press they grow up and out
towards your finger in the
direction of the final state,
where it actually finally just
pops open.
So, that's hinting.
It makes the gestures feel
expected, and predictable.
Now, the next important
principle is to keep touch
interactions lightweight.
You know the lightness of
multitouch is one of the most
underrated aspects of it, I
think.
It enables the airy swipes and
scrolls, and all the taps and
stuff that we're all used to.
It's all super lightweight.
But, we also want to amplify
their motion.
You want to take a small input
and make a big output, to give
that satisfying feeling of
moving or throwing something and
having a magnified result.
So, how does this apply to our
interfaces?
Well, it starts with a short
interaction.
A short, lightweight
interaction.
And, we use all our sensors, all
our technology, to understand as
much about it.
To, sort of, generate a profile
of energy and momentum contained
within the gesture.
Using everything we know,
including position, velocity,
speed, force, everything we know
about it to generate a kind of,
inertial profile of this
gesture.
And then we take that, and
generate an amplified extension
of your movement.
It still feels like an extension
of you.
So, you get that satisfying
result with a light interaction.
So, a good example of this is
scrolling, actually.
Your finger's only onscreen for
a brief amount of time, but the
system is preserving all your
energy and momentum, and
gracefully transferring it into
the interface.
So, what if it didn't have this?
Those same swipes, well, they
wouldn't get you very far.
And, in order to scroll, you'd
have to do these long, laborious
swipes that would require a lot
more manual input.
It would be a huge pain to use.
Another good example of this is
swipe to go home.
The amount of time that your
finger's onscreen is very light.
And, it's-- ends up making it a
much more liquid and lightweight
gesture that still feels native
to the medium of multitouch.
While still being able to reuse
a lot of your muscle memory from
a button, because you move your
finger down on the screen, and
back up to the springboard.
And, it's not just swipes, it's
taps too.
It's important for an interface
to respond satisfyingly to every
interaction.
The interface is signaling to
you that it understood you.
It's so important for the
interface to feel alive and
connected to you.
So, that's the topic of
lightweightness and
amplification.
The next one is called
rubberbanding.
It means we're softly indicating
boundaries of the interface.
So, in this example, the
interface is gradually and
softly letting you know that
there's nothing there.
And, it's tracking you
throughout.
It's always letting you know
that it's understanding you.
What happens if you didn't do
that?
Well, it would feel like this.
It would feel super harsh and
disconcerting.
You kind of hit a wall there.
It would feel broken, right?
And, you actually wouldn't know
the difference between a frozen
phone, and phone that's just at
the top of the edge of the
screen, right?
So, it's really important that
it's always telling you that
you've reached the edge.
And, this applies to
transitions, too.
It's not just about when you hit
the edge, it's also when you
hand off from one thing to
another thing.
Tracking. So, a good example of
this is when you transition from
sliding up the dock to sliding
up the app.
It doesn't just hit a wall, and
one thing stops tracking, and
then the other thing takes over.
They both smoothly hand off in
smooth curves, so that you don't
feel like there's this harsh
moment where you hand off from
one thing to another.
Next one is to design smooth
frames of motion.
So, imagine I have a little
object here moving up and down.
It's very simple.
But, we all know this object is
not really moving, right?
We're all just having the
perception of it moving.
Because we're seeing a bunch of
frames on the screen all at
once, and it's giving us the
illusion of it moving.
So, if we took all of those
frames of motion, and kind of,
spread them out here.
And we see the ball's in motion
over time, the thing that we're
concerned about is right around
here, where there's too much
visual change between the
adjacent frames.
This is when the perception of
the interface becomes a little
choppy.
You get this visual strobing.
And, this is because the
difference between the two
frames is too much.
And, it strobes against your
vision, so.
Here's an example of where you
have two things both moving at
30 frames per second.
But the one on the left looks a
bit smoother than the one on the
right, because the one on the
right is moving so fast, that
it's strobing.
My perception of vision is, kind
of, breaking down.
I don't believe that it's moving
smoothly any more.
So, the important thing to take
away is that it's not just about
framerate.
It's what's in the frames.
So, we're kind of limited by the
framerate, and how fast we can
move and still preserve a smooth
motion.
So, this one's in 30 frames per
second.
If we move it up to 60 frames
per second, you can see that we
can actually go a little bit
faster, and still preserve
smooth motion.
We can do faster movement
without strobing.
And, there's addition tricks we
can do too, we can do things
like motion blur.
Motion blur basically bakes in
more information in each frame
about the movement, like the way
your eyes work, and the way a
camera works.
And you can also do-- take a
page from 2D animation and video
games by stretching, this-- this
technique called motion
stretching stretches the content
in each frame to provide this
elastic look as it moves with
velocity.
And so, in motion, it kind of
looks like this.
So, each of the different
techniques, kind of, tries to
encode more information visually
about what's going on in the
motion.
And, I want to focus a little
bit on this last one here,
motion stretching, because we do
this on iPhone 10, actually.
You know, when you launch an
app, the icon elastically
stretches down to become the app
as it opens.
And, it stretches up in the
opposite direction as you close
the app.
To give you that little bit
extra information between each
frame of motion to make it a
little bit smoother-looking.
Lastly, we want to work with
behavior rather than animation.
You know, things in the real
world are always in a state of
dynamic motion, and they're
always being influenced by you.
They don't really work like
animations in the animation
sense, right?
There's no animation curve
prescribed by real life.
So, we want to think about
animation and behavior more as a
conversation between you and the
object.
Not prescribed by the interface.
So, to move away from static
things transitioning into
animated things, instead think
about behavior.
So, Nathan's about to dive deep
into this one.
But, here's a quick example.
So, in Photos, there's less mass
on the photos, because it's
conceptually lighter.
But, then when you swipe apps,
there's more mass on the apps.
It's conceptually heavier, so we
give more mass to the system.
[ Applause ]
So, that's a little bit about
how to design interfaces that
think and work like us.
In-- it starts with response.
To make things feel connected to
you, and to accommodate the way
our minds are constantly in
motion.
To maintain spatial consistency,
to reinforce a consistent sense
of space, and symmetric
transitions within that space.
And, to hint in the direction of
the gesture.
To play into our prediction of
the future.
And, to maintain lightweight
interactions, but amplify their
output.
To get that satisfying response,
while still keeping the
interaction airy and
lightweight.
And, to have soft boundaries and
edges to the interface.
That interface is always
gracefully responding to you,
even when you hit an edge, or
transition from tracking one
thing to tracking the other.
And, to design smooth dynamic
behavior that works in concert
with you.
So, that's some principles for
how to approach building
interfaces that feel like an
extension of our minds.
So, let's dive in a little
deeper.
I'm going to turn it over to
Nathan de Vries, my colleague,
to design motion-- to talk about
designing motion in a way that
feels connected to motion, to
the motion of both you and the
natural world.
[ Applause ]
>> Thanks, Chan.
Hi everyone.
My name's Nathan, and I'm super
excited to be here today to talk
to you about designing with
dynamic motion.
So, as Chan mentioned, our minds
and our bodies are constantly in
a state of change.
The world around us is in a
state of change.
And, this introduces this
expectation that our interfaces
behave the same way, as they
become more tactile, it shifts
our expectations to be much
higher fidelity.
Now, one way we've used motion
in interfaces is through timed
animations.
A button is tapped on the
screen, and the reins are, kind
of, handed over to the designer.
And, their job is to craft these
perfect frames of animation
through time.
And, once that animation is
complete, the controls are
handed back to the person using
the interface, for them to
continue interacting.
So, you can kind of think of
animation and interaction as
being-- as moving linearly
through time in this, kind of,
call and response pattern.
In a fluid interface, the
dynamic nature of the person
using the interface kind of
shifts control over time away
from us as designers.
And, instead, our role is to
design how the motion behaves in
concert with an interaction.
And, we do this through these
continuous dynamic behaviors
that are always running, that
are always active.
So, it's these dynamic behaviors
that I'm going to, really focus
on today.
First of all, we're going to
talk about seamless motion.
And, it's this element of motion
that makes it feel like the
dynamic motion is an extension
of yourself.
Then, we're going to take a look
at character.
How, even without timing curves,
and timed animations, we can
introduce the concept of
playfulness, or character, or
texture to motion in your
interfaces.
And finally, we'll look at how
motion itself gives us some
clues about what people intend
to do with your interface.
How we can resolve some
uncertainty about what a gesture
is trying to do by really
looking at the motion of the
gesture.
So, to kick things off, let's
look at seamless motion.
What do I mean by seamless
motion?
So, let's look at an example
that I think we can all
familiarize with.
So, here we have a car, and it's
cruising along at a constant
speed.
And then, the brakes are
applied, slowing it down to a
complete stop.
Let's look at it again, but this
time we'll plot out the position
of the car over time.
So, at the very start of this
curve it's, kind of, straight,
and pointing up to the right.
And, this shows that the car's
position is moving at a constant
rate, it's kind of unchanging.
But then, you'll notice the
curve starts to bend, to
smoothly curve away from this
straight line.
And, this is the brakes being
applied.
The car is decelerating from
friction being introduced.
And, by the end of the curve,
the curve is completely flat,
horizontal, showing that the
position is now unchanging.
That the car is stopped.
So, this position curve is
visualizing essentially what we
call seamless motion.
The line is completely unbroken,
and there are no sudden changes
in direction.
So, it's smooth and it's
seamless.
Even when, actually, new dynamic
behaviors are being introduced
to the motion of the car, like a
brake, which is applying
friction to the car.
And, even when the car comes to
a complete stop, you'll notice
that the curve is completely
smooth.
There's this indiscernible
quality to it.
You can't tell when the car
precisely stopped.
So, why am I talking about cars?
This is a talk about fluid
interfaces, right?
So, we feel like the
characteristics of the physical
world make for great behaviors.
Everyone in this room finds the
car example so simple because we
have a shared understanding, or
a shared intuition for how an
object like a car moves through
the world.
And, this makes us a great
reference point.
Now, I don't mean that we need
to build perfect physical
simulations of cars that
literally drive our interface.
But, we can draw on the motion
of a car, of objects that we
throw or move around in the
physical world around us and use
them in our own dynamic
behaviors to make their motion
feel familiar, or relatable, or
even believable, which is the
most important thing.
Now, this idea of referencing
the physical world in dynamic
behaviors has been in the iPhone
since the very beginning with
scrolling.
A child can pick up an iPhone
and scroll to their favorite app
on the Home screen, just as
easily as they can push a toy
car across the floor.
So, what are some key, kind of,
characteristics of this
scrolling, dynamic behavior that
we have?
Well, firstly it's tapping into
that intuition, that shared
understanding that we all have
for objects moving around in the
world.
And, our influence on those
objects.
The motion of the content is
perfectly seamless, so while I'm
interacting with it, while I'm
dragging the content around, my
body is providing the fluidity
of the movement, because my body
is fluid.
But, as soon as I let go of the
content, it seamlessly coasts to
a stop.
So, we're kind of maintaining
the momentum of the effort being
put into the interface.
The amount of friction that's
being used for scrolling is
consistent, which makes it
predictable, and very easy to
master.
And finally, the content comes
to an imperceptible stop, kind
of like the car, not really
knowing precisely when it came
to a stop.
And, we feel that this distinct
lack of an ending kind of
reinforces this idea that the
content is always moving, and
always able to move, so while
content is scrolling, it makes
it feel like you can just put
your finger down again, and
continue scrolling.
You don't have to wait for
anything to finish.
So, there are innumerable
characteristics of the physical
world that would make for great
behaviors.
We don't have time to talk about
them all, but I'd like to focus
on this next one, because we
personally find it incredibly
indispensable in our own design
work.
So, materials like this
beautiful flower here, the
natural fibers of this flower
have this organic characteristic
called elasticity.
And, elasticity is this tendency
for a material to gracefully
return into a resting state once
stress or strain is removed.
Our own bodies are incredibly
elastic.
Now, we're capable of running
incredibly long distances, not
because of the strength of our
muscles, but because of their
ability to relax.
It's their elasticity that's
doing this.
So, our muscles contract and
relax once stress and strain is
removed.
And, this is how we conserve
energy.
Makes us feel natural and
organic.
The same elasticity is used in
iPhone 10.
Tap an icon on the Home screen,
and an elastic behavior is
pulling the app towards you.
Bring it exactly where you want
it to be.
And, when you swipe from the
bottom, the app is placed back
on the Home screen in its
perfect position.
We also use elasticity in
scrolling.
So, if I scroll too far and
rubberband, like Chan was
talking about, when you let go,
the content uses elasticity to
pull back within the boundaries,
helping you get into this
resting position, ready for the
next time you want to scroll.
So, let's dig in a little deeper
on how this elasticity works
behind the scenes.
You can think of the scrolling
content as a ball attached to a
spring.
On one end of the spring is the
current value.
This is where the content is on
the display.
And, the other end of the spring
is where the content wants to go
because of its elasticity.
So, you've got this spring
that's pulling the current value
towards the target.
Its behavior is influencing the
position of the content.
Now, the spring is essentially
pulling that current value
towards the target.
And, what's interesting about a
spring is, it does this
seamlessly.
This seamlessness is, kind of,
built in to the behavior.
And, this is what makes them
such versatile tools for doing
fluid interface design.
Is that you, kind of, get this
stuff for free.
It's baked in to the behavior
itself.
So, we love this behavior of a
value moving towards a target.
We can just tell the ball where
to go, and we'll get this
seamless motion of the ball
moving towards the target.
But, we want a little bit more
control over how fast it moves.
And, whether it overshoots.
So, how do we do that?
Well, we could give the ball a
little more mass, like make it
bigger, or make it heavier.
And, if we do that, then it
changes the inertia of the ball,
or its willingness to want to
start moving.
Or, maybe its unwillingness to
want to stop moving.
And, you end up with this little
overshoot that happens.
Another property that we could
change is the stiffness of the
spring, or the tensile strength
of the spring.
And, what this does, is it
affects the force that's being
applied to the ball, changing
how quickly it moves towards the
target.
And, finally, much like the car,
and the braking of a car, we can
change the damping, or the
friction, of the surface that
the ball is sitting on.
And, this will act as, kind of,
a brake that slows the ball down
over time, also affecting our
ability to overshoot.
So, the physical properties of a
ball and a spring are, kind of,
physics class material, right?
It's super useful in a
scientific context, but we've
found that in our own design
work they can be a little bit
overwhelming or unwieldy for
controlling the behavior of
objects on the screen.
So, we think our design tool
should have a bit of a human
interface to them.
That they need to reflect the
needs of the designer that's
using the tool.
And so, how do we go about that?
How do we simplify these
properties down to make it more
design friendly?
So, mass stiffness and damping
will remain behind the scenes,
they're the fundamental
properties of the spring system
that we're using.
But, we can simplify our
interface down to two simple
properties.
The first is damping, which
controls how much or little
overshoot there is from 100%
damping, where there will be no
overshoot to 0% damping where
the spring would oscillate
indefinitely.
The second property is response.
And, this controls how quickly
the value will try and get to
the target.
And, you might notice that I
haven't used the word duration.
We actually like to avoid using
duration when we're describing
elastic behaviors, because it
reinforces this concept of
constant dynamic change.
The spring is always moving, and
it's ready to move somewhere
else.
Now, the technical terms for
these two properties are damping
ratio and frequency response.
So, if you'd like to use these
for your own design work, you
can look up those terms, and
you'll find easy ways to convert
them.
So, we now have these two simple
properties for controlling
elastic behaviors.
But, there's still an infinite
number of possibilities that we
can have with these curves.
Like, there's just hundreds,
thousands, millions of different
ways we can configure those two
simple properties and get very
different behavior.
How do we use these to craft a
character in our app?
To control the feel of our app?
Well, first and foremost, we
need to remember that our
devices are tools.
And, we need to respect that
tools, when they're used with
purpose, require us to not be in
the way, not get in the way with
introducing unnecessary motion.
So, we think that you should
start simple.
A spring doesn't need to
overshoot.
You don't need to use springy
springs.
So, we recommend starting with
100% damping, or no overshoot
when you're tuning elastic
behaviors.
That way you'll get smooth,
graceful, and seamless motion
that doesn't distract from the
task at hand.
Like, just quickly shooting off
an email.
So, when is it appropriate to
use bounciness?
There's got to be a time when
that's appropriate, right?
Well, we feel if the gesture
that's driving the motion itself
has momentum, then you should
reward that momentum with a
little bit of overshoot.
Put another way, if a gesture
has momentum, and there isn't
any overshoot, it can often feel
broken or unsatisfying to have
the motion follow that gesture.
An example of where we use this
is in the Music app.
So, the Music app has a small
minibar representing Now Playing
at the bottom of the screen, and
you can tap the bar to show Now
Playing.
Because the tap doesn't have any
momentum in the direction of the
presentation of Now Playing, we
use 100% damping to make sure it
doesn't overshoot.
But, if you swipe to dismiss Now
Playing, there is momentum in
the direction of the dismissal,
and so we use 80% damping to
have a little bit of bounce and
squish, making the gesture a lot
more satisfying.
Bounciness can also be used as a
utility, as a functional means.
It can serve as a helpful hint
that there's something more
below the surface.
With iPhone 10, we introduced
two buttons to the cover sheet
for turning on the Flashlight,
and for launching the Camera.
To avoid accidentally turning on
the flashlight by mistake, we
require a more intentional
gesture to activate the
Flashlight.
But, if you don't know that
there's a more intentional
gesture needed to activate it,
when you tap on the button, it
responds with bounciness.
Has this kind of playful feel to
it.
And, that hint is teaching you
not only that the button is
working, but that it's
responding to you.
But, it's kind of teaching you
that if you press just a little
bit more firmly, it'll activate.
It's like teaching you.
It's hinting in the direction of
the motion.
So, bounciness can be used to
indicate this kind of thing.
Now, so far we've been talking
about using motion to move
things around, or to change
their scale, change their visual
representation on the screen.
But, we perceive motion in many
different ways.
Through changes in light and
color, or texture and feel.
Or even sound.
Many other sensations that we--
our senses can detect.
We feel this is an opportunity
to go even further, go beyond
motion, when you're tuning the
character of your app.
By combining dynamic behaviors
for motion with dynamic
behaviors for sound and haptics,
you can really fundamentally
change the way an interface
feels.
So, when you see, and you hear,
and you feel the result of the
gesture, it can transform what
was otherwise just a scrolling
behavior into something that
feels like a very tactile
interface.
Now, there's one final note I
want you thinking about when
you're crafting the character of
your app.
And, that's that it feels
cohesive, that you're staying in
character.
Now, what does this mean?
So, even within your app, or
across the whole system, it's
important that you treat
behaviors as a family of
behaviors.
So, in scrolling for example,
when I scroll down a page, using
a scrolling behavior, and then I
tap the status bar to scroll to
the stop of the page, using an
elastic behavior.
In both cases, the page itself
feels like it's moving in the
same way, that it has the same
behavior, even though two
different types of behaviors are
driving its motion, are
influencing its motion.
Now, this extends beyond a
single interaction like
scrolling.
It applies to your whole app.
If you have a playful app, then
you should embrace that
character, and make your whole
app feel the same way.
So, that people-- once they
learn one behavior of your app,
they can pick up another
behavior really easily, because
we learn through repetition.
And, what we learn bleeds over
into other behaviors.
So, next up, I'd like to talk a
little bit about aligning
motion, or dynamic motion, with
intent.
So, for a discrete interaction
like a button, it's pretty clear
what the intent of the gesture
is.
Right? You've got three distinct
visual representations on screen
here.
And, when I tap one of them, the
outcome is clear.
But, with a gesture like a
swipe, the intent is less
immediately clear.
You could say that the intent is
almost encoded in the motion of
the gesture, and so it's our
job, our role, to interpret what
the motion means to decide what
we should do with it.
Let's look at an example.
So, let's say I made a FaceTime
call, a one-on-one FaceTime
call, and in FaceTime, we have a
small video representation of
yourself in the corner of the
screen.
And, this is so I can see what
the person on the other end
sees.
We call this floating video the
PIP, short for picture in
picture.
Now, we give the PIP a floating
appearance to make it clear that
it can be moved.
And, it can be moved to any
corner of the screen, with just
a really lightweight flick.
So, if we compare that to the
Play, Pause, and Skip buttons,
like, what's the difference
here?
So, in this case, there's
actually four invisible regions
that we're dealing with.
No longer do we have these three
distinct visual representations
on screen that are being tapped.
We kind of have to look at the
motion that's happening through
the gesture, and intuit what was
meant.
Which corner did we intend to go
to?
Now, we call these regions of
the screen endpoints of the
gesture.
And, when the PIP is thrown, our
goal is to find the correct
endpoint, the one that was
intended.
And, we call this aligning the
endpoint with the intent of the
gesture.
So, one approach for this is to
keep track of the closest
endpoint as I'm dragging the
PIP.
Now, this kind of works.
I can move the PIP to the other
corner of the screen, but it
starts to break down as soon as
I move the PIP a little bit
further.
Now, I actually need to drag the
PIP quite far, like past halfway
over the screen.
Pretty close to the other
corner.
So, it's not really magnifying
my input.
It's not really working for me.
And, if I try and flick the PIP,
it kind of goes back to the
nearest corner, which isn't
necessarily what I expected.
So, the issue here is that we're
only looking at position.
We're completely ignoring the
momentum of the PIP, and its
velocity when it's thrown.
So, how can we incorporate
momentum into deciding which
endpoint we go to?
So, to think about this, I think
we can set aside endpoints for a
moment, and take a step back.
And, just really simplify the
problem.
Ultimately, what I'm trying to
do here is move content around
on the screen.
And, I actually already have a
lot of muscle memory for doing
exactly that with scrolling.
So, why don't we use that here?
We use scrolling behaviors all
the time, so we have this
natural intuition for how far
content goes when I scroll.
So, here you can see that when I
scroll the PIP instead, it
coasts along, and it slows down,
using this familiar deceleration
that we're familiar with from
scrolling.
And, basically by taking
advantage of that here, we're
reinforcing things that people
have learned elsewhere.
That the behavior is just doing
what was expected of the system.
Now this new, hypothetical,
imaginary PIP position is not
real.
We're not going to show the PIP
go here in the interface.
This is what we call a
projection.
So, we've taken the velocity of
the PIP, when it was thrown.
We've, kind of, mixed in the
deceleration rate, and we end up
with this projected position
where it could go if we scrolled
it there.
And so, now instead of finding
the nearest endpoint to the PIP
when we throw, we can calculate
its projected position and move
there instead.
So now, when I swipe from one
corner of the screen to another
with just a lightweight flick,
it goes to the endpoint that I
expected.
So, this idea of projecting
momentum is incredibly useful.
And, we think its super
important.
I'd like to share some code for
doing this with you, so that you
can do this in your own apps.
So, this function will take a
velocity like the PIP's position
velocity, and deceleration rate,
and it'll give you the value
that you could use as an
endpoint for dynamic behavior.
It's pretty simple.
If we look at my FaceTime
example of the pan gesture
ending code, you can see that
I'm just using the
UIScrollView.DecelerationRate.
So, we're leaning on that
familiarity people have with
scrolling and how far content
will go when scrolled.
And, I'm using that with my
projection.
So, I take the velocity of the
PIP and the deceleration rate,
and I create that imaginary PIP
position.
And, it's this imaginary,
projected position that I then
use as the nearest corner
position.
And, I send my PIP there, by
retargeting it.
So, this idea of using
projection to find out the
endpoint of a position, is
incredibly useful for things
being dragged or swiped, where
you really need to respect the
momentum of the gesture.
But, this projection function
isn't just useful for positions,
you can also use it for scales,
or even for rotations.
Or, even combinations of the
two.
It's a really versatile tool
that you should really be using
to make sure that you're
respecting the momentum of a
gesture, and making it feel like
the dynamic motion in your app
is an extension of yourself.
So, that's designing with
motion.
Dynamic motion.
Behaviors should continuously
and seamlessly work in concert
with interactions.
We should be leaning on that
shared intuition that we have
for the physical world around
us.
The things that we learn as
children about how objects
behave and move in the physical
world, apply just as readily to
our dynamic interfaces.
You should remember that
bounciness needs to be
purposeful.
Think about why you're using it,
and whether it's appropriate.
And, make sure that as you add
character, and texture, that
you're balancing it with
utility.
And finally, remember to project
momentum.
Don't just use position, use all
of the information that's at
your disposal to ensure that
motion is aligned with the
intent of where people actually
want to go.
And then, take them there.
So, to talk a little bit more
about how to fluidly respond to
gestures and interactions, I'd
like to introduce my colleague,
Marcos, to the stage.
Thanks for having me, everyone.
[ Applause ]
>> That was great.
Thanks, Nathan.
Hi everyone.
My name is Marcos.
So far, we've seen how important
fluidity is when designing
interfaces.
And, a lot of that comes from
your interaction with a device.
So, in this section, we're going
to show you how touches on the
screen become gestures in your
apps.
And, how to design these
gestures to capture all the
expression and intent into your
interfaces.
So, we're going to start by
looking at the design of some
core gestures like taps and
swipes.
Then, we'll look at some
interaction principles, that you
should follow when designing
gestures for your interface.
And then, we'll see how to deal
with multiple gestures, and how
to combine them into your apps.
We're going to start by looking
at a gesture that is apparently
very simple, a tap.
You would think that something--
you would think that a tap is
something that doesn't have to
be designed, but you'll see how
its behavior has more nuances
than it seems.
In our example, we're going to
look at tapping on a button, in
this case, on the Calculator
app.
The first thing to remember is
that the button should highlight
immediately when I touch down on
it.
This shows me the button is
working, and that the system is
reacting to my gesture.
But, we shouldn't confirm the
tap until my touch goes up.
The next thing to remember is to
create an extra margin around
the tap area.
This extra margin will make our
taps more comfortable, and avoid
accidental cancellations if a
touch moves during interaction.
And, like my colleague Chan was
saying, I should be able to
change my mind after I've
touched down on the button.
So, if I drag my finger outside
the tap area, and lift it, I can
cancel the tap.
The same way, if I swipe it back
on the button, the button should
highlight again, and let me
confirm the tap.
The next gesture we're going to
talk about is swipe.
Swipes are one of the core
gestures of iOS, and they're
used for multiple actions like
scrolling, dragging, and paging.
But, no matter what you use it
for, or how you call it, the
core principles of a gesture are
always the same.
In this example, we're going to
use a swipe to drag this image
to the right.
So, the interaction starts the
moment I touch down on the image
with intention to drag it.
But, before we can be sure it's
a swipe, the touch has to move a
certain distance.
We learn to differentiate swipes
from other gestures.
This distance is called
hysteresis, and is usually 10
points in iOS.
So, once the touch reaches this
distance, the swipe begins.
This is also a good moment to
decide the direction of the
swipe.
If it's horizontal, or vertical
for instance.
We don't really need it for
example, but it's very useful in
some situations.
So, now that the swipe has been
detected, this is the initial
position of a gesture.
After this moment, the touch and
the image should stay together
and move as one thing.
We should respect the relative
position, and never use the
center of the image as the
dragging point.
During the drag, we should also
keep track of the position and
speed up the touch, so when the
drag is over, we don't use the
last position.
We use the history of the touch,
to ensure that all the motion is
transferred fluidly into the
image.
So, as we've seen, touch and
content should move together.
One-to-one tracking is extremely
important.
When swiping or dragging, the
contents should stay attached to
the gesture.
This is one of the principles of
iOS.
You enable scrolling, and makes
the device feel natural and
intuitive.
It's so recognizable and
expected that the moment the
touch and content stop tracking
one-to-one, we immediately
notice it.
And, in the case of scrolling,
it shows us that we've reached
the end of the content.
But, one-to-one tracking is not
limited to touch screens.
For instance, manipulating UI on
the Apple TV was designed around
this concept.
So, even if the touch is not
manipulating the content
directly, having a direct
connection between the gesture
and the interface puts you in
control of the action, and makes
the interaction intuitive.
Another core principle when
designing gestures, is to
provide continuous feedback
during the interaction.
And, this is not just limited to
swipes or drags.
It applies to all interactions.
So, if you look again at the
Flashlight button on the iPhone
10, the size of button changes
based on the pressure of my
touch.
And, this gives me a
confirmation of my action.
It shows me the system is
responding to my gesture, but it
also teaches me that pressing
harder will eventually turn on
the flashlight.
Another good example of
continuous feedback, is the
focus engine on the Apple TV.
So, the movements on the Siri
remote are continuously
represented on the screen.
And, they show me the item that
is currently selected, the
moment the selection is going to
change, and the direction the
selection is going to go.
So, having our UI respond during
the gesture is critical to
create a fluid experience.
For that reason, when
implementing your gestures, you
should avoid methods that are
only detected at the end of the
gesture, like
UISwipeGestureRecognizer.
And, use ones like the actual
touches, or other
gestureRecognizers that provide
all possible information about
the gesture.
So, not just the position, but
also the velocity, the pressure,
the size of the touch.
In most situations though, your
interfaces must respond to more
than one gesture.
As you keep adding features to
your apps, the complexity and
number of gestures increases,
too.
For instance, almost all UIs
that use a scroll view will have
other gestures like taps and
swipes competing with each
other.
Like in this example, I can
scroll the list of Contacts, or
freely touch on one of them to
preview it.
So, if we had to wait for the
final gesture, before we show
any feedback, we would have to
introduce a delay.
And, during that wait, the
interface wouldn't feel
responsive.
For that reason, we should
detect all possible gestures
from the beginning of the
action.
And, once we are confident of
the intention, cancel all the
other gestures.
So, if we go back to our
example, I start pressing that
contact, but I decide to scroll
instead.
And, it's at that moment that we
cancel the 3D touch action, and
transition into the right
gesture.
Sometimes, though, it's
inevitable to introduce delay.
For instance, every time we use
the double-tap in our UIs, all
normal taps will be delayed.
The system has to wait after the
tap, to see if it's a tap or a
double-tap.
In this example, since I can
double-tap to zoom in and out of
a photo, tapping to show the app
menu is delayed by about half a
second.
So, when designing gestures for
your applications, you should be
aware of these situations, and
try to avoid delays whenever
possible.
So, to summarize, we've seen how
to design some core gestures,
like taps and swipes.
We've seen that content and
touch should move one-to-one,
and that is one of the core
concepts of iOS.
You should also provide
continuous feedback during all
interactions, and when having
multiple gestures, detect them
in parallel from the beginning.
And now, I'd like to hand it
back to Chan, who will talk
about working with fluid
interfaces.
Thanks, everyone.
[ Applause ]
>> Nice job.
Alright, I'm back.
So, we just learned about how to
approach building interfaces
that feel as fluid, as
responsive, and as lively as we
are.
So, lets talk about some
considerations now that we're
feeling a little bit more
comfortable with this, for
working within the medium of
fluid interfaces.
And that begins with teaching.
So, one downside to a gestural
interface is that it's not
immediately obvious what the
gestures are.
So, we have to be friendly and
clever about how we bring users
along with us in a way that's
friendly and inviting.
And so, one way we can do that
is with visual cues.
So, the world is filled with
these things, right?
You can learn them once, and you
can use them everywhere.
They're portable.
And so, when you see this, you
know how to use it.
So, we've tried to establish
similar conventions in iOS.
Here's a couple examples.
So, if you have a scrolling list
of content, you can clip the
content off the bottom there, to
indicate that there's more to
see, that invites me to try and
reveal what's under there.
And, if we're dealing with pages
of content, you can use a paging
indicator to indicate that
there's multiple pages of
content.
And, for sliding panes of
content, you can use an
affordance, or a grabber handle
like this, to indicate that it's
grabbable and slidable.
Another technique you can use is
to elevate interactive elements
to a separate plane.
So, if you have an interactive
element, lifting it up to a
separate plane can help
distinguish it from the content.
So, a good example of this is
our on/off switch.
We want to indicate that the
knob of the switch is grabbable,
so we elevate it to another
plane.
This helps visually separate it,
and indicate its draggable
nature.
So, floating elements,
interactive elements like this,
above the interface can help
indicate that they're grabbable.
Next, we can use behavior, you
know, to show rather than tell
to use-- how to use an
interface.
So, we can reinforce a dynamic
behavior with a static
animation.
So, an example of this is
Safari.
In Safari, we have this x icon
at the top left to close the
tab, and when you hit that
button, we slide the tab left to
indicate it's deleted.
This hints to me that I can
slide it myself to the left.
And, accomplish the same action
of deleting the tab through a
gesture.
So, by keeping the discrete
animation and the gesture
aligned, we can use one to teach
the other.
And, there's another technique
we can use, which is
explanations.
This is when you explicitly tell
users how to use a gesture.
So, this is best when used
sparingly, but it's best when
you have one gesture that's used
repeatedly in a bunch of places,
and you explain it once up
front, and then you just keep
using it, and keep reinforcing
it.
Don't use it for a gesture
that's used only intermittently.
People won't remember that.
Now, I want to talk a little bit
about fun and playfulness.
Because this is one of the most
important aspects of a fluid
interface.
And, it only happens when you
nail everything.
It's a natural consequence of a
fluid interface.
It's when the interface is
responding instantly and
satisfyingly.
When it's redirectable and
forgiving.
When the motion and gestures are
smooth.
And, everything we just talked
about.
The interface starts to feel in
sync with you.
And, something magical happens
where you don't feel like you
need to learn the interface, you
feel like you're discovering the
interface.
And so, we think it's great when
we allow people to discover the
interface through play.
And, it doesn't even feel like
they're learning it, it feels
fun.
So, people love playing with
stuff.
So, we think it's great to play
into our natural fiddle factor.
You know, play is our mind's
internalizing the feel of an
interface.
So, it's great when we're
building this stuff, when we're
prototyping it, just to build
it.
You know, play with it yourself.
See how you fiddle with it.
Hand it to others see how they
play with it.
And, think about how you can
reinforce that with something
like an animation, or behavior,
an explanation.
And, it's surprising how far
play can go, and having
interface teach itself to
people.
Let's talk a little bit about
fluidity as a medium.
How we actually go about
building this stuff.
You know, we think interfaces
like this are a unique medium,
and it's important that we
approach it right.
So, the first thing is to design
the interactions to be
inseparable from the visuals,
not an afterthought.
The interaction design should be
done in concert with the
visuals.
You shouldn't be able to even
tell when one ends and another
begins.
And, it's really important that
we build demos of this stuff.
The interactive demo we think is
really worth a million static
designs.
Not just to show other people,
but to also understand the true
nature of the interface
yourself.
And, when you prototype this
stuff, it's so valuable for you
because you get to almost
discover the interface as you're
building it.
You know, this technique is
actually how we built the iPhone
10 interface.
And, it's really important
because it also sets a goal for
the implementation.
We're so lucky here at Apple
that we have this amazing
engineering team to build this
stuff, because it's really hard
to build.
And, it's so important also to
have that kind of magical
example that reminds yourself
and the engineering teams, and
yourselves that what it can feel
like, you know?
And, it's really important to,
kind of, remember, remind
yourself of that.
And, it makes-- when you
actually build it, it makes
something that's hard to copy,
and it gives your app a unique
character.
So, you know, multitouch is such
an amazing medium we all get to
play in.
We get to use technology to
interface with people at an
ancient, tactile level.
It's actually really cool.
You know, all those principles
we talked about today, they're
at the core of the design of the
iPhone 10 gestural interface,
you know, responsive,
redirectable, interruptible
gestures, dynamic motion,
elegant gesture handling.
In a lot of ways, it's kind of
the embodiment of what we think
a fluid interface could be.
When we align the interface to
the way we think and move,
something kind of magical
happens.
It really stops feeling like a
computer, and starts feeling
like a seamless extension of us.
You know, as we design the
future of interfaces, we think
it's really important to try and
capture our humanity in the
technology like this.
So, that one of the most
important tools of humankind is
not a burden, but a pleasure and
a delight to use.
Thank you very much.
[ Applause ]