Transcript
[ Music ]
[ Applause ]
>> Hello everyone, I'm Julian,
later we'll be joined by Hugo
who designs sounds across our
products.
Before we begin I wanted to
explain a little bit about why
we're going to be talking about
a button for a while here.
I design and prototype user
interfaces for future hardware.
Because of the nature of my work
I can't always use standard
UIKit.
I have built and rebuilt a lot
of basic user interface elements
like scrolling, zooming, paging,
and of course buttons and I've
learned a lot about the details
involved in designing and
building these controls.
So I'm here to share a little
bit about what I've learned with
you.
I want to show you how something
so simple, a button, has a lot
of consideration and complexity
behind it.
And I hope that through looking
at this button you can learn a
few things.
I want you to use standard
controls more thoughtfully.
To consider when to build your
own custom controls.
And when you do build custom
controls I want you to consider
your interactions all the way
through.
So let's get started.
Hugo and I are expanding our
artisanal toast app empire.
We are building a new simpler
app that is connected to a
toaster.
If you imagine for a moment
you're getting ready in the
morning, you want to make sure
you have breakfast ready by the
time you're heading out the
door.
All you have to do is launch our
app, press a button and expect a
fresh piece of toast waiting for
you in your kitchen.
Now this works with our
connected toaster.
Our connected toaster goes
through a few different states,
it's ready and waiting for a new
piece of toast, it's making
toast, and finally the toast is
done, the important part.
But the way that our connected
toaster ends up transitioning
between these states is people
using our app.
When they use our app they can
request a piece of toast to
start the toaster going.
When the toast is done they will
of course eat it.
Maybe for some reason they'd
want to cancel it, I'm not
really sure about this one.
And of course we want to make
sure our toaster is always ready
to serve.
I'm not going to disclose
exactly how this works but trust
me it does and it's great.
We also have big plans for the
future, we could have AR toast,
queues of toast, multiple
toasters per household, and
eventually we'll connect this
back up to our toast social
network.
My point is there's a lot going
on in here and for today's
session we're going to dive deep
into the details of a single
screen, in fact a single button
on a single screen of our app.
If we get to the details of our
most important interaction right
the rest will flow from it.
So if we look at our system
again I think the most important
interaction here is actually
when we request toast.
Obviously, all the rest of this
has to work, but requesting
toast is the first impression
and if people aren't able to
start the toaster the rest of it
doesn't really matter.
So to do this today we're going
to start by talking a little bit
about what a button is, then I'm
going to share with you a way of
thinking about interactive
controls.
And finally, Hugo will join us
at add some sound to our Button.
So what is a button anyway?
The buttons that we are talking
about today or onscreen user
interface controls.
Buttons are indirect controllers
of action.
What I mean by this is when you
tap a button the action happens
somewhere else.
This is as opposed to a direct
interaction where here I'm
grabbing the lever and
controlling the lever directly.
Direct interactions can
reference the real-world
experience and feel more fun.
But the real power of indirect
interactions like buttons is
that they can be clearer and
more powerful because of their
separation.
The interesting thing here being
that even the physical buttons
that these onscreen buttons are
imitating are indirect.
Even though the action, the
result is happening somewhere
else the button controls that
action.
And just like the action a
result can be designed a button
can be designed, we can consider
both of these things separately
and connected.
To do this today we're going to
be relying heavily on a single
lens of design feedback.
Feedback is how you let people
know what your app is doing.
I'm sure a lot of you have
thought of a variety of
different ways that you can
provide feedback within your
apps and are already doing so.
But I think there are two major
categories of feedback.
The first is that you can tell
people, you can literally put
text on screen explaining what's
going on or design icons that
explain things.
The second is that you can show
people, you can allow people to
experience the results of their
action more like the real world
through visual, audio or even
haptic change over time.
Now we're going to apply
feedback to three phases of an
interaction, not just its
completion.
We can think of these phases in
terms of a physical button.
Before I have pressed the button
it's just hanging out there.
During my interaction with that
button it's being pressed in.
And finally after I've pressed
that button my finger is lifting
off of it and it's returning to
its original state.
So what does it even mean to be
providing feedback before I have
begun interacting with a button,
why does this even matter, the
button's just there right.
Well if we think about our
iPhones for a moment, the iPhone
screen is just a piece of glass.
This piece of glass happens to
have glowing pixels underneath
it and what matters is what
people think those pixels mean.
Glass doesn't tell you that it's
tappable, but something on
screen might.
You could see that you could
page through something, drag
something or of course for the
purposes of this presentation
tap on something.
The academic term for what I'm
talking about is perceived
affordance.
Affordance refers to the
connection between an object and
the person interacting with it.
A person might know that they
can grab a handle or press in on
a physical button.
But the perceived part of this
is talking about what people
know about those glowing pixels
on screen.
People only know what they can
interact with on that iPhone
screen because of their prior
experience having used iOS
before and their current
context, the fact that they're
on this phone in an app that
they may or may not understand.
So let's apply this to our toast
app of course.
How do I know what will happen
when I tap this button, I just
put a button up and call it okay
right, no.
What we do is provide a label, a
label is feedback in which you
are telling people literally
what is going to happen.
In this particular case I tried
the label toast, but
unfortunately toast refers to
both the verb to toast and it
also refers to the delicious
piece of bread you're going to
be eating for breakfast.
So what should we call this
label, toast toast?
No. We could try to design an
icon for this button but I think
this isn't particularly clear,
especially for something that
people will be encountering for
the first time in an unfamiliar
context.
So we thought about it a little
bit and we came up with the
phrase make toast.
I know that in our diagram we
refer to this as request toast
but we spent some time thinking
about what people using our app
might understand and what feels
most comfortable for our app.
The next thing to think about is
how do I even know that this is
a button, just a rectangle
floating on a black background
might not be enough.
But if you add some context like
an iPhone screen you might start
to understand it or you could
add some prior experience.
It could be the shape of the
button, the manner in which is
rendered or if it's used
consistently enough just the
color of text.
In this case, I don't think
people are going to be very
familiar with our app they're
going to be launching it once a
day, more if we're lucky so
we're going to focus on more
standard iOS button shapes
because we don't want to throw
in something new.
So we try these out on screen.
Another thing that we can think
about with respect to feedback
before we begin using this
button is how do I even know
what this button relates to.
It could be something about
where in the app a button is
located, these buttons have
different meaning and feel.
Or it could be something about
how it's grouped with other
controls or how close it is to
other objects on screen.
I'm starting to like this last
one here so we're going to try
it out on a real screen.
I drew this user interface in
Keynote and I put it on a phone
and I'm holding this phone in my
bedroom where I might expect to
use it in the morning before I
order toast.
And I'm just trying to
understand do I know what this
button will do.
The next thing we should think
about is what happens during my
interaction with this button.
For a physical button I'd be
pressing it in.
Now the term feedback refers to
telling people something about
the result of what has happened
and in this case, we haven't
actually done anything yet, so
we might use the term
feedforward for this.
Because feedforward refers to
helping people understand what
is happening while they're
interacting with this button.
Feedforward is a component of
making interactions feel fluid,
you can convey responsiveness
for things like 3-D touch, pinch
or slide movements.
Thinking about how feedforward
applies to our button we should
be thinking about what is
happening now, what is happening
while my finger is making
contact with the screen, with
that button.
In the case of our overall toast
system we are in the middle of
touching a button to request
toast.
So the part we're going to focus
on is the toaster being ready
and eventually transitioning to
the toaster making toast.
We should be thinking about what
is happening and how I can hint
toward what will happen.
Let's say I've begun touching
this button, how do I know that
I've begun tapping this button,
why does it even matter?
Well, for one, the hit area of
this button might not be what
you expect, if you look closely
at the circle maybe I'm not
quite touching it.
But perhaps the hit area is
actually a little bit larger
than the visual size, this is
particularly important for very
small controls if we made that
lever interactive.
So how do I know I've begun
tapping this button?
Well you might be tempted to
animate the button, to have it
fade in or grow very slowly, but
this can make things feel
unresponsive.
This kind of animation might be
appropriate for 3-D touch if
it's responding continuously as
I'm pressing in.
But for a quick contact with the
screen we might want to do
something faster, if we are
going to animate it do it very
quickly.
So what we do is provide a
confirmation sound, a haptic or
visual change.
In this case, we're just going
to highlight it instantaneously,
make it feel a little bit
faster.
This also helps us know that a
button is enabled and that it
can work.
Maybe we try connecting the
feedforward on the button to the
feedforward of the toaster
itself, as I said we can start
to hint toward what will happen.
So here what I'm trying is as I
touch this button maybe we show
the piece of toast starting to
appear.
I feel like that might be a
little bit too much for this
button though, so I'm not going
to go forward with that but I
tried it.
Another thing that we should
think about with feedforward is
what happens if I change my
mind, let's say I'm kind of
groggy in the morning, I pick up
my phone I accidentally touch
this button but I don't want to
start my toaster yet.
Well for most buttons I can
actually drag my finger outside
of the button to begin canceling
it.
This is something that's
actually very important for all
sorts of fluid interactions
because it's what makes a fluid
interaction undecided.
If you're paging through
different screens you don't know
which screen it's going to end
up on until you lift your finger
and the same thing applies to a
very simple button.
I can bring my finger back
inside the button and have the
same feedforward show me that I
am now activating this button
again.
Now that we've decided a little
bit about the feedforward that
we're going to have we should
try this out on a device as
well.
So again, I know this seems
really simple for a button, but
this is very important for
scroll, pinch, paging, a lot of
touch interactions.
Finally, we should think about
how to apply feedback after I
have lifted my finger up off of
this button.
For a physical button it's going
to return to its original state.
For our onscreen button we're
going to have lifted our finger
off the screen.
This is of course the realm of
classic feedback that most of
you are familiar with.
But the thing I want to point
out here is that we can be
thinking about feedback both for
the button itself and the
action, the result that is
happening, and we can think
about how to connect the two so
people understand what has
happened.
Again, within our system we're
only focusing here on getting
people into making toast.
So what happens when I lift my
finger from this button?
Does the unhighlight happen
immediately or is there a slight
delay so that even fast taps can
be seen?
Is there a timer, does this
button support double tap, if so
I may have to wait a little bit
before showing the confirmation?
If we think back to the two
types of feedback we can just
literally tell people what is
happening.
I can put some text on screen, I
think it's very clear but it
also requires reading and more
text.
So maybe we can show people,
maybe the button itself provides
some feedback here it's flashing
or maybe it's something that
represents the action that
provides feedback.
For example, the icon of the
toaster itself could show you
that it's starting to toast.
Or what we can do is we can just
do both, I can put text on
screen and I can show the
toaster.
This is great because I might
have missed that animation, but
I can still see what state the
toaster is in.
But similar to interacting with
the button with feedforward I
should think about what happens
if I change my mind.
I've started toasting but maybe
I want to cancel, maybe I want
to stop the toaster.
We could replace that button
with a stop toasting button, but
I think that's not particularly
clear because this button looks
an awful lot like the make toast
button that was up there before.
So maybe there's a separate stop
button.
I feel like that red is a little
bit too much, so maybe we try
that same button but separate
from our make toast button.
This is starting to feel a bit
better, so what I should do is
try it on device again.
We should see how the feedback
on the button itself connects to
the feedback from the toaster
communicating what has happened.
To recap a little bit of what I
went over.
Feedback is how you let people
know what it is that your app is
doing.
You can literally tell people,
you can put text on screen it's
fine or you can show people, you
can allow them to experience the
change over time.
We can apply feedback to all
three phases of an interaction,
even a simple button.
And something I was implicitly
doing here is trying out options
to understand what's too little
and too much.
The only way I can really know
if an animation is appropriate
or not is to try it.
So what we've done here is we've
designed a very important part
of a much larger system, but I
think it'll be a lot easier to
figure out the flow of the rest
of this once we get this one
piece right.
Now that our button is designed
and pretty great I think we can
start to think about sound, so
I'd like to invite Hugo up to
the stage.
[ Applause ]
>> Thank you Julian.
Hello everyone.
Now we can make this quick and
add a simple click sound to this
button.
Sounds pretty good or could it
be better?
Could Julian have done a better
job and invited me a little
earlier on in the process?
I'd like to tell you a few
things about how as I a sound
designer could approach a
project like this.
I'll talk about where to look
for inspiration, about designing
the sound, and some key building
blocks that we use while doing
this.
But before we dive in why do we
talk about sound?
Now some of you may think our
users mute the sound on their
phone quite often, is it
important to add sound to my
app?
Well of course it's up to you
but George Lucas has said that
50% of the experience of a movie
is determined by what we hear,
by its music and sound design.
And sound is all around us, we
use it to navigate through the
world.
And sound can help enrich the
experience of your products.
And it helps us remember the
experiences of your app, as well
as your brand, so it can have a
big impact.
So where do we look for
inspiration when we want to
design sound for a button?
Well we can start by taking a
little detour and looking at
some real-world buttons.
Some of them are made of cheap
materials and sound like this,
while others are made of high
quality materials and sound like
this.
Well this was actually a
recording of the volume button
on an iPhone X, of course we
amplified it quite a bit.
These sounds though are
essentially byproducts, they're
not necessarily designed they're
defined by the materials these
buttons are made of.
But because we're designed for
software we have a lot of
freedom, so we can choose to not
play any sounds at all and often
that is what we do.
But if we do decide it's useful
to add sounds to UI elements
then we can take cues from
real-world analogs to add
meaning to these UI sounds and
to help our users understand
them.
Now do you want to add sound to
your app?
Well it depends, think about
category your app falls in, who
will use the app and what will
their expectations be, and what
is the context, where will the
app be used.
For our app we know it will be
used by people who like to eat
toast and they're used to making
toast in the kitchen, but from
now on they can make it from
anywhere in the house.
They can still though use the
audible cues they're used to
getting from their toaster in
the kitchen, so for that reason
we think that this app is a
perfect candidate for the use of
sound.
So let's look at the process of
making toast and listen to
sounds we encounter.
Let's define if these sounds are
useful and if they are if we
want to use them in our design.
So we'll use Julian's timeline
for this and listen to the
sounds that this toaster makes.
First, we pull down the lever
and then we press it in place.
Now what we hear then is the
toaster heating up with a subtle
buzzing sound of the coil.
[ Buzzing Sound ]
Until our toast is done and the
toaster ejects the toast.
Now let's go through these steps
one by one.
First, when we pull down the
lever we feel the resistance of
the spring while we push it
down, it's almost like if we
compare it to what happens in
our app it's like designing
sound haptics and animation and
synchronizing them all.
But in our app, we decided to
replace this with a simple
button so we don't need to
design a sound for it.
And when we request the toast
that is when we press our
button, the next step, and that
is what Julian asked me to
design a sound for so let's get
to work on that.
We can record sounds, we can use
electronic instruments,
synthesizers to generate sounds
and then we can use software to
scope these sounds into the
right shape.
Now I'm not going into great
detail about this today, but
please watch last year's talk
Designing Sound if you want to
learn more about this process.
In the end, we came up with
three different options.
This is option A, this is option
B, and this is option C.
Okay now option A is quite
minimalist, it's so simple, it
works pretty well with the
simple UI of the app but it
feels like there's something
missing.
And we found out that while we
were designing these that it's
much more satisfying if these
sounds have two clicks, one when
you press down and one when you
lift off your finger.
Now option B has this character,
but and now it is a sound that's
much more realistic, it doesn't
really sound like a toaster but
it does have this physical
quality to it.
For the same reason though it
does sound a little harsh and a
little metallic.
And then there's option C, it's
a bit of an odd one but it does
have character and it's a sound
that I would recognize if I hear
it again if I use my app again.
Another thing that's good about
this sound is that it has this
tonal confirmation built in,
it's like throwing a checkmark
telling you your action
succeeded.
I'll play it one more time see
if you can hear this.
And for all these reasons I
think that option C is the one I
would recommend going for, it's
a simple sound, it has a
friendly character, and it's
something unique it's not
something I'd recognize from any
other app.
Okay so it looks like we're
done, at least we made our
button click but we're not done
making toast yet.
So let's keep going, maybe there
are some other opportunities for
the use of sounds in this app.
Now as good sound designers we
want to tell the whole story.
So after pressing our button the
toaster heats up and we hear
this subtle buzzing sound.
Now we can -- is this helpful at
all this sound?
Maybe, it tells us something
about the state of the device.
So we can choose to play a sound
that resembles this on our app
as well, maybe a continuous
sound, something like this.
[ Notification Sound ]
But making toast may take
several minutes and having this
sound play over and over again
might not be the experience
we're looking for right.
So how about something a little
more subtle like a timer sound
that fades in and out every half
a minute or so.
That could work.
But does this really need a
sound at all because what we're
really waiting for is what comes
next, that's when our toast is
done and this is what we want to
notify our users about, that's
the important moment.
So we want to play a clean and
simple notification sound that
they won't miss, something like
this.
[ Notification Sound ]
Okay so now let's listen to
these sounds together.
First, we press our button and
then we decided for the next
step when the process is active
we don't really need a sound
until we get our notification
and the toast is done.
And it's important to listen to
these sounds together to make
sure they family well together
because this is how our app
communicates to its users, this
is the voice of our app.
So now our app is in good shape.
I'd like to tell you about some
key building blocks of sound
that we use when designing any
kind of sound.
The first one is timbre or tone
color.
And most of you will instantly
recognize this one.
It's a piano right, but how do
we know it's a piano?
How do we know it's not a
vibraphone?
It's determined by the material
an instrument is made of, by its
shape, and how it's being played
using a mallet, a hammer, a bow
or your hands.
So these instruments can play
exactly the same tone but sound
very different.
Now for our app we decided to go
for a character, for a timbre
that is friendly and it's not
too harsh or metallic.
The next one is frequency or the
pitch of the sound.
Some instruments play at very
high frequencies and others play
at very low frequencies.
But also nonmusical sounds can
be played at very high or low
frequencies.
And when we do this it gives an
indication of the size of an
object.
So if we take our toaster and we
record the toast eject sound, we
play it at a very high frequency
it sounds like a tiny little
toaster.
But if we take the same sound
and play it at a low frequency
it sounds like this big giant
toaster.
So next up is duration or the
length of a sound.
Sounds can be very short or ring
out much longer.
And if we design UI sound and we
know our button is going to be
pressed a lot of times in a row
it's important for this sound to
be very short and subtle.
But if it's only pressed once in
a while, which is actually the
case for our toast app, it's
okay if it's a little longer, if
it's a little more elaborate.
In fact, this sound only plays
once every time we use our app.
And then there's loudness or
amplitude or volume.
Now when we design ringtones or
alarm sounds we don't want our
users to miss them, we don't
want them to miss their phone
calls or sleep through their
alarms.
[ Notification Sound ]
Our audio engineer Mitch is
calling me from the back of the
room and I hear it loud and
clear.
But if a UI sound would be loud
like this it would become
unpleasant.
It only needs add this subtle
[inaudible] layer to the
interaction.
Now let's quickly reply to
Mitch.
[ Keyboard Sound ]
Now if I would miss this
keyboard sound because I'm in a
noisy environment it's not a
real big problem, but I
definitely don't want to miss my
ringtones.
So to recap the four building
blocks of sound, our timbre,
frequency, duration and
loudness.
And when you start thinking
about designing sounds and
adding sounds to your app it's
really good to keep these in
mind.
And that concludes what I want
to tell you about sound today,
but before we end the session
these are the most important
things that Julian and I want to
leave you with.
Details are designed even if
they seem obvious, even if
they're as simple as a button,
even if they're as simple as a
click sound.
Now you can take inspiration
from real-world elements, but
there's no need to copy them
one-to-one.
Also use the freedom and the
flexibility of designing for
software to create something
that is completely unique and
fresh.
And then what we see, the
animation, and what we feel, the
haptics, and what we hear, the
sound, they all combine into
this one single experience.
And lastly, but most
importantly, learn by trying
things out.
You know we tried out a lot of
things today and some of them
didn't work that well while
others worked great.
But the only way for us to find
out was to try them out first.
Now I hope this all inspired you
to spend some more time on the
details you might normally take
for granted.
If you get these details right
it can make your apps even more
amazing.
Thank you for listening, have a
great day.
[ Applause ]