Transcript
[ Applause ]
>> Hello.
Welcome, everyone.
Imagine, the next version of
your app will have no graphical
UI at all.
How would you interact with it?
What could that sound like?
I'm Hugo Verweij, sound designer
in the Human Interface team.
I had the pleasure on working of
many of the sounds you hear
through your Apple devices, and
I love thinking about these kind
of questions.
There are probably some
musicians in the room, maybe
even one or two sound designers,
but that is by no means required
for this talk.
This session is for everyone,
even if you've never recorded a
single sound in your life.
I'm also not going to teach you
how to become a sound designer,
but I hope I'm going to give you
some ideas of how to incorporate
sound in your design process and
why it makes sense to do so.
Now, as a developer, you have a
lot on your plate and adding
sound to your app is probably
not super high on your to-do
list, but I like to make the
case that sound, if used the
right way, really can help
transform the experience of your
product.
Now at Apple and at WWDC, we
talk a lot about visual design.
For now though, let's just try
to use our ears for a little
while.
I'd like to ask you all to just
close your eyes for one minute,
and imagine you're in a forest.
This might help you.
[ Forest sounds ]
It's a quiet afternoon, and
you're going for a walk.
[ Footstep sounds ]
[ Birds squawking ]
The weather seems very nice.
But then [phone notification
sound] Dark Sky is predicting
rain.
You better pick up the pace.
[ Footstep Sounds ]
[ Rain and Thunder Sounds ]
Okay, you can open your eyes
now.
Oh, we're not in a forest.
We're actually at WWDC.
Sound has this magical ability
to calm us down, to get on our
nerves, to scare us, or to
delight us.
When we close our eyes, it has
the power to transport us to
places.
I want to talk about the
following things today.
Why sound is an important aspect
of design, designing sounds for
notifications, designing sounds
for UI elements, and some tips
and techniques to get you
started.
Sound is everywhere, all around
us, and we never really pay much
attention to it.
[ Church Bell Ringing ]
Church bells have been telling
the time for many centuries in
countries all across the world,
like here in my beautiful home
town, Amsterdam.
They tell us the time.
They tell us where we are.
They give us a sense of context.
We also use our ears for an
early warning system, like when
we're about to cross the street
and we hear this --
[car accelerating] [brakes
screeching]
-- we don't need our eyes to
understand it's not safe to
cross.
Or we listen for quick
confirmations every day.
When you put on your seatbelt
[seatbelt click], you don't have
to double check it's fastened.
You hear the click, and you know
you're good to go.
We can also listen for more
nuanced things, like the sound
of someone's footsteps may tell
us something about their
emotional state
[slow footsteps].
Are they calm?
Are they in a hurry?
And if we know a person very
well, we may even recognize them
by the sound of their footsteps.
[footstep sounds].
So what we hear has a big impact
on how we feel and what we do,
and sound helps us make sense of
the things going on around us.
Now we know that sound is such a
powerful medium.
How can we use it in our design?
Many people think that design is
about what a product looks like
or at Apple we pay a lot of
attention to, how it works.
We think design is also about
what it feels like and what it
sounds like.
So what is the role of sound in
design?
It's like using a universal
language that is already
understood by everyone.
It helps shape the experience of
a product.
It helps us understand
interactions better.
And it gives our devices a voice
so they can tell us things, they
can talk back to us.
So if your iPhone is playing a
sound, what is it trying to tell
you?
[beeping].
You just used Apple Pay
successfully.
[beeping].
Or maybe it didn't go so well.
Try again, please.
[beeping].
Somebody just texted you.
[ Alarm ringing ]
Or good morning.
It's time to wake up.
[ Alarm ringing ]
Or at the end of the day [music]
it's almost time to go to bed.
Many times though, when an app
plays a sound, we don't really
know what it means.
It sounds like this.
[beeping].
We have to look at our screen to
understand what's going on.
It's like a way for our phone to
say, "Hey, look at me.
Something happened, but I don't
really know what."
It's not a very intelligent way
of communicating.
So how can we improve on this?
Well, if your app relies on
notifications, it makes a lot of
sense to think about creating
your own notification sound.
What do you want people to
experience when they receive
that notification?
Do you want people to recognize
that it's coming from your app?
If you don't do this, you are
stuck with the default we just
heard, and it doesn't really say
much about where it's coming
from and what it means, so it's
a missed opportunity to
distinguish your app, your app's
unique identity.
Now, before, when we were in the
forest, we heard the Dark Sky
notification, and Dark Sky is a
weather app, and it sends this
notification when rain is
imminent.
It does this pretty well.
Let's listen to it again.
And why does this work so well?
[beeping].
This sound is easily
recognizable.
When you hear this, you know
it's coming from Dark Sky.
It conveys a message.
It's not a direct recording of
raindrops, but it has this
liquid quality to it.
It helps you connect the sound
to the message it's sending.
It's a friendly sound.
It's not jarring, but telling
you in a friendly way it's time
to find cover soon.
And it's a simple and clean
sound, quite pleasant to listen
to as well.
So let's try this ourselves.
Let's create a sound for an app.
Shall we? Well, first we need an
app, of course.
Well, luckily, some years ago,
our prototyping team did a
session.
It was called "Fake It Till You
Make It."
It was about creating fake apps
in order to learn from how
people would interact with them.
They create an app called Toast
Modern.
Now, maybe some of you know that
toast has become very popular in
San Francisco, and people pay a
lot of money for a slice of what
some call hipster toast.
Toast Modern, the app they
created, helps you find
overpriced artisanal toast on
the streets of San Francisco
[laughter] and share this with
your friends and family.
Now if someone sends you some
newly discovered toast, you get
a notification.
So we can create a sound for
this notification.
Let's do that.
We can start by asking ourselves
some questions, like, "What is
the essence of what this app is
about?"
Well, in this case, simple.
It's about finding toast and
sharing it with your friends.
What emotion do we want to evoke
here when people receive this
notification?
Well, we want them to be happy
they found some new toast to try
and eager to try it out.
How do we match the app's
aesthetics?
We just saw the screen, and it
looks pretty functional and
simple, so let's keep the sound
similar to that, keep it simple
and to the point.
And if we notice, we can start
with some sound sources.
We can record some sounds to be
inspired by.
In this case, let's just get
some bread.
Let's get a toaster and record
those sounds.
So that's exactly what we did,
and here are some of the sounds
we recorded.
[ Toaster popping ]
Toaster ejecting toast.
Spreading butter on a piece of
toast.
[ Bread Crumbling ]
The sound of toast breaking.
And taking a bite out of a piece
of toast.
Now we can also create a little
bit of a story from the
beginning to the end, you know,
making toast and eating toast,
and record the whole process.
[toaster popping].
[timer ticking].
[chiming].
[scraping toast].
[toast crunching].
[laughter].
Okay, now we have a nice
palette.
You notice we added some sounds
to this, because in the initial
set, there was not that timer
sound and the chime sounds, so
we recorded those two.
And now we can start making
notification sounds.
There's one sound that I like
quite a bit.
[toast crunching].
It's a great sound.
It's very characteristic, but
maybe it doesn't work so well as
a notification coming from your
phone every day, right?
It's a little harsh.
So how about this one?
[timer ticking].
[chiming].
There's this little timer sound
and then the chime, but the only
thing I don't really like about
it is the sound of the toaster
ejecting the toast.
It's a little harsh.
So we can find a more expensive
toaster maybe, that'll sound
better, or we can construct our
own toaster sound.
But why don't we just take away
the parts we don't like and see
what we're left with?
So let's take that sound out.
[timer ticking].
[chiming].
Actually, this might work pretty
well.
Let's see what it looks like
when the notification comes in.
[timer ticking].
[chiming].
I think this is pretty good for
now.
Now remember, we're just trying
things out.
We're just prototyping.
Even the app is fake.
So, you know, we're not all
sound designers, but once we
start working with a sound
designer, they can take this
inspiration and maybe create
something that's more unique,
more ownable, and maybe more
abstract, like the Dark Sky
notification.
Now what did the creative
process look like here?
We started out with all these
toast recordings.
Those are the blue dots.
And then we realized we needed
some other sounds to complete
the palette, so we added the
timer sound and the chime sound.
Those are the yellow ones.
And then we created some
candidates and tried them out on
our device.
We found out that a lot of these
sounds didn't really work so
well, so we ended up with the
timer and the chime sound.
Now why am I showing you this?
Because it illustrates how often
we start out with this
assumption that something might
work, but then we find out that,
going this exploration phase, we
find out that what we really
need is something different.
We have to go through this to
understand that.
First we have to make mistakes
to come up with something great.
Now what kind of sound sources
can you record when you do an
exploration like this?
You could record an acoustic
instrument or an electronic
instrument, like a synthesizer
software or hardware.
But really, we found out you can
record anything.
Anything that makes a sound can
be an inspiration.
Now at Apple, we use any of
these sound sources.
It really depends on the project
we're working on, and some of
them are easily recognizable
instruments.
These sounds may be played
millions of times per day.
Earlier this year, Tim mentioned
that at one point during the
Super Bowl, 380,000 messages
were sent per second.
Never though in the history of
the sound has it been played
live on stage.
Really, nobody ever thought of
that.
But today's a special today.
So ladies and gentlemen, for the
first time in the history of
this sound since it was
introduced with iOS 7.
Here it is.
[laughter].
Concentration.
I hope I hit the right one.
[bell ringing].
[ Applause ]
Thank you very much.
We use this notification sound
for messages.
Why does this work well for
messages?
We have to think about, when
will people hear this sound, and
what will they be doing?
They have their phones in a
pocket, in their jacket.
Maybe they are on a busy city
street or an airport.
So we have to cut through the
noise of these locations.
It needs to be very clean and
simple, but there's another
thing that is important that we
don't really want to, you know,
convey the information of what
you're sending.
It's very different than the
Dark Sky notification that can
only mean one thing.
It's going to rain.
In this case, it can mean
anything, anything you're
sending with it.
So we want it to be very neutral
as well.
Now this here is a kalimba.
Maybe you've seen one before.
This is what it sounds like.
[ Kalimba playing ]
You recognize it?
Maybe you do now.
[kalimba playing].
[laughter]
[ Laughter ]
So we recorded this sound as the
default notification for --
anyone?
Calendar, that's right.
Absolutely.
It's a little bit more of a
characteristic sound, because
it's not a very common
instrument, but again, it's very
simple and clear as well.
Now when we were designing for
the watch, we asked our self the
question, how can we be inspired
by this beautiful industrial
design of this device?
And how can we create the
illusion that what you hear,
that the sounds are actually
made by the device itself, that
they're coming from the device
itself?
So we decided to record the
different materials, the
stainless steel, the aluminum,
and the gold.
And what I have here is the
housing of a stainless steel
Apple watch.
Now we can hit that in different
ways and see if it makes a
sound.
[ringing sound].
If I hang it up, try a couple
times.
[ringing sound].
Hit it the right way.
[ringing sound].
It actually sounds quite nice.
So we decided to use this as an
inspiration, and we used these
actual recordings for some of
the sounds that are on your
watch today, like [ringing] the
notification and the watch ring
tone.
[ Ringing sound ]
So together with haptics, what
you feel, what you see, and what
you hear really come together
here in one experience.
Now you may still think, "I have
no idea how to record a sound.
I've never done that before.
Well we've got some very simple
tools you can start with,
already on your phone.
There's Voice Memos.
It's a great recording app.
You can record any kind of sound
with that.
And there's Music Memos.
You can download it for free
from the app store, and it's a
great tool for more musical
ideas, and it works really well
with Garage Band.
Speaking of which, Garage Band
can be used on your Mac to
record sounds and edit sounds.
And for those who want to dive a
little deeper, there's always
Logic Pro, a great tool for the
more seasoned users.
So to recap, what are the
characteristics of a good
notification sound?
It's distinguishable.
You know which app is sending
the sound.
It matches the design of the
app.
So you're creating an app that
looks very cartoony, let's say.
You may want to look at some
sounds that really match that
aesthetic.
Or if your app is very
minimalist, then maybe you want
to start with some very simple
sounds.
It's unobtrusive and repeatable.
This means that even if you use
it multiple times per day that
it doesn't become annoying.
A great way to try this out is
just to put it on your phone for
a little while, for a week or
so, and live with it for a
while.
If after a week you still like
the sound, it's probably a
pretty good one.
And then it's clear.
It's clean.
It cuts through noise, but it's
not abrasive.
All right.
Next, let's look at designing
sound for UI elements.
Now every object you interact
with in the real world makes a
sound, so it's very natural to
expect that an object you
interact with on the screen
makes a sound as well, but we
definitely don't do this for
every interaction.
We really try to find the right
balance between no sound and too
much.
So UI sounds are things like
keyboard sounds, scrolling
sounds, pressing a button,
tapping, swiping, or any kind of
transitions.
Now I want to focus on three of
these and see how we approach
them in our own devices.
Let's start with the keyboard
sound.
Now following our previous
design strategy, we may want to
start by recording some actual
keyboards, like an old
typewriter.
[ Typewriter clicking ]
Now, this rattles quite a bit,
but there's something satisfying
about the letters hitting the
paper.
So let's remember that.
How about a computer keyboard?
[ Typing sounds ]
There's something pleasant about
this sound as well.
It's not very static.
It's very lively.
So not every key sounds the
same.
So we can remember this when we
create our own sound for our iOS
keyboard.
We're designing for software, so
we can make it sound any way we
want.
For instance, we can change the
sound of the space bar, of the
modifier keys, and the backspace
keys.
Let's listen to it.
[ Popping sounds ]
So we create a character that is
a bit more playful and not so
static.
And we do another thing.
When you type faster, we bring
down the volume of the sound
ever so slightly.
It's not something that you may
hear in everyday use, but even
these little things make a
difference in the experience.
Let's listen to it one more
time.
[ Popping sounds ]
Okay, now next it's pressing a
button.
For those of you with an iPhone
7, I invite you to bring it up
to your ear and click the Home
button a couple of times.
Go ahead.
What does it sound like?
Now since the iPhone 7, we have
a solid, safe Home button.
It's durable, and it's
waterproof, but it doesn't
actually move or make a "click"
sound.
So we used the Taptic Engine to
create this button experience,
and we learned while designing
for the watch that combining
sounds and haptics the right way
and synchronizing them
precisely, we can come up with
this great multisensory
experience, and we were looking
for a similar experience for
this button, so we decided to
add a very subtle, quiet "click"
sound to the tap of the Taptic
Engine, resulting in what you
hear right now when you press
it.
And this also allowed us to
create multiple "click" fields.
So when you set up a new iPhone,
you can choose between two
different clicks.
There's one more thing that was
very interesting when we were
doing this exploration.
We found out that, if you take
one haptic tap, and you
synthesize it to a very sharp
and precise sound, like the one
on the left here.
It sounds like this.
[ Clicking sound ]
And people think that what they
feel is very precise and sharp
as well, but if we take exactly
the same tap, exactly the same,
we didn't change anything, and
synchronize it to another sound
like this [sound], then people
think that what they feel is
very different.
It's less sharp and less
precise.
So it's really interesting how
we can fool ourselves using
sound this way.
And lastly, let's look at
transitions.
These are the sounds that
accompany a movement between two
different states, like in the
Apple TV.
We're looking for two things
here.
A feeling of depth and a feeling
of movement.
Let's listen to it.
[ Transition sound ]
So the special character of this
sound emphasizes the feeling of
depth, and the envelope of the
sound is shaped in a way similar
to that of an object passing you
by, creating that feeling of
movement.
We could have used another sound
with similar characteristics,
but you really also want it to
match the aesthetic of the Apple
TV user interface perfectly.
We think this sound does the job
really well.
And there's also a sense of
direction, because we have two
different sounds.
You know if you're moving
forward or backward.
Let's listen to it one more
time.
[ Forward transition sound ]
[ Backward transition sound ]
So what are the characteristics
of good UI sound design?
First off, it's used sparingly.
Really, again, try to find a
balance between no sound and too
much.
It's a delicate balance.
It has a much lower volume than
a notification sound, because
people have their phones in
their hands.
We can keep it very quiet.
We only want to add a very
subtle layer to the experience.
Now they help the users, and
they delight the user by
creating this great visceral
experience combined with haptics
and animation.
Now in this last section, I'd
like to give you some tips and
techniques to keep in mind when
you're working on your own app.
First of all, keep sound in mind
from the start.
When you're beginning your
project, you can ask yourself
these questions.
Will my app send frequent
notifications?
If this is the case, it makes a
lot of sense to think about
creating your own notification
sound.
Can sound play a role in my
app's branding?
Well, if the answer to the first
question is yes, and you do
create your own sound, then it's
already playing a role.
You're already creating a voice
for your brand.
Now how can the UI benefit from
an audible component?
Is there any way you can help
people understand what's going
on on the screen by adding some
subtle sounds to it?
And then how would I understand
my app without a graphical user
interface?
The question I started out with.
Of course, you don't have to use
your app without looking at it,
but it's an interesting question
to help yourself, and it may
help you come up with some
innovative ideas.
This is an important one.
Silence is golden.
It may sound funny from a sound
designer, but the absence of
sound is as important as adding
sound.
Really try to find the right
balance.
If we wouldn't do this, our Maps
app could easily sound like
something like this.
[ Different sounds playing ]
[ Laughter ]
[ Different sounds playing ]
>> 3D.
[ Laughter ]
[ Different sounds playing ]
>> A bit much, I think.
[applause].
Now, of course, if you're
creating a game, it makes a lot
of sense to come up with a whole
world of sound, like a whole
sound world that fits your game
world.
But we don't want every app to
sound like a game.
Now if you do add UI sounds,
also add the option for people
to turn them off.
Now the last thing we want is
for people to always hit the
ringer switch, or even worst,
delete your app because they
don't like the way it sounds.
So be very careful when adding
these sounds.
And if you do, add the option
for people to turn them off.
Just as much as you pay
attention to every single detail
of the visual design of your
app, do the same thing for
sound.
Here's some things to keep in
mind, especially when designing
for a device like the iPhone.
If I'm listening to music, and
your notification sound comes
in, the music ducks, which is
fine because I choose to get
these notifications.
They give me information.
But if your notification sound
is too long, it ducks for a very
long time, so it becomes
annoying.
So ask yourself the question,
does it get in the way of the
user in any way?
We all know how frustrating it
is if you're watching a video,
and what you hear doesn't line
up with what you see.
[footstep sounds].
You know, it gives us this
uneasy feeling that something is
not right here, and it's so much
more satisfying when everything
lines up nicely.
[footsteps sounds].
So if you're adding sound to
haptics or animation, then
really try to find the right
moment to synchronize to.
We found that adding sounds to
haptics, that pushing it back or
forward by as little as ten
milliseconds sometimes makes a
difference in the experience, so
really experiment with this a
little bit and try to find the
right timing.
Now for a couple more pro tips,
this is a spectrogram.
It's a visual representation of
a sound and of the frequency
present in a sound.
The brighter the color, the
louder that frequency.
This particular sound sounds
like this.
[ Bell ringing ]
There's something that can be
improved upon here.
First off, there's a bunch of
silence in the beginning.
The actual sound starts here.
And then I don't know if you
heard it, but there was a lot of
noise in the background.
You can see it on the screen as
well.
It's the visual noise.
And then the sound was truncated
at the end, so it cuts off.
It sounds very unnatural.
Now a professional with some
expensive software can easily
make it sound like this.
[ Bell ringing ]
It's much cleaner.
The background is darker, less
noise there, and it fades out
nicely.
But like I said, you need some
expensive software for it, or do
you really?
Because we found that using
Garage Band, you can actually
come pretty close.
You can adjust the beginning of
the region, stripping out the
silence, and then you can fade
out the sound using track
automation.
We found a preset called iChat
Voice that has a denoiser
function.
It's meant to be used for
speech, but you can use it for
any kind of sound, and we can
come up with this.
[ Bell ringing ]
It's much cleaner.
So if you're inventive, you can
come a long way with a simple
tool like Garage Band as well.
Also pay attention to the
frequencies you're playing.
The iPhone speaker is a
wonderful speaker, but there are
some physical limitations,
especially low frequencies.
It may not translate that well,
so it makes sense to filter out
those frequencies that you don't
use.
This will clean up the sound and
makes it sound more clear.
How do you find out what works
and what doesn't work?
Just test it out on the device
that it's going to be played
upon.
So if you're creating a game
that will run on iPhone as well
as on iPad, then test out the
sounds on the iPhone as well as
on the iPad, and realize that
people will be wearing
headphones, so also test it out
for headphones.
Now when you're done exploring
and done prototyping, maybe it's
time to work with an expert.
It's time to get serious.
So a sound designer can help you
come with an overall sound
concept and help you design the
sounds, and a sound engineer can
help you clean these sounds up
and make them ready for everyday
use.
Now we all know this guy, and
even I have to admit I use it
now and then.
Our goal is not for everyone to
leave their sound on all the
time, everywhere they go.
But when we start hearing more
and more wonderful notifications
and UI sounds that help our
users and delight our users,
then maybe, just maybe, you'll
start seeing people reach for it
a little less.
That will be awesome.
Thanks for listening to me
today.
[ Applause ]