Transcript
[ Music ]
[ Applause ]
>> Hi, everyone.
My name is Neil Desai, and I'm a
watchOS Frameworks engineer, and
I'm really excited to be here to
talk about how to create some
great audio apps for watchOS.
And honestly, we've been waiting
a little over a year to give
this talk, so I'm really happy
to be here, so let's just jump
right in.
In watchOS 4, we unveiled a
revamped music and radio app.
And in watchOS 5, we unveiled a
brand-new podcast app.
And we're finding that users
absolutely love being able to
play audio on the go or in the
middle of a workout.
And we're also finding that
users really like to be able to
remote control some playback
that's occurring on another
device.
For example, your iPhone or
HomePod.
And I especially really like
this feature.
I love just being able to play
some music out of my iPhone and
then being able to control the
volume, for instance, directly
with my watch.
That's a great feature.
And today, with watchOS 5, we're
going to show you how to build a
great experience like you saw on
the keynote like Audible was
able to make and just like
Pandora was able to make.
And so they were able to use
some of the new tools available
in watchOS 5.
And today, we're going to talk
about how we can do the same
sort of thing.
And to do that, we're going to
talk about four major themes.
We're going to talk about our
native controls and how we can
now embed our system controls
directly within our application.
We're going to talk about, how
do we get content over to the
watch?
And then, once we have that
content, how do we actually play
something back for our user?
And lastly, we're going to go
over the whole audio experience
and how to build a really rich
experience for our users that
integrates well throughout the
entire system.
So let's talk about our native
controls.
In watchOS 5, we're unveiling
two new controls that you can
embed directly within your
application.
And the first is a Now Playing
view which you can embed within
your watchOS app.
And this is really great for
content that may not be yours,
for instance.
And you can also embed a volume
control so you can build your
own custom UI to allow your user
to control the volume.
And you might be wondering for
that first view, that Now
Playing view, when exactly would
I ever want to allow my user to
control some audio, but the
content's not exactly mine?
Well, let me give you an
example.
We do this ourselves in the
workout app.
So I'm in the middle of a HIIT
workout.
I swipe to the left.
I get my power song, "Eye of the
Tiger," and I'm a little bit of
a dork, so I actually do listen
to this song sometimes when I
work out.
I love it.
And this is a great experience.
You just swipe over, and you can
allow your user to control the
playback.
And so if you're building a
workout app, and you can do
this, build the same sort of
experience, just embedding the
Now Playing view directly within
your application.
Let's talk about the
characteristics of that Now
Playing view.
And the first thing I want to
note is that the Digital Crown
controls the volume.
And because of that, you're
going to want to place this view
in a non-scrolling controller.
So for instance, it's, in my app
here, I've actually just pushed
directly to a controller that
has the Now Playing view in it.
Or you could reload directly to
a page that has this controller
or place it in a page interface
similar to the workout app.
And this view is going to
automatically switch the sources
on behalf.
So no matter if it's the iPhone
or the watch that's providing
the source of audio playback,
this view's going to take care
of it for you.
You don't have to worry about
that.
And lastly, the Now Playing
view, it's going to grab your
application's tint color so it
fits within the look and feel of
your application.
Now, to use this, let's see how
it looks in Interface Builder.
Here we are in Xcode, and I've
selected my Object Library, and
then I can just drag and drop my
Now Playing view into a
controller.
Once my user navigates to that
view, everything will be done
for you.
The system will take care of it
all.
And it really is just that
simple.
So that's our Now Playing view.
Now, let's talk about our volume
control.
So the volume control can be set
up to either control the
iPhone's volume or the local
watch's volume.
And just like our Now Playing
view, we're going to grab your
application's tint color and
apply it appropriately to this
particular control.
And in the case of when the user
is actually scrolling with the
Digital Crown, then, in that
case, the volume control will be
tinted to the system green color
to match the rest of the system.
So for the volume control, to
actually decide when or not to
control the local or iPhone's
volume, there, in the Attributes
Inspector is a check box for
Controls Local Volume.
So if you leave it unchecked,
then the volume control will
control the iPhone's content.
And if you check it off, then
the volume control will control
the watch's content.
And so if your application might
have multiple scenarios where it
could either remote control some
content or play some content
locally, then, in that case, you
might have different controls
with different volume controls
within them, and you make those
decisions at design time.
All right, so those are really
our system controls, so now
let's talk about, how do we
actually get content over to the
device itself?
So I've been building a museum
guided tour type application.
I want to allow my user to just
tap in, listen to a guided tour
while they're in the museum, and
it's a great experience on the
watch because I can just put my
phone away.
I can just have the entire
experience on my wrist and enjoy
the beautiful art around me.
And let's say I had a, like a
buffer that allowed my user to
download the content on, when
they're on their way to the
museum.
In that case, I would want to
use URLSession and grab that
directly from my server, all my
content, and download it to the
watch itself.
So let's jump into some code and
see how we could do that.
So here we are.
I have an IBAction which is tied
to my WK Interface button.
And the first thing you'll
notice is I set my frontmost
timeout extended property to be
true.
And we're going to talk a little
bit more about frontmost later,
but the thing I wanted to
mention is that the user has
made a user interaction, so it's
fair to say that the user, next
time they raise their wrist,
would want to see our
application.
And because we're extending the
frontmost timeout, then we're
getting some additional priority
by the system for our
URLSessions.
And again, I'll talk a little
bit more about that later.
And the user can, of course, put
their wrist down at any point,
so I want to make sure to use a
backgroundSession.
So that way, when my application
goes into the background, my
download is still occurring.
And then, I can just download
that task and appropriately
resume it.
And when you're building an
experience like this, you want
to indicate download progress
and, of course, handle any
errors.
So let's look at how to do that.
So here we are.
I have my class, my
DownloadManager, which is my
SessionTaskDelegate and my
URLSessionDownload Delegate.
And in the first function, I can
actually grab the total bytes
written and divide it by the
total bytes expected to write.
So that way, I can get the
progress of my particular
download whenever URLSession
calls me and gives me that
information.
And when I do this, I can just
update progress for my user in
my UI.
And then, when the download
completes, I can, of course,
manage the file, and I might
want to post a notification
alerting my user that there's a
playback opportunity.
And, of course, I'm also going
to be called into a
didCompleteWithError, so I just
want to clean up appropriately
and handle any errors at that
point, if there are any.
And that's wonderful,
URLSession, for when we can
download directly to the watch,
but sometimes the content might
already exist on our iPhone.
In that case, we can just use
watch connectivity and transfer
the file directly over to our
watch.
And to do so, we'll just use the
transfer file API.
And new in watchOS 5, you can
get the progress of that
transfer.
So let's take a look.
Here we are in code, and I see
my WCSession FileTransfer.
And new, there's a progress
property that's available in
watchOS 5 and iOS 12.
And then, I can just get my
fileTransfer, query for the
progress, and update my view on
iOS.
And if you want to know some
more about watch connectivity.
There's some great sample code
that's available online.
Highly recommend you check it
out, the simple watch
connectivity sample code.
And you might be wondering --
there's URLSession, there's
watch connectivity -- which one
do we use, exactly?
And when does it make sense?
Well, for URLSession, we want to
use that when it's user
initiated on the watch itself.
And then, when we do that, we
want a query the waits for
connectivity property to know
that we have a network
connection at that point.
And I just wanted to mention
that if the iPhone is nearby,
then requests are proxied
through the iPhone when it's in
range.
And if you want to know some of
the nuances of URLSession,
there's a great talk from last
year that's available online,
Advances in Networking, Part 2.
I definitely recommend checking
it out.
And then, on the flip side,
there's watch connectivity, and
that's really important when
your user flow might be when
it's initiated on the iPhone
itself or in the case where
you've already downloaded some
content and there's no need to
request it from your server
again.
And for both and especially for
watch connectivity, you're going
to want to set expectations for
your user.
Make sure to instruct them of
when the download, or the
transfer's most likely to
complete.
And that's going to be when the
watch is on the magnetic
charger.
And for example, we do this
ourselves.
In the Apple Watch app in the
Music pane, at the top, we
actually show our user how many
songs have actually been
transferred over to the watch
itself, and we even instruct our
user, hey, music's going to sync
when the Apple Watch is on its
charger, so keep that in mind.
All right, so we talked about
getting content, and now that we
have it, let's talk about
actually playing something back.
And on watchOS, we have a bunch
of different tools we can use.
And some of those audio tools
are in WatchKit itself.
So there's a presentMedia
PlayerController with options,
completion that you can use.
And there's also WKAudioFile
QueuePlayer.
And so you can give it a list of
items, and then the queue player
will appropriately play back
that content.
So the presentMedia
PlayerController option will
present the system UI.
So the controls are taken care
of for you.
There's not really much you have
to do.
And this is perfect for when
it's short-form content and you
don't really want to have to
think too much about it.
And WKAudioFIleQueuePlayer is
great for when you have more
long-form content that you want
to play in the background.
And this is more of a conveyer
belt type approach where you
have some items, you just want
to hand it over to the system,
and the system will play it on
your behalf.
And that's exactly what
WKAudioFileQueuePlayer is for.
And there's also AVFoundation at
our disposal.
So there's AVAudioPlayer and
AVAudioEngine.
And prior to watchOS 5, it was
tied to the workout background
mode, or when your application
was foreground and screen on.
For example, Nike Run Club app
actually did this exact thing,
where you're in the middle of a
run, you want some mindfulness,
you can play some guided
meditation with Headspace.
And prior to watchOS 5,
background playback was, with
AVFoundation, was really only
possible for these types of
workout apps.
But new in watchOS 5, there's a
brand-new audio playback
background mode, which I'm
really thrilled for.
So let's talk about what we have
now at our disposal.
Kind of broadened our tool kit.
And so we have AVFoundation, and
it doesn't have to be tied to
any other background mode, and
we can now use just AVFoundation
directly in the background.
We also are exposing the ability
to allow you to, allow your user
to pick a different route.
So maybe some different types of
Bluetooth headphones.
And new in watchOS 5, we're also
exposing the MediaPlayer.
framework, so you can provide
some Now Playing information and
also handle any media remote
commands.
And just like our own Music,
Podcasts, and Radio app, the
long-form content is restricted
to Bluetooth routes, which
really is just a fancy way of
saying you can't play it through
the speaker.
So let's take it step by step
and talk about how we actually
get this background long-form
content to play.
So the first thing, you add
audio to your info.plist
background modes.
And then, you just appropriately
set up your session, and you're
going to want to tell the system
that you want to play this
long-form content in the
background.
You want to allow your user to
select a different route.
And then, you're just going to
want to call play and
appropriately handle the
playback when the user's in the
middle of that playback session.
So let's break that more, break
that down more a little bit.
And so in Xcode in the Project
Settings, in the Capabilities
tab, you just flip the switch
for Background Mode, you check
off Audio, and you're good to go
to the next step.
And so to set up your session,
you're going to want to set your
routeSharingPolicy to longform
on your AVAudioSession.
And then, you call the new
activate withOptions completion
method on AVAudioSession.
And then, once it completes,
you're going to call play.
And in between when you first
call the activate function and
then when the completion comes
back, what we call the route
picker is going to get displayed
on your behalf.
And so this is what the route
picker looks like.
And so this is going to be
showing up within your
application, which is going to
allow your user to pick
whichever Bluetooth headphones
route they want to have and
control their playback.
So let's talk a little bit more
about that route picker.
Again, all it is, you just call
your asynchronous activate
function, which is going to call
and show your route picker, and
then it's going to come back
into your completion, and you
handle that, and you just call
play.
And you're going to want to do
this every time you want
playback to occur.
So that way, you can ensure that
you have appropriate route to
cause some playback to occur.
All right, so let's jump into
code and set up our session.
And so we have our
AVAudioSession, and then we're
going to set our category to
playback.
And then, we're going to set our
mode to default.
And then, most importantly,
we're going to set our route
sharing policy to longform.
And then, you want to activate
your session, which is going to
display the route picker, and
then you're going to call play
appropriately.
And now, for that route picker,
I wanted to call out that we
really think of routes in two
different ways.
And the first way is our Apple
wireless chip-based routes,
which are commonly referred to
as our W1 routes.
So these are our AirPods or
Beats Studio Wireless.
And then, we just have our
normal Bluetooth headphone
routes.
And it's important to make that
distinction when we talk about
the route picker itself.
And so if the user has an active
connected route, then the route
picker's going to automatically
select that route on your
behalf.
What's great is that the route
picker might not even display to
your user.
You're just going to get called
back in your completion
immediately.
And in the case where the user
has an active W1 route but it's
being used by the iPhone, then
the system's going to grab that
route from iPhone and bring it
directly to your watch.
And there's some cases where
that won't happen.
It's really just when your
iPhone has more priority.
So for example, say I'm in the
middle of a phone call on my
phone.
Then, we're not going to take
that route and put it on your
watch because phone call's
probably pretty important.
And if there are no active
routes, then the route picker
will just appear on your behalf.
All right, so that's our route
picker, and now let's talk
about, how do we actually get
some playback to occur?
Well, you just use the same
AVAudioPlayer and AVAudioEngine.
And there's a variety of
different formats that are
supported on watchOS.
And what's great about this is
if you have iOS code that uses
these classes, then you could
have a shared framework between
your iOS and watchOS playback
operations.
And in the case of
AVAudioEngine, you can actually
play some DRM content in
conjunction with AVAudioPlayer
node.
And so you can now play your own
DRM content, decrypt it
yourself, and play it back for
your user.
An especially important thing to
note for watchOS is the power of
the device itself, so only play
audio when absolutely necessary,
when the user actually wants the
audio to occur.
And for AVAudioEngine, the
autoShutdownEnabled property is
enabled by default, so if there
are no active nodes on your
AVAudioEngine, then we're going
to automatically power down your
AVAudioEngine when necessary.
And so that's our playback.
And then, we, let's talk about
our MediaPlayer.
framework.
So we want to, of course,
provide our Now Playing
information to the system.
And then, the Now Playing UI
will update, and that's the
system-wide notion of what's
playing, and it's going to be
updated based on the information
you provide.
So this is going to show up in
the Now Playing app, and even
inside apps such as Workout, or
in the cases of any third-party
apps that embed that Now Playing
view.
This will also show up there.
And then, with the MediaPlayer.
framework, we just want to
handle the events, any type of
media remote commands -- for
example, a Next Track on
AirPods, for instance.
All right, let's jump into some
code and see how we can provide
that Now Playing information.
So here we are.
We have our MPNowPlaying
InfoCenter.
We just want to grab that.
And then, we set up our
nowPlayingInfo.
And then, we have, we want to
set our artwork, our MPMedia
ItemArtwork.
And it's not currently available
in seed 1, but it's coming in
seed 2.
And in this case, the artwork
that you provide will display on
the Siri watch face.
And then, we just set up, in
this case, our title, album,
artwork, and then we set it to
our nowPlaying InfoCenter.
And then, when we do something
such as this, then our data will
appear in the Now Playing app.
And this is the actual Now
Playing app.
And I provided some, a song that
I was currently listening to in
my application.
And when we do this, then, also,
the Now Playing complication
will also get updated.
So in this case, the bottom
shows the "Lights in the Sky"
song that I was listening to.
And then, for media remote
commands, again, just use the
MediaPlayer.
framework.
And now, in watchOS, you can
handle those commands however
you wish.
So let's look at some code.
So we just grab our
MPRemoteCommandCenter.
And then, instead of doing a
Next Track, let's talk about how
we can actually skip forward for
our user.
So I have this function.
It's a enableSkipForwardCommand
and, just by default, takes in
15 seconds.
And I set my preferred intervals
on my skip forward command.
And then, I just add a target
and selector, and then I just
enable the command within my
remote command center.
If you want to know more about
the media player framework and
especially these media remote
commands, I highly recommend
checking out the MP remote
command sample that's available
online.
And in this case, when I
actually set my preferred
intervals, then the system's Now
Playing app will appropriately
get adjusted.
That Now Playing UI, wherever it
is, will show the minus 15
seconds or plus 15 seconds, and
that's all done on your behalf.
All right, so that's the audio
experience on watch, or that's
our playback, so let's talk
about our audio experience.
So there's 4 major things I want
to talk about first for the
audio experience.
There's our auto-launching
feature where your app will
automatically get launched.
There's the frontmost app state
and how we can take advantage of
that in our application.
There's our new interactive
notifications.
And how to integrate with Siri
Shortcuts.
And so we've been talking a lot
about playback occurring on the
watch itself, but I want to take
a step back, and I just wanted
to talk through that remote
control case.
So let's say I was playing some
Apple Music on my iPhone, and
automatically the Now Playing
app will get displayed on my
watch.
And I love this feature.
It feels like magic when I just
look at my wrist and I can
easily control my playback.
I, it's the coolest thing.
And in the case where it's our
application that's actually
causing the playback to occur on
iPhone, then our app
automatically gets launched on
the watch.
And we call this the auto-launch
audio apps feature.
And so whenever the Now Playing
session is occurring on iPhone,
then our Apple Watch app will be
brought frontmost at that time.
And we're going to stay
frontmost for the duration of
that session.
So as long as the user is
listening to content on iPhone,
then our app will stay
frontmost, unless the user
decides to navigate away.
And new in watchOS 5 is we're
exposing a Now Playing session
API where you're going to know
when you're launched for this
Now Playing session on iPhone,
and so you can take your user
directly to the appropriate
view.
And to do this, you're going to
use the handleRemoteNow
PlayingActivity on your
WKExtension Delegate.
So let's take a look at some
code and see how we could use
this.
So here we are.
I have my extension delegate,
and then I can implement my
handleRemoteNow PlayingActivity.
And the first thing I want to
check is, where is my user at
that time?
So I can just grab the visible
interface controller, I can
check where my user is, and if
the user already is at my Now
Playing UI, that's great.
Let's leave them right there.
But in the case they're not, we
can just reload directly to our
view.
So that way, we have that same
magical experience where I raise
my wrists and I see the remote
controls that I want to use.
And in some cases, you might
want to opt out of this feature
if it doesn't make sense for
your user.
So just do the right thing.
Make that decision.
And let's say our app has
nothing to do with our phone
app, and that's an appropriate
use of actually opting out of
this capability.
And so to do so, use the auto,
the opt out of auto-launch
feature on your WK extension's
info.plist.
And in the case you do opt out,
then the Now Playing app will
show instead.
And now, let's talk about that
frontmost app state, and let's
give a little bit of a refresher
of that.
So here I have my app.
It's foreground running.
I'm looking at it.
The screen is on, and this app
is considered active and
frontmost.
But then, let's say I put my
wrist down.
In that case, the application is
still considered frontmost
because if I raise my wrist
again, I'll see that
application.
And in that case, the
application is just in the
background.
So with the frontmost app state,
we get some enhanced
capabilities from the system.
Our background transfers from
iPhones -- so say we were
transferring our content from
iPhone to watch itself, those
background transfers, when they
complete, the resumes will
directly just wake up our app if
we're in the frontmost app
state.
And the same thing in, with our
URL session resumes.
So if the URL session, our
background session before was
downloading and it completes,
then our app will automatically
get waked up and deliver the
payload.
And if we're frontmost, we're
going to get called into, so we
can decide whenever, if a
notification appears, how we
properly want to handle that
notification.
Maybe we want to keep our user
in the app, or maybe we want to
show the notification to our
user.
And in the frontmost app state,
we can make that decision.
And when you're in the frontmost
app state, you can play some
haptics.
And if you want to know a little
bit more about frontmost app
state, there's the talk from
last year, The Life of a watchOS
App.
It's a great talk.
You might recognize the speaker.
He's okay.
And so you're going to stay
frontmost for 2 minutes, but, if
you recall, I extended my
frontmost earlier.
And so in that case, it's just
going to be extended by default
to 8 minutes.
And so, again, this is great for
when we were downloading that
content from before, and so I
can just extend my time out.
And when we're actually playing
back some audio, our app will be
kept frontmost for the duration
of that session, just like when
we were actually remote
controlling the audio when it
was occurring on iPhone.
And again, if the user ever
navigates away, then you're no
longer frontmost, but your
background playback will still
occur appropriately.
And in that case, you can, of
course, properly handle all of
your background events.
All right.
So now, we're -- and when we are
in that frontmost state, then we
also can integrate properly with
notification.
So we could post, for example,
when the download completes, a
content available local
notification.
And we might want to have play
as our primary action.
And with our new interaction
notifications on watchOS, we can
allow our user to actually
configure some playback right
from within the notification and
then be able to call play as the
primary action, then get into
your app and immediately cause
some playback to occur.
So for example, say I had a
Salvador Dali exhibit.
It's ready to go.
I can then adjust the playback
speed, immediately call play,
that'll launch my app, and then
I can play that great content at
the appropriate rate for my
user.
If you want to know more about
some interactive notifications,
check out the What's New in
watchOS talk given earlier,
which is available online.
It was a great talk, and you'll
know a lot more about
interactive notifications after
watching it.
All right, so do you remember
this code from before where I
actually posted that
notification alerting our user
of this type of playback
opportunity?
Well, let's say our app was
frontmost at that time.
In that case, when I got
delivered the notification from
frontmost in user notification,
I'm going to get called into and
will present notification with
completion handler.
And in that case, I can actually
decide whether or not I want to
still post that notification.
Or maybe because my app's
frontmost, I could just play a
haptic and let my user know
that, hey, raise your wrist, and
there's some new content.
I can update my view.
And again, if you want to know
some more about the frontmost
notifications, check out the
Life of a watchOS App.
Great, so that's how we used our
interactive notifications and
the frontmost app state, but we
also want to show up on the Siri
watch face.
And with audio apps, you can
provide suggestions of audio to
play on the Siri watch face
using relevant shortcuts.
So for example, that could be
the next episode of your user's
favorite podcast or the new
music that you just synced over
to the watch.
For example, Audible was able to
build the same experience where
they can just get their user
directly within their
application from the Siri watch
face and resume the playback of
an audiobook.
And whenever a user interaction
is occurring, just wanted to
highlight, think through your
donations.
Make sure to donate your INMedia
PlaybackIntent to the system so
that way the system knows what
your user is commonly doing on
the watch.
And that way, you can become
more relevant on the Siri watch
face.
And also, just think through
your shortcut phrases like
design appropriate phrases that
your user could use within Siri
to get your user directly within
your application to allow for
some playback to occur.
And all in all, on the Siri
watch face, design a great
experience.
You might have some glanceable
information like, hey, there's
an exhibit coming soon.
Or you could have some tappable
actions like Audible was able to
make where, hey, we just want to
play that audio tour and get our
user directly in the app.
And so for shortcuts, donate
your INMediaPlaybackIntent.
Use the relevant shortcut API to
appear on the Siri watch face.
And think through your shortcut
phrases.
And there's been some great
talks this week about shortcuts,
and I definitely wanted to
highlight the Shortcuts on the
Siri Watch Face talk.
I saw that myself.
It was a great talk, and I
learned a lot.
And so in summary, for the first
time, you can embed your native
controls directly from within
your application.
You can allow background
playback to occur however you
want and handle media remote
commands however you want.
And there are better ways that
transfer audio assets to your
watch and convey that type of
progress to your users.
So if you want some more
information, feel free to go
online or join us in the lab
this afternoon.
I'm really excited to see what
type of audio apps are built on
watchOS, and thanks, everyone.
Have a great WWDC.
Thanks.
[ Applause ]