Transcript
[ Music ]
>> All right.
Good morning, everyone.
My name is Jonathan Bennett and
I work with the Connected Media
team at Apple.
And I'm happy to be here today
helping kick off the conference,
talking with you about AirPlay.
So to start, let's talk about
what you can do with AirPlay.
You can use AirPlay to
wirelessly send a video to watch
on your TV or listen to music
throughout your home.
And you can also share pretty
much anything from your iOS or
macOS screen using AirPlay
Mirroring.
And so for today's talk, we're
going to focus on video and
we'll go over some of the
enhancements we've made to make
sure that you can find out about
all the great TV destinations
and AirPlay experiences on
AirPlay sending devices.
So when it comes to
destinations, Apple TV used to
be the only way that you could
AirPlay a video to a TV, and
it's still a great choice for
many people.
And as you heard earlier this
year, we've been working with
some of the largest TV
manufacturers to build AirPlay
support directly into the TVs.
This dramatically increases the
choices people have on where
they can AirPlay their video to,
and it adds a huge number of TVs
that you can now take advantage
of.
So just look for this label on
the box and you'll know that it
supports AirPlay Mirroring,
multi-room music and, of course,
video.
And this is a high-quality video
experience with both Apple TV
and AirPlay capable TVs able to
support up to 4K HDR video with
a surround sound.
So those are some of the great
new additions that we've made to
TVs.
And when it comes to AirPlay
sending devices, I'd like to go
over some of the enhancements
we've made to iOS to make it
easier than ever for people to
get the content they love from
your app to the TV.
So first, your video app can now
sort TV destinations to the top
of the AirPlay list, making it
easier than ever to find the TV
you want to play a video on.
We've also made enhancements and
enriched the remote controls
available to allow people to
interact with your video quickly
and easily.
And we've also made improvements
to allow people to better
multitask on their iOS devices
so they can receive a phone call
or check out that cute video
that they just got from an
iMessage while your video
continues playing on the TV.
And lastly, iOS can now
intelligently suggest to TV to
AirPlay to using on-device
learning.
So I can just sit back on the
couch, launch your video app,
and just tap on my favorite show
and have it automatically show
on the TV.
With great new TV destinations
and all of these experience
improvements, now is the time to
make sure that your video apps
fully support AirPlay.
So I'd like to take the rest of
this talk to go over five key
areas that you can add this
great experience.
First, we'll make sure that you
have high-quality video
available for the TV.
Then we'll provide a way for
people to get that video onto
the TV using an in-app picker.
And once it's on the TV, we'll
make sure that people can
control it and we'll make sure
that that video keeps on playing
while they use their iOS device
for other things.
And lastly, if you have an app
that plays primarily long-form
video like movies or TV, there
are some additional steps that
you should take to make sure
that your users benefit from
intelligent AirPlay Suggestions.
So let's start by talking about
high-quality video.
HTTP Live Streaming is a fast
and adaptive way to stream
high-quality video to any
device.
And you're likely already using
HLS to deliver video to your iOS
apps.
Now, some of you might be
curating your playlist to focus
on those type of devices.
So, an iPhone screen or an iPad
screen.
But remember at any time, a user
could take that video and want
to send it and AirPlay it to a
TV.
So it's important that you have
a full range of variants in your
playlist that are appropriate
for a TV, including variants
that support 4K, HDR and
surround sound.
Now, if in your playlist you
have multiple video codecs that
you use to encode your content,
be sure to also provide a full
range of variants for each video
codec.
This makes sure that your video
is fully supported on any device
it plays back on and it also
makes sure that the player can
transition smoothly between
variants using the same video
codec.
Now, one thing that I find
myself doing frequently is fast
forwarding or seeking through
video.
And whether that's to skip a
scary part of a movie with--
that I'm watching with my kids
or to find that interesting
topic in the middle of a WWDC
video having fast, rapid, frame
previews while seeking allows me
to find what I want quickly and
confidently.
So you can add the support in
your videos by including an
I-frame variant to your HLS
playlist.
And lastly, if you have
high-value content that you need
to protect and manage, you're
likely using FairPlay Streaming
to do that.
And FairPlay Streaming is fully
supported whenever and wherever
you AirPlay, that includes
AirPlaying to an Apple TV.
And I'm also happy to say that
FairPlay Streaming is supported
on every smart TV with built-in
AirPlay support.
Now, there are more details
about validating your HL streams
for AirPlay than we can cover
here today, so we've made a
separate video associated with
this session and I encourage you
to check that out.
So, having high-quality content
is great but we also want to
make sure that we have smooth
playback.
Now, if you have pre, mid or
post-roll content that you need
to insert into your video, use
HLS's discontinuity tag to do
that.
This allows the player to remain
intact and the estimated
bandwidths to remain intact so
both your interstitial video
plays in high quality.
And when it ends and you resume
the main content, that content
can immediately begin in high
quality.
It's, again, important to make
sure that these interstitial
videos have the same video
formatting as the main content,
such as the same video codec and
this allows these transitions to
happen quickly and smoothly.
Now, if you can't use HLS
discontinuity tag for some
reason or you have multiple
pieces of content that you want
to play serially, we encourage
you to use AVQueuePlayer when
AirPlaying.
And if you can't use
AVQueuePlayer when AirPlaying,
please use the same AVPlayer
instance and just replace the
current item with your new
content.
You don't want to destroy an
AVPlayer and create a new one
for every piece of video or swap
between instances, because when
AirPlaying, the system doesn't
know that a new piece of video
is going to be arriving.
And so it might start tearing
down the engine and you don't
want to introduce this type of
delay just because you're
transitioning between videos.
All right, we've got
high-quality content.
Now, let's give a way for people
to get that to the TV.
And you can do that by adding an
AirPlay picker into your app.
And new in iOS 13, you can
decide how you want to order
those destinations.
To add an AirPlay picker, use
AVRoutePickerView.
It's been available since iOS 11
and it's now available on macOS.
Now, some of you might be using
MPVolumeView's route button
properties to get a picker in
your app.
And in iOS 13, the route button
properties are now deprecated.
Now, don't worry, you can still
use MPVolumeView for putting
volume UI into your app.
But when it comes to route
buttons, use RoutePickerView.
So let's see how you can add
that, and it's really simple.
All I have to do is create a
AVRoutePickerView and add it to
my view hierarchy.
And in iOS 13, I can set
prioritizesVideoDevices to true
and that will give me a route
button icon that is focused on
video with the AirPlay video
logo, and it also sorts those TV
destinations to the top making
it super easy to find that TV I
want to play to.
So now that I have video playing
on the TV, let's make sure that
people have a way to control it
and this is really important.
And so we've made sure that
remote controls are more
convenient and capable than ever
throughout the system.
For example, I can control my
video with lock screen controls
and we've added additional
capabilities when you've
targeted a TV.
I can also use Siri and issue
play commands and say, you know,
play this on my TV.
And it's not just about
controlling what's on your iOS
device but also the devices
around you.
For example, you can pull down
Control Center and you can
target HomePods that are playing
music throughout your home or
even a tvOS app that is playing
a video and I can see what's
playing and control it from
almost any device.
So, what happens if your app
doesn't support remote controls?
Well, here I have my app and
it's AirPlaying to the TV.
But if I were to lock the
screen, I wouldn't know that the
video is playing and, more
importantly, I would have no way
to interact with that video.
So by supporting remote
controls, you can know what's
playing, how far along I am in
that video, a great way to
interact with it quickly, and
then there's even a way to get
back to the app that's currently
playing that video by just
tapping the artwork or the
title.
So, let's go through how you can
add these remote controls in
your app and it starts with
becoming the Now Playing app.
You become the Now Playing app
by doing two things, one is to
at least support one remote
control command and the other is
to initiate playback with
immediate configuration.
And on iOS, that means
activating your audio session
with a non-mixable category.
And for most video apps, that's
going to be the playback
category.
On macOS and iOS code running on
macOS, you'll do something
slightly different.
You're going to grab the
MPNowPlayingInfoCenter and
change the playback state
whenever you start or end your
video experience.
When it comes to supporting
remote controls, if you're using
AVKit's AVPlayerViewController
to present your video, it
handles remote controls for you
so you're all set.
If you're using AVFoundation and
a custom UI to present your
video, you'll need to use
MPRemoteCommandCenter and
MPNowPlayingInfoCenter in order
to get equivalent functionality.
And so I'd like to spend some
time going through step by step
about how you use this media
player APIs and build out this
functionality.
So here I am back at my lonely
lock screen, and let's just add
some basic support by supporting
the play command.
I can do that by creating a
setup method for supporting
remote controls.
And we'll be coming back to this
method again and again as we
continue to build out our
support.
I first grab the shared instance
of the remote command center.
And the remote command center
supports many different types of
media commands.
Here, we're supporting the play
command.
And to that, we'll add a target
handler to receive that command
and do something with it.
So here, we'll start our
playback.
And once I've done that, I'll
return a status on whether that
command succeeded or failed.
And this is the basic structure
you're going to use for any
remote command that you support.
So to flash out our playback
command handling, we'll support
the pause command and the
togglePlayPauseCommand by adding
handlers for those.
So this got me a platter with a
play/pause button and I at least
know what app is playing, but
it'd be really great to know
what video is playing.
And you can do that by providing
Now Playing info.
And Now Playing info comes in
two parts, one is the item
metadata such as the artwork or
title and then the playback
state such as the duration and
the current elapsed time.
So in my code, there are two
major events where I'll want to
update this information.
One is when a item changes, so
going from episode one to
episode two.
The other is when I have a
current item and the playback
state changes so I-- my rate
changes or I seek to a new
position in the video.
So when an item changes, I'm
going to take a metadata that
I've packaged up in my app with
a structure just to pass it
around through my app, I'm going
to update that static Now
Playing info such as the title
and the artwork.
And if I have it, I'm going to
update the playback info at this
time as well.
Now, when a playback state
change happens, all I need to do
is update that playback info.
I don't need to update artwork
and spend time doing all of
that.
So let's see how you can update
some of this information.
And the first thing I'm going to
do is grab the Now Playing info
dictionary from the shared info
center.
And to this dictionary, I'm
going to update my metadata.
So here, I'm updating title and
artwork.
And there are many other types
of metadata that are supported
and that I might support in my
app.
Regardless of what I support,
it's important to update this
metadata or clear it out, nil it
out if a particular item doesn't
have that type of property.
You don't want a mix of old and
new metadata in the dictionary.
Once I've updated it, though, I
just pass it back to the info
center for presentation.
So, it's really simple to update
Now Playing information.
And now, let's see how we update
the playback info.
And it's done very similarly.
I grab the Now Playing
information, the info
dictionary, and to that, I'm
going to update the duration,
the elapse playback time, and
the playback rate.
And now, I only need to update
this on significant change
events.
I don't need to update this
every second to see progress
through the remote UIs.
As long as I update these three
items when a rate changes or I
have a significant jump, then
the system can infer where you
are in your video at any given
moment and update the remote UIs
accordingly.
And again, once I've updated
this, I'll just pass it back to
the info center to get
presented.
So I know what video is playing
and it's really nice to be able
to navigate through that video.
So you can add the support for a
seeking with a playhead scrubber
or skipping forward and backward
in your video.
So let's see how we do this.
So we're back in our method for
setting up remote control
commands.
And I'm going to add support for
the skip forward command.
And this is one of the commands
that supports additional
configuration, and here, I'm
setting the preferred skip
interval to 15 seconds.
I then add a handler for that
command.
And it's important to note here
that I'm using the interval
that's passed to me through the
commands event.
Just because I set a preferred
interval doesn't mean that's
going to be used every time.
For example, I could ask Siri to
jump forward by 42 seconds.
And that's the value that's
going to get passed to me and
the one that I should honor.
I'll then round out my support
by supporting the skip backward
command and the change playback
position command which is used
when seeking.
Now, for some apps and some
types of video, you might need
to temporarily disable
navigation and you can easily do
that by setting the isEnabled
property on the commands that
you need to disable.
You don't need to remove the
target handler and add it back.
So one thing new this year that
we've added to remote control
UIs is the ability to select and
change audio languages and
subtitles.
So if you have video with these
types of options, you're going
to want to add this type of
support for remote controlling.
This allows people to watch the
video the way they want and need
easily.
So you can add this support by
providing a list of audible and
legible language options that
represent the audio languages
and subtitles.
And you can also then identify
the current selections in each
of those groups.
And lastly, you're going to want
to handle changes to those
selections via the remote UIs.
So let's see how we provide this
information.
And it starts by creating a
mapping.
So I'm going to load the audible
and legible language groups from
my AVAsset.
And I'm going to create a
mapping of these by translating
them into the media player
objects that represent these
options.
And I can use convenience
functions to do that.
And I'll maintain this mapping
in my app so that when I receive
a language option from a remote
command, I can eventually
translate it into the
AVFoundation object and set it
on my AVPlayer item to change
the selection.
So once I have this mapping, I
can update that info.
And here, I'm doing that in the
update method for my playback
info, where I've already added
the playback duration and elapse
time.
And to that, I'm going to add
the list of available option
groups and their current
selections.
So now, we have to handle
changes to those selections.
And I can do that in my setup
method for remote controls or
I'll add support for the
enableLanguageOptionCommand.
And this handler is going to
first get the current playback
item.
And then use my mapping to take
the language option that I
received and translate it to the
AVFoundation objects.
I'll then use AVPlayer items,
select option and group method
to select the new option on the
current item.
And I'll want to update my Now
Playing info and other internal
state and handle that playback
change.
Now, there are some types of
option groups that could have
nothing selected.
For example, turning off
subtitles and closed captions.
And this comes through with the
disableLanguageOptionCommand.
So you want to make sure to
support both the enable and
disable commands.
And the handler is very similar,
first, getting the current item
and using my mapping to find the
group.
But now, I'm going to check if
that group allows empty
selection and make sure that it
can.
And if it does, I'm going to use
the same select method but
passing nil as the option and
this will deselect all the
options in that group.
So I've got a full set of remote
controls that I can support.
When do I register and
deregister for these?
For most video apps, you're
going to register when you have
content to play and that's when
a user selects a video in your
app.
And you also want to deregister
when you have nothing to play.
So, a user exits your video.
And you'll use that opportunity
to also clear out the Now
Playing info dictionary to make
sure that there's no still data
hanging around.
Now, I mentioned that it's not
just about controlling what's
playing from your device but the
devices around you.
So if you have a tvOS app that
you make, try playing some
content on Apple TV and pulling
down Control Center on an iPhone
and target that Apple TV.
See that you're seeing the video
information that you expect and
you're able to control it as you
expect.
The same goes for using Siri to
control your TV.
So the examples that I went
through just now are all part of
a sample project that we've made
available associated with this
session.
And I encourage you to check
that out.
All right.
We got high-quality content that
we can get on the TV and we can
control it.
When it's playing on the TV,
it's really nice to be able to
use my phone or my iPad for
other things.
And there are two parts to
supporting a great multitasking
experience.
One is to make sure that your
video keeps on playing while the
user multitasks.
And the other is to make sure
that if there's already video
playing on the TV that your app
and navigating through its UI
doesn't accidentally interrupt
that experience.
So, when I'm watching a video on
my phone, my video app is
usually staying in the
foreground because I need to
look at it on the screen.
But when I'm AirPlaying on the
TV, I'm now free to use my
iPhone for other things like
check my messages or mail, or
social media.
If I'm using a video app that
doesn't support background
media, the video is going to
stop the moment I leave that
app.
And so we want to make sure that
your video can continue playing.
And you can do this by
supporting the audio background
mode in your app's Info.plist.
And you'll also configure your
AVAudioSession for background
playback.
For most video apps, that means
using the playback category.
And this provides media
appropriate behaviors in your
app, including playing in the
background.
Another thing about the playback
category devices, it'll
interrupt the other media.
So it's really important to make
sure that you only activate your
audio session when a user
presses play and begins playback
on a primary piece of content in
your app.
Now, once you go active, you can
stay active.
There's no need to deactivate
your audio session just because
the user paused the video.
Now, media interruptions can
happen at any time.
And so, it's important to listen
to those notifications on
AVAudioSession.
And when an interruption begins,
you update your UI and internal
state.
And when it ends, check the
notifications payload and make
sure that you check the
shouldResume key and honor that
value.
So, only automatically resume
your video if it returns true.
When you have video already
playing on the TV, it's
important to test your app and
launching it, navigating to the
UI and make sure that there's
nothing that accidentally
AirPlays that you didn't expect,
or that it interrupts that
current video experience.
And I like to go over a couple
examples that you might run
into.
For one, there's sometimes video
that should never AirPlay.
So if I have an animation on app
launch that I achieved using a
video asset, you don't want that
interrupting the current
experience.
The same goes if I have video
assets I use for other types of
visual flourish or to promote
content within my app.
You can make sure that those
videos don't interrupt the
current experience and stay
local on the device by setting
allowsExternalPlayback to false.
Your app should only AirPlay a
video when a user has directly
interacted with that video.
And to that end, you should not
auto-play primary content just
because your app launched or
came into the foreground.
When it comes to media, people
should always be in control of
when playback begins.
And this is really important,
and I want to go over a couple
examples of why.
So here, I am listening to music
on my iPhone and I have a video
app that supports on-demand
video that I can watch but also
auto-plays a live TV channel
when the app comes to the
foreground.
So if I'm listening to music, I
launch that app, that music is
going to stop and I had no
control over it.
And I might have launched that
app for a purpose other than
watching that live video.
You want to make sure that
people can launch your app
confidently knowing that their
current experience won't get
interrupted.
And I like to call out one more
important example that is new
this year.
For long-form video apps, iOS
can suggest a TV to
automatically play to when the
video begins.
And when this happens, the
system provides a notification
telling the user that the video
will automatically play on the
TV.
And people can tap that
notification or the status bar
icon to change that suggested
destination.
Now, if I have an app that
auto-plays its content, it's
possible that that video starts
playing on the TV before I've
had a chance to really see and
interact with that notification.
By requiring to use the action
to begin playback, you can
provide people the power to
decide both when and where they
play their video, and that
creates a great media
experience.
And so, to talk with you more
about long-form video apps and
their integration with
intelligent AirPlay Suggestions,
I'd like to invite up on to the
stage Marty Pye.
[ Applause ]
>> Thank you, Jonathan.
Hello, everyone.
My name is Marty Pye.
I'm an AVKit engineer here at
Apple.
And I'm here today to tell you a
little bit more about the
advanced routing features we
built into iOS 13 and how your
long-form video apps can take
advantage of these features.
So first, we'll talk about what
a long-form video app actually
is.
Then I'll talk a bit more about
what these specific routing
behaviors are that we've built
for long-form video apps.
And finally, I'll show you how
to integrate these features into
your code.
So what is a long-form video
app?
As the name suggests, any app
that primarily plays long-form
video content is considered a
long-form video app.
This content includes things
such as movies or the TV type
content like TV shows, sports,
et cetera.
If your app primarily plays
content from social media or
other types of short news clips,
then it would not qualify as
long-form video.
We encourage you to assess your
app holistically to decide in
which category of movie it
belongs.
So once you've decided that your
app is long-form video, how will
the AirPlay routing behavior
differ for such apps?
Let's take a look at an example.
If-- Imagine you're watching a
documentary on banana slugs and
you're AirPlaying this content
from your iPhone to an Apple TV.
New in iOS 13, when your friend
sends you his latest sweet dirt
biking video, you can go ahead
and watch this short content
locally on your device without
interrupting the banana slug
documentary that's playing on
the TV.
Another feature we've built into
iOS 13 is AirPlay Suggestions.
If the user goes to play some
content and the AirPlay routing
system detects a nearby likely
AirPlay device, then the system
may prompt the user and suggest
to AirPlay to this device.
Or, depending on the confidence,
the system might also
automatically route playback to
that device.
So now that you know about these
features, how to take advantage
of them in your app?
Well, there's two things you
need to do.
And the first thing is
registering your app as a
long-form video app.
So how do we do this?
It's really very simple.
All you need to do is add the
AVInitialRouteSharingPolicy key
to your app's Info.plist and set
it to a long-form video.
By setting this on the plist, we
can ensure that your app behaves
like every other long-form video
app on the system before you
create any of your playback
objects and the moment it gets
launched.
Now, the second thing you need
to do in order to leverage the
AirPlay Suggestions is prepare
the routing system prior to
playback.
I'm first going to give you a
little conceptual overview of
how that works, and then we'll
dive into how to achieve it in
code.
So in your app, you most likely
have some sort of browsing UI
where users can scroll through
different pieces of content,
select the piece of content they
would like to play, and hit
Play.
Well, it's at this transition
from browsing to actual playback
where you should prepare the
AirPlay routing system.
And what happens when you
prepare the routing system?
Well, it may decide to
automatically play the content
locally on the device or route
it to an external screen, be it
via a likely nearby AirPlay
device, or if you have an
external screen wired into the
device.
However, depending on the level
of confidence, the system may
also prompt the user asking
whether they would like to play
to a nearby AirPlay device.
And then depending on the user's
choice, control will then be
handed back to your app with
information about where the
content is going to play.
So let's take a look at how to
do this in code.
When the user decides to hit
Play, you should call
prepareRouteSelectionForPlayback
on the AVAudioSession.
And this method takes a
completion handler.
Now, this completion handler
will be called right away if the
AirPlay can-- if the AirPlay
routing system can make an
automatic decision.
However, there may also be a
prompt shown asking the user
whether they would like to play
to this nearby AirPlay device.
If the user decides they would
like to play the content
locally, then your completion
handler will be called with
route selection type local.
Now, you can go ahead and
present your playback UI.
For example,
AVPlayerViewController, or you
can present your own custom
playback UI.
If the user instead decides they
would like to AirPlay the
content and they select the
living room, then your
completion handler will be
called with route selection type
external.
And here, you have the
opportunity to present some sort
of custom remote playback UI
with controls dedicated to
controlling something that's
playing on an external screen.
Lastly, don't forget to handle
the case that the user decides
to cancel playback.
Now, in order to test your app
and how it behaves in the
various different scenarios with
AirPlay Suggestions, we've built
some toggles into the developer
settings.
If you drill down to settings
developer, you'll find this new
entry, AirPlay Suggestions.
And in here, you have the choice
of three distinct modes.
The Default mode is just the
default system behavior.
You can also select between a
mode where the system will
always prompt the user and ask
whether a content should AirPlay
to a specific device.
Or you can force the system into
a mode where it will always
automatically route to a
specific device.
And you can select which device
this applies to in the nearby
list of routes below.
Once you've forced the system
into a specific mode, make sure
that you reset the media
services.
If you're concerned about
whether AirPlay Suggestions will
handle every possible use case
where your users might want to
AirPlay, then don't worry.
At the end of the day, you're
still fully in control when and
where things will AirPlay.
You can always select to change
which device you're AirPlaying
to via AVRoutePickerView inside
of your app via the controls
inside of AVPlayerViewController
or via control center.
These new features just make it
even easier for your users to
get your content onto the device
of their choosing.
We encourage you to check out
the associated sample code where
we demonstrate in a simple test
app how to integrate the
behaviors we've described today.
So what have we learned today?
We've learned how to integrate
an AirPlay picker into our app
via AVRoutePickerView.
We've learned how to handle and
register for remote controls
commands via
MPRemoteCommandCenter, and how
to populate system UI with
relevant Now Playing
information.
We've learned how to take
advantage of AirPlay
multitasking and how to register
your app as a long-form video
app.
And lastly, we've learned how to
prepare the routing system prior
to playback in order to fully
leverage AirPlay Suggestions.
With that, I would like to thank
you all for joining us on
today's session.
We can't wait to see these new
AirPlay features integrated into
your apps.
Enjoy the rest of the
conference.
[ Applause ]