Transcript
[ Music ]
[ Applause ]
>> Good morning and welcome to
the session, Live Screen
Broadcast with ReplayKit.
My name is Alexander and I'm
very excited to be here and talk
to you about ReplayKit concepts
and new features that we bring
to ReplayKit this year.
We've got lots to discuss so
let's get started.
ReplayKit is a framework that
allows you capture application
screen audio and microphone
content in real time or record
to a video file that the users
can later edit or share.
And ReplayKit also supports live
broadcast.
For application generating
content like games, ReplayKit
provides tools that allow them
to stream audio and visual
content using [inaudible] and
broadcast services.
And for the broadcast
applications ReplayKit provides
ability to receive that content
captured from other applications
or entire iOS and encode and
stream to the servers directly
from the same device.
ReplayKit delivers its content
in high quality with lowest
latency and very low performance
overhead and better usage.
It also safeguards the user's
privacy by asking for approval
before any recording or
broadcast begins, and also
showing a prominent indicator
whenever content is being
captured.
In this session today I'm going
to talk about the live broadcast
functionality of ReplayKit and
we're going to start with
overview and talk about
[inaudible] broadcast and iOS
system broadcast, and then we
are going to talk about the new
API called System Broadcast
Picker.
Now, this is an API that allows
you to initiate the broadcast
directly from your broadcast
application.
After that we're going to
discuss how you can actually
implement the broadcast
extensions and how that would
work for you with the new API
that [inaudible].
Also, we're going to talk about
the protecting of the content of
your applications from being
captured.
So let's start with overview of
the live broadcast.
As I said, ReplayKit allows you
to broadcast your app screen,
audio and audio content to
search a third-party broadcast
service directly from your iOS
or [inaudible], and on iOS you
can also provide voice or video
commentary using microphone or
camera.
And all this content is
absolutely secure and only
accessible to the servers that
you're using.
So when you're streaming your
game play to Mobcrush or
YouTube, share your screen in
WebEx video call.
Use TeamViewer to work as
customer support, or maybe
stream your drawing app on a
Facebook.
All of that is powered with
ReplayKit technology.
Originally, with the ReplayKit
live broadcast the user would be
in the app and would use the app
to start and stop the broadcast.
An app would do that by directly
communicating to a ReplayKit API
and ReplayKit itself would
involve the broadcast extensions
and begin providing it with
visuals and audio content from
that application, and the
extension would encode the media
and stream to the servers and
this is what we call In-App
Broadcast.
And what was new with ReplayKit
2 is that instead we have entire
system in a pack being
broadcasted.
And how this works is the user
would initiate that broadcast
from within the Control Center.
So this is where they start and
stop the broadcast.
And that begins a system wide
ReplayKit 2 broadcast session
and, thus, audio and video
samples, again, go to the
extension and upload it to the
servers.
And conceptually this is the
difference between the two.
And just to talk about the
In-App Broadcast in the
beginning.
In the In-App Broadcast this is
your app or your game that is
providing the content that is
captured live as you play in a
game and your game is calling
the ReplayKit API to start and
stop the broadcast.
And it also presents as
broadcast activity controller so
that the user would pick the
service.
And in line with that, the
broadcast application is
providing you the broadcast
extension which ReplayKit is
working on behalf of that game
to sign users into the service
and upload the broadcast content
live to their servers.
That was our original way of
using ReplayKit Live Broadcast
and we covered this before.
And in case your application
does need this fine [inaudible]
control for starting and
stopping the broadcast, for more
information please find the
session called Go Live with
ReplayKit from a couple of years
ago.
But really, what today's session
is about is what we call iOS
system broadcast.
Now, this is, again, the
broadcast of all your onscreen
activity and all your sounds,
and instead of being initiated
from within your application and
starting and stopping when the
app is paused, for example, this
session is started and stopped
within Control Center, and this
is systemwide and it runs
continuously as the user moves
from the home screen to the app
or as they move from one app to
another.
And all of this is built into
iOS 11 and above and this is
what was introduced last year as
ReplayKit 2.
So here's some game and you're
playing a game and you want to
start the live stream and
initiate the broadcast.
So you pull down the Control
Center and say you touch into
that screen recording button and
that will bring up the picture
for you to record your game
play, or if you tap on one of
the [inaudible] of broadcast
providers, it will stream out to
their service.
So once you have done that,
you're back to your game and now
you're streaming to that
service.
The broadcast extension receives
the media samples captured from
the screen and uploads the video
stream to its back end and the
viewers they have in their web
browsers, or however they want
to use that stream, are watching
your broadcast on their own
devices around the world.
This is how you start the iOS
system broadcast.
And now, what's interesting
about this kind of broadcast
again, is that, as I said, it
continues regardless of you
moving from one app to another.
So I'm broadcasting from my home
screen here and then my home
screen is what is being
broadcasted.
And then if I'm launching a
different app, that app is going
to be broadcasted now, and
that's really the core thing
now.
So the broadcast itself is a
systemwide experience and you
can go back to the Control
Center and stop it or you can
tap on the status bar at the top
and it will bring up the
controls to stop the broadcast.
And another thing is that if
next I ran as a landscape,
there's also a way to support
that and you can move between
landscape and portrait as you go
from one app to another.
So this was an iOS system
broadcast we enabled last year
and the feedback and adoption
we've seen since then was
amazing.
And our live broadcast API today
is used in many apps in
categories like communication,
streaming, house, help desk,
education and social.
But at the same time we had a
lot of feedback that people are
having hard time finding this
broadcast UI in the Control
Center so we wanted to make it
easy for you to edit right into
your app.
And this year our goal was to
allow you integrate iOS system
broadcast into your application,
and we are very excited to
announce System Broadcast
Picker.
So let me go back to my diagram.
So what we added now is sort of
a hybrid approach to last two
[inaudible] styles.
And in this hybrid approach we
edit up the ability for your
application to also be initiator
of iOS system broadcast.
So your app now can do the same
thing as what Control Center is
doing; start the broadcast and
then allow it to continue as you
move from one app to another and
then move to the home screen,
and so on.
So as the user starts a
broadcast from your application,
he can later go to the Control
Center and stop it from there or
vice versa; he could start from
the Control Center and then go
to your app and tap the button
that stops.
So again, the System Broadcast
Picker allows users to start the
iOS system broadcast without
leaving your application, and
this is just a simple button in
the view that we provide and
this is a new feature we are
enabling in iOS 12.
Here is example of that.
Here I have the sample app, Fox
2 and I edit app with button at
the top.
So if I hit this Broadcast
Picker button, that's going to
bring up the same UI as you
would have seen if you would go
to the Control Center.
So I pick out the broadcast
provider and start my broadcast.
Now I'm back in the app and now
I'm just broadcasting to that
provider and people around the
world can watch my live stream.
So how is this done?
Okay. There is a new class
called
RPSystemBroadcastPickerView and
it's just a subclass of UI View
and you can add it to your
application using the interface
builder as a view with a custom
class, or you can do it
programmatically and all you
need to do is just run it in
instance of
RPSystemBroadcastPickerView and
add it to your view hierarchy.
So this is a more simple way to
use the broadcast picker, but I
know that many of you in this
room broadcast service
developers.
And finally, with this new API
your users can start the
broadcast directly from your
application and there's no more
need to include all those
tutorials, how you can enable
the screen recording and
settings and how to find your
service and that [inaudible] in
the Control Center.
But, however, you would probably
like the Picker to show the
broadcast extension for just
your service, and ReplayKit
provides API that lets you do
exactly that.
And it is just a property of
that view and what you need to
do is just get the bundle
identifier from your broadcast
extension and assign to the
property of the view called
preferredExtension.
Here's a code example of how you
could do this.
You would probably like to set
the property right after
correcting the view, and the
only difference from that
example would be that you need
to replace the string called
com.your-app.broadcast.
extension.
It's a bundle idea of your
extension.
So here's a diagram, and I just
want to emphasize the point
about the picker view that you
create in your app.
So that view is just a shortcut
that brings up a system UI of
the broadcast picker, the same
UI that you can access using the
Control Center.
And the picker itself belongs to
the system and your application
does not own any part of the
broadcast state and also cannot
start or stop the system
broadcast programmatically.
So this is what you need to do
to add the iOS system broadcast
to your app.
There's a new API and it is
fairly simple and you could
really be done adopting it
before leaving this session.
And next we are going to walk
through the process for the
broadcast extension developers
and discuss how you can actually
implement the broadcast
extension and how things would
work for you if your app
provides both the extension and
the broadcast picker.
So let me come back to that
diagram.
We are sort of moving to the
broadcast extension site now and
we want to show you the path
assuming the system broadcast.
Let's start with talking about
the broadcast application and
the broadcast extension and what
each of them needs to do during
the system broadcast session.
So you install the broadcast
extension along with the
application that contains it.
So if you install Facebook, you
also install Facebook broadcast
extension.
And broadcast extension and the
application are separate
binaries and each runs in its
own process and those processes
are invoked directly by the user
and independently from each
other.
And typically you would use
application to allow user sign
in or sign up for your service.
And now for the apps that will
adopt the broadcast picker API,
you can also let user to type in
some message before posting this
particular broadcast to his
Facebook.
And the broadcast extension,
that's all the actual work when
everything required to start the
broadcast is already available.
So again, the extension does the
work of taking samples edited by
ReplayKit, the raw samples of
the audio from the app, video of
the screen and perhaps audio
from the microphone and encoding
starts and uploading to your
service using whatever
technology you prefer to use.
To help you get started
developing the new broadcast
services we provide the Xcode
template and just add the
extension to your project using
the template and you'll be ready
to begin.
So now let's look how you can
code all of this.
So when you create an extension
in your project using Xcode
template, you'll get a
SampleHandler class like this
and this is where you add your
code to handle different events
during the broadcast.
And also, you would implement
this function that handles
incoming audio and video
samples.
This is how the life cycle of
the broadcast usually looks
like.
So we have states called setup,
initialized, started, processing
and stopped.
Over on the left there's this
green square which corresponds
to the state when your broadcast
extension is not running yet and
at this point your app could,
for example, get the login
credentials from the user and
use a shared keychain to process
credentials to the extension.
Or again, here you could let the
user to type in that message for
the Facebook part and pass it to
the extension like one way or
another.
So once the user initiates a
broadcast, using the Control
Center or your application that
adopted the broadcast picker,
ReplayKit launches an extension
process and creates an instance
of your SampleHandler class.
And at that point you can
override the default initializer
of SampleHandler and do
something essential for your
broadcast and not directly
related to handling the media
samples.
So, for example, you could get
login credentials from this
keychain and establish a
broadcast session with your
servers.
So once the process is launched
and the instance of your
SampleHandler already exists,
the extension is notified that
ReplayKit will begin providing
it with audio and video samples
using the function called
broadcastStarted, and in this
function you'd probably like to
create your media engine or do
anything else that you need to
start receiving the samples in
real time and encoding them and
uploading to your servers.
So once a broadcast is running
now ReplayKit will provide raw
audio and video samples and your
extension needs to encode it and
upload it to your video service
using whatever technology you
prefer to use.
ReplayKit provides your
extension three types of the
samples; samples with video
captured from the screen,
samples with audio tapped from
the application and samples with
audio captured from the
microphone, and your extension
will encode all of this and
upload to the service.
And all of those are going to
the extension and are handled by
the function called
processSampleBuffer.
This function receives a single
CMSampleBuffer, as you see, and
the type of the buffer.
And what it needs to do is to
encode and upload the media
samples, and in this code
example we show you how you can
encode the video using the Video
Toolbox.
So we recommend using the Video
Toolbox as it provides hardware
accelerated encoding
capabilities, and that is
especially important for our use
case because extensions have
much lower memory limits
compared to their applications.
So once you're encoding and
streaming the video to the
broadcast servers the viewers
around the world can or could
watch your broadcast live, but
they need some way to find your
broadcast.
And say if a person likes
playing and watching Angry
Birds, there should be someplace
on your website or in your app
where he can find all the live
broadcasts with that game.
And in order to implement this
you need to add to your video
stream some information about
the game that is being
broadcasted now, and for that
ReplayKit provides API function
called broadcastAnnotatedWith
ApplicationInfo, and as soon as
a player starts his game Angry
Birds sends this example,
ReplayKit notifies your
extension that that has happened
and provides a dictionary with
details about the app that was
started.
And as an example, you can use
the key defined by ReplayKit
called RPApplicationInfoBundle
IdentifierKey and get the bundle
ID of that game and pass it as a
metadata to your stream.
When the user stops the
broadcast, one way or another,
ReplayKit is using function
called broadcastFinished to
communicate to your extension
that broadcast session is over
and there will be no more
samples delivered to the
SampleHandler.
And in this function you would
probably like to finish
uploading your buffered video
and [inaudible] your media
engine or anything else that you
need to release.
So this is how the broadcast
life cycle usually goes.
And there's one more point that
I would like to mention here.
As we discussed, everything
required to set up the broadcast
should happen in the application
and you would get the login
credentials using the app or you
could get the name for the
broadcast in the application,
and we have a state for this,
this green square on the left
side.
However, this could be the case
that when the broadcast has
started, your extension is
missing something to proceed as
a broadcast.
For example, the login process
could have failed or something
else is needed, and I want to
show you how you can handle this
case.
ReplayKit provides an API that
allows extension to communicate
to the user that broadcast
cannot be started and provide a
particular action required to
fix the problem.
And the way it works, the
extension calls some API
function and ReplayKit will
terminate the session and
present the user the alert that
will include the failure reason
provided by the extension, and
in this example this error
message is from Mobcrush and it
says that user is not logged in.
So next is the user tabs at
bottom go to the application,
ReplayKit is launching your
application and what you need to
do there is handle this case in
some way, present the UI by user
will sign in or sign up, and the
next time the broadcast will
just work.
So we have this code sample for
the broadcastStarted before, and
here I want to show you how do
you handle this case when user
is not logged in.
So there's a function called
finishBroadcastWithError, and
what you need to do is provide
the user info dictionary with a
failure reason and ReplayKit is
going to use that string and
allow that is shown after the
session is stopped.
Okay. This is how you develop
the broadcast extension, and as
you see, the process is pretty
straightforward.
The application handles the
broadcast setup and then
communicates that info to the
extension.
And after that the session
starts and you receive and
encode the media samples.
And also, you can get
information about application on
the screen so the viewers could
find your broadcast.
And so just one more point I
would like to discuss today.
This is protecting content of
your applications.
So your app may display some
content that you don't want to
be captured in screen recording
or live broadcasts.
That applies to both audio and
visual content.
There's an API that you can use
to tell if your content is being
captured and that API is
provided UIKit framework and I'm
talking about the property of
UIScreen called Captured.
So you can register to receive
notifications whenever the value
of this property is changed and
stop your audio playback or hide
the visual content when
recording starts.
There's also a special case that
you need to take care of, which
is Airplay Screen Mirroring, and
I would like to show you how you
could do this using this code
example for the notification
handler.
And basically, in the case of a
screen mirroring there's always
more than one screen and this
array called Screens is going to
have more than one element, and
even though that isCaptured is
going to return yes during the
screen mirroring session, we
have this extra check that will
allow your playback to continue.
So this is the Live Screen
Broadcast and how it really
works.
And to summarize, ReplayKit
provides high level API that
allows your broadcast content of
just one app or all your
onscreen activity.
In iOS 12 we provide you an API
or a way to initiate the system
broadcast directly from your
broadcast application.
And what you actually need to
code is a broadcast extension
that does all the actual work of
encoding the video.
And also, we just discussed the
way to protect the content of
your apps.
You can find more information on
the website of our talk and we
also will be having the
ReplayKit labs later today at
3:00 p.m. in the Technology Lab
5, so if you have any questions
or comments, please come.
We'd love to hear from you.
So thank you so much for being
here with us today.
We are really excited of
bringing you the new approach
for live stream broadcast in iOS
and looking forward to see test
applications adopting it.
So please go ahead, create your
first broadcast extension using
the Xcode template, or open your
existing project and add the
broadcast picker to your app and
enable live stream broadcast
from iOS to your platform today.
Thank you.
[ Applause ]