WWDC2016 Session 601

Transcript

[ Music ]
>> Good morning.
Welcome everyone to Go
Live with ReplayKit.
In this session, we
are going to tell you
about all the new features that
we've added to the framework.
I am Ben Harry, a
software engineer
on the Game Technologies
Team here at Apple.
And later, you'll be
hearing from Edwin Iskander.
Now we know people love
to record their gameplay
and share those recordings
on social media.
ReplayKit gives our
users those abilities
with little effort required
from you, the developer.
To get started today, I am going
to take a few minutes to talk
about the existing features
that are offered with ReplayKit.
ReplayKit provides the ability
to record your app's
audio and visual content.
to record your app's
audio and visual content.
In addition, you may
enable microphone recording.
By doing so, your users can
provide voice commentary while
playing their games.
After a user trays a recording,
they may share the
recording directly
from their game using
the System Share sheet.
ReplayKit has a simple API, and
as you're going to see today,
a small amount of
code is required
to bring these features
to your games.
Moving along, ReplayKit will
generate HD-quality videos
with little impact on
your game's performance,
and uses a minimal amount of
power from the device's battery.
Here at Apple, user privacy
is very important to us,
so we have implemented
some privacy safeguards.
For example, we present a user
prompt before recording begins,
so user's can give permission
before recording their gameplay.
Furthermore, recording
will exclude system UI,
including notifications.
Therefore, when you receive
a text message while you're
playing a game, that message
is not going to be recorded.
playing a game, that message
is not going to be recorded.
Finally, ReplayKit was
made available in iOS 9.
So this is a list of
the current features
that we offer with ReplayKit.
Now, I am going to take a
minute to list the new features
that have been added
to ReplayKit framework.
First up, we are bringing
ReplayKit support to Apple TV.
So now, you'll be able
to record your gameplay
and share it directly
from your TV OS games.
The next new feature we've
added is live broadcasting.
Now, users will be able
to broadcast for gameplay
in real-time to third
party broadcast services.
And for our third and final
feature, we're adding support
to record the Face Time camera.
In addition, we've
made enhancements
to the microphone recording API.
So before I get into
ReplayKit and Apple TV,
I'm going to take a minute
to give you an overview
of the ReplayKit architecture.
I want to show you how your app
uses the framework to interact
with the operating system
to create a recording.
with the operating system
to create a recording.
So here we have your
application.
Your application needs to
notify the operating system
when it wants to
begin recording.
So ReplayKit provides an
RPScreenRecorder class.
You will tell this class
to start recording.
At this moment, a message is
sent to the Replay Daemon.
The Replay Daemon will start
writing your app's data
to a movie file.
When you tell the
RPScreenRecorder
to stop recording, the Replay
Daemon will finish writing your
movie, and now we have
a movie on the system.
Now this movie needs to get
back up to your application
so your users can preview
the recording they just made.
To do so, we provide the
RPpreviewViewController.
You present this view
controller in your game,
so now your users
have an interface
where they can preview,
they can edit
and they can share the
video that they just made.
As we just saw, we have
the RPScreenRecorder class.
As we just saw, we have
the RPScreenRecorder class.
This is the class you
will use to start,
stop and to discard recordings.
You will also use it to
check the availability
to record on the device.
There is a delegate property
on the RPScreenRecorder,
and it will notify you if the
availability to record changes.
Also it will let you know of
recording stops due to an error.
The second class I brought up
was the RPPreviewController.
You present this view
controller in your game
to give your users ability to
preview the recording on iOS,
and to share that recording
directly from the game.
We also have a delegate
method on this class,
and this delegate will let you
know when the user is finished
with the preview user interface.
So this is the reflected
architecture
and the classes you will use
to record and share content
on both iOS and tvOS,
which now brings me
to ReplayKit and Apple TV.
And to begin I'm going
to show you a demo.
So here I have a
game called Fox.
Fox was developed
for WWDC, 2015.
It was developed using SceneKit.
You may also recognize it
because it is available
as sample code on our
developer website.
The goal of this game
is to walk on the level,
and collect these flowers
like the one I just got.
I'm going to make my
way through the level,
and collect the flowers.
There's three on the
level, and I got the first.
You can optionally collect
these pearls along the way,
but I'm just going to
focus on the flowers.
I got two.
I know the third flower is on
the top, and is in the middle,
on the topo of the rock.
So I'm over here, so
here I'm going to take,
I'm going to bring
up my game menu,
because I want to record this.
There is an obstacle
at the head,
and I've been practicing
this a lot this past week,
so I wanted to share this
with my friends to show them
that I can get through
this most of the time now.
So I'm going to start
my recording,
and here is the consent
prompt I mentioned.
and here is the consent
prompt I mentioned.
And I am going to
give permission.
So now I'm recording.
So notice the recording
indicator
at the top of the screen.
This lets me know that our
recording is taking place.
Okay, so I made it
through the flier.
So all I have to do now
is make my way to the end,
to do the final flower,
so I can finish the level,
and ultimately finish
the recording.
Okay so that's it.
I'm going to stop my recording.
Now, once I'm stopped, I have
the option where I can preview
or share the video I
just made, so I'm going
to start off by previewing.
So we have a video playback,
and we have a timeline we can
scrub forward and backward.
I'm just going to
start playing it.
Notice here that recording
indicator is not included
in the recording.
I'm going to show you
how this was accomplished
in just a few minutes.
So I have a video, I'm happy
with how it turned out.
So now I want to share it.
So I'm going to go over
here and select share.
And now we are presented
with an AirDrop interface.
So I'm going to grab my phone.
I'm going to select my phone.
Anyway, when I do accept
the file on my phone
and the file is transferred,
once it's on my phone,
I do have the ability where
I can preview the video,
I can edit and trim the
video if I wanted to,
and finally I can share it to
my favorite social media sites.
So that was how ReplayKit
looks on Apple TV.
I now want to take a few moments
to call out four main points
from the demo, and then I'm
going to show you the code
that corresponds with each
of those points in a minute.
So first I'm going to begin.
I'm going to have
the end game menu
where I started my recording.
Then we have the gameplay where
we have the recording indicator
on screen, back to the game
menu to stop the recording,
on screen, back to the game
menu to stop the recording,
and finally when I stop the
recording I have the options
to preview or share
the recorded video.
So I'm going to walk
through each one of these,
and show you the code that
corresponds to each one.
We began or start our recording.
So when I press my
start recording button,
first I'm going to grab
the shared instance
of the RPScreeningRecorder.
And I simply am going to
tell it to start recording.
Now once I'm recording,
I'm going to present
the indicator view.
As I mentioned in the video,
the indicator was not
included in the recording.
This is because ReplayKit will
only record the application's
main window.
Therefore, I'm going to
create a new UI window.
And then the indicator view.
The indicator view will be
your responsibility to create.
This is because now you can make
it match the style of your game.
So we have an indicator view,
I'm going to instantiate an
instance of that indicator view,
and simply add it as a sub view
to that main window I created,
to the window I created above.
Now the indicator
view is on screen,
Now the indicator
view is on screen,
and we know we're recording.
When we are done recording, we
press the stop record button.
Again, I'm going to
grab the shared instance
of the RPScreenRecorder.
Now notice we return the
preview view controller,
which is the RPPreviewController
class I mentioned earlier.
I'm going to hide
my indicator view.
And here it's important that
we hold onto a reference
of the preview view controller,
because we are going to use it
to present in the next
step for previewing
and sharing the video.
Finally I'm going
to set the delegate.
So we saw that we have
one preview controller,
but we have two interfaces,
one for previewing,
and one for sharing.
So at Apple TV, we have
introduced a new mode property.
When we want a preview, we
simply set the mode to preview,
and then present
the view controller.
Similarly, to share, we
have the mode property.
We are going to set the
mode to share, and again,
present the view controller.
Now we will be presented
with the AirDrop interface.
Now we will be presented
with the AirDrop interface.
In both cases, when the user
is done with each interface,
we have a delegate method
that is going to be called.
Preview Controller did finish.
It's important when this
is called that you just--
that you dismiss the
preview controller,
since your app was
responsible for presenting it.
Finally, when we are done
with the recording and we know
that we no longer need
it, it's good practice
to discard that recording.
Now ReplayKit will automatically
discard the previous recording
when a new recording begins.
This is because one recording
is allowed at a time per app.
You could also explicitly
discard the recording
when you know that the preview
will no longer be available.
For example, maybe at
the end of a level,
there could be no
more opportunity
to present a preview.
So now you can explicitly
discard it
by calling RPScreenRecorder's
discard recording method.
Through these last few slides,
I've shown you the small amount
of code that is required
to bring these features
to your tvOS games.
to bring these features
to your tvOS games.
I encourage you all to add
ReplayKit to your tvOS apps,
and also to your iOS apps, if
you have not done so already.
So now, let me take a minute
to summarize ReplayKit
and Apple TV.
Now, with ReplayKit
and Apple TV,
you can record your app
audio and video content.
Under Apple TV, the microphone
is reserved by the system,
so you cannot provide voice
commentary on the platform.
But you can allow users
to preview their videos,
and they can share their videos
directly from their games.
As you saw, we have
a very simple API.
In fact, it's the same
API that we offer for iOS.
All these features
for Apple TV are going
to be available new in tvOS 10.
So that was ReplayKit
and Apple TV.
So now it's time to move on to
the second of our new features,
which I personally am
very excited about.
So, at this time, please welcome
Edwin Iskandar up to the stage.
[ Applause ]
>> Alright, thanks guys.
Hey, how is everyone doing?
My name is Edwin Iskandar.
I am a Software Engineer
on the Game Technologies
Team here at Apple.
And like Ben, I'm super
excited to talk to you guys
about our next feature,
Live Broadcast.
So with Live Broadcast,
players will be able
to broadcast gameplay to
third party streaming services
directly from their
iOS or tvOS device.
So this is really exciting,
because this is the first time
that this will be possible
without any additional hardware
or third party SDKs, will
also be allowing your users
to take advantage of their
device's powerful Face Time
camera and microphone to
provide rich commentary all
in real-time.
And finally, we are going to
do this all securely and ensure
that all audio and video data
is only accessible to the system
and the broadcast service.
Alright, so to illustrate how
this works, we've been working
with flair games to implement
the broadcast feature,
and the flagship
game, Olympus Rising.
So it's a really awesome game.
It's available in
the app store today.
And it has this unique
blend of strategy,
RPG and action elements.
Okay, now that we have a
game to broadcast from,
we also need somewhere
to broadcast to.
So we've also been working with
Mob Crush, a broadcast service
that specializes
in mobile games.
Alright, so now that
we have a game
and a service, we're
ready to roll.
So let's see this in action.
Alright so in Olympus Rising,
the Flair developers have
implemented an end game button
to start broadcasting.
To initiate a broadcast,
the user taps this button,
and are pesented with
broadcast services
that they have pre-installed
on their device.
In this particular example,
the user has installed
the Mob Crush app,
so it appears on this list.
So at this point, the user
taps on the Mob Crush icon,
and they are presented
with Mob Crush's UI
to set up the broadcast.
The user then goes ahead and
adds a title for the broadcast,
and once they're done,
they simply press
on the start broadcast button,
which returns to the game,
on the start broadcast button,
which returns to the game,
which then presents a countdown,
so that the player can get
ready, and finally, go live.
And now that they're live,
and the user is playing,
video and audio data is streamed
to the broadcast service.
Spectators all around the world
can now launch the Mob Crush app
from their device, and watch
the gameplay in near real time.
Within the Mob Crush app,
spectators can also discuss
the gameplay as they watch.
And even cooler, broadcasters
can even be instantly notified
when this happens while
playing their game.
So it's just like in sports,
there is something
really special
about watching an event live.
Now, this will be
possible natively with iOS
and tvOS games as well.
Alright so now that we've
seen the entire flow,
as a game developer, you're
probably wondering what it takes
to implement this
feature in your games?
Here is the player
flow broken down.
As you can see, there
are many steps.
Initiating the broadcast,
selecting a broadcast service,
setting up the broadcast,
starting
setting up the broadcast,
starting
and stopping the broadcast,
indicating a broadcast is
in progress, and also uploading
that video and audio data
to the back-end servers.
So this might look a little
bit daunting at first,
but the good news for
game developers is three
of these steps are
handled by ReplayKit
and the broadcast service.
So let's take a look at the
code for each of these steps.
So to initiate a
broadcast programmatically,
we use a new class
called RPBroadcastActivity
ViewController, and
call its load method
to get an instance of it.
We then simply present it as you
would any UI view controller,
which will then present
the user with a list
of broadcast services, and
ultimately allow the user
to set up the broadcast.
Just before we do that, though,
we also set the
ActivityViewController's
delegate method.
Because we want to be notified
when the setup is complete.
At this point, the user
can select the service
to broadcast to, sets
up the broadcast,
and once we you done setting up,
the activity view controller's
delegate method is fired,
and this delegate method
is supplied an incidence
of another new class,
RPBroadcastController,
which allows us to
start a broadcast.
But before we do that,
though, we also want
to dismiss the
ActivityViewController,
since we are the ones
that presented it.
Start our in game
countdown timer,
and when the countdown
timer ends,
we finally start the broadcast
by calling Start Broadcast
on the new RPBroadcastController
instance, that was passed in.
And now we are live.
And since we're live
broadcasting,
we need to clearly
indicate this to the user.
So Olympus Rising
does this very well
by animating the
broadcast button.
Since the game has
fairly complex controls,
they're reusing the button
UI to also indicate an
in progress broadcast,
maximizing the available
screen space.
Something to note is that the
indicator is strictly required
during broadcasting and will
be enforced during app review
before you add this
to your games.
So programmatically check
if broadcast is in progress,
simply query is broadcasting
prompt
of the broadcast controller.
You can use the value of
this property to start
or stop animating
your indicator UI.
To allow users to
finish broadcasting,
the Flair developers have
implemented a fly out UI
that reveals a sub menu
which includes a stop button.
When the player presses
on the stop button,
we simply call finish
broadcast on the controller.
When broadcasting is
finished, we update our UI.
Alright, now that we
covered the basic flow,
let's look at a few more
details, like error handling.
So since there are a lot of
moving parts to this feature,
it's possible that
things could go wrong.
So it is important to
handle this gracefully.
And so let's take a look at how
we do this during a broadcast.
To handle errors,
simply set the delegate
to the broadcast controller.
Once set, your delegate
will now have its did finish
with error method called
and when this is triggered,
with error method called
and when this is triggered,
you should let the
user know about it
and update any necessary UI.
So now another detail
to consider is
when the user backgrounds
the app,
or the app is interrupted
during a broadcast,
like when receiving a
phone call, or example.
So when the application
goes to the background,
ReplayKit will automatically
pause the broadcast.
So in this particular example,
when the application is
reactivated and comes back
to the foreground, we prompt
the user and ask if they would
like to resume the broadcast.
If the user wants
to resume, call,
the resume broadcast
method, if they don't,
simply call the finish
broadcast method.
And that's it.
For the entire flow, the game
developer needs to only interact
with two classes,
and its delegates.
RPBroadcastActivity
ViewController,
to present broadcast services,
and allow users to set
up a broadcast, and
RPBroadcastController to start,
pause, resume and
finish a broadcast.
As you can see, it couldn't
be easier to add broadcasting
into your games, and
I urge you to do so,
into your games, and
I urge you to do so,
because the benefits are huge.
There is really no better
way to spread the word,
increase player engagement
and even build a
community around your games.
So that covers game
implementation,
but what about the
broadcast services
that players are streaming
to, like Mob Crush?
Let's take a moment
to talk briefly
about what those services
are responsible for.
So for all the game
developers in the audience,
or watching online, I just
want to make it extra clear
that the topics discussed
in the next section are
not your responsibility
to implement in your games.
So let's go back to
that flow diagram.
We've seen that the
gamer is responsible
for initiating the
broadcast, and controlling
when to start and
stop a broadcast.
We have also seen that the UI
for broadcast selection
is taken care
of by ReplayKit's
BroadcastActivityViewController.
So this leaves us with
two remaining tasks.
Setting up the broadcast, and
uploading video and audio data
to the back-up servers.
So to allow broadcast services
to accomplish these steps,
So to allow broadcast services
to accomplish these steps,
we're introducing a new pair
of app extensions
specifically for this feature.
A UI extension, to allow the
user to set up a broadcast,
and a non-UI extension
for processing
and uploading the
audio and video data.
For those of you
unfamiliar with extensions,
they're target embedded in your
parent application and are a way
to extend your app so that
you can execute alongside
other apps.
They run in a separate
process from your parent app,
but can share data with
your parent application,
which can be handy
or sharing things
like authentication
data, for example.
So one thing to keep
in mind, though,
is that extensions are limited
in resources compared
to applications.
So try to refrain from doing
resource intensive tasks
within them.
We made it incredibly
easy to set
up these extensions
via an exotemplate.
These are available as new
targets for both iOS and tvOS.
If created from this template,
the extensions will be
pre-configured so they will show
the extensions will be
pre-configured so they will show
up in the broadcast
activity control list.
So let's take a look at
the two new extensions
that a broadcast service
needs to implement,
starting with the
broadcast UI extension.
UI extension has a few
key responsibilities,
first it's responsible for
authenticating the user,
and providing sign up if the
user is not yet registered.
Ideally, this should all be
done within the extension
so that the user's
experience is not interrupted.
But is also perfectly
acceptable to link
up to the parent application
as long as the user has a way
to get back to the game.
So, during sign up
it is also required
that the user be shown
terms and conditions
for using the service, and be
provided the ability to accept
or decline those
terms and conditions.
So as we saw earlier, UI
extension is also responsible
for allowing the user to
set up their broadcasts,
like adding a title to it,
and it can also allow the user
to notify others
that broadcasting is
about to begin via social media.
Finally, its final
responsibility is notifying
Finally, its final
responsibility is notifying
ReplayKit, and ultimately
the game
that broadcast set
up is complete.
So now that we looked at the
UI extension, let's switch
over to the upload extension.
It is responsible for
receiving and processing video
and audio data, and
uploading that data
to the back-end servers.
So we are aware there are
many different implementations
of data processing and
uploading to live streaming.
So I won't be getting into any
specific implementations during
this session.
Instead, if you are a broadcast
service, please get in touch
with us, so we can
work directly with you.
This way we can implement
a solution
that delivers the best
experience to our customers.
So going back to our flow
diagram, now we have looked
at the broadcast
set up and upload.
We have gone through the
entire broadcast slow.
And we've clearly broken
down the responsibilities
of the game, ReplayKit
and the broadcast service,
and that was live broadcasting.
We really feel this is
a game changing feature
that you players
are going to love,
and for developers it is
going to introduce a new way
for your fans to experience your
games, not only by playing them,
but by watching them as well.
So before we go, I'd like to end
with a few new enhancements
we made to the API
around commentary for iOS.
Namely, support for the
front-facing camera,
and also a new capability
for the microphone.
So first up, we've added
Face Time camera support,
making it incredibly convenient
for you to add picture
in picture video
commentary to your games.
Player reaction is
priceless, and always adds
that special flavor to
recordings and broadcasts.
So to enable this
in Olympus Rising,
the Flair developers have added
a button to toggle the camera
in their fly out menu.
Pressing this button enables
the camera displaying a preview
at the top left.
So let's look at the API
they used to achieve this.
To toggle the camera on,
we simply set the
IsCameraEnabled property to true
on RPScreenRecorder, this
turns on the camera hardware,
on RPScreenRecorder, this
turns on the camera hardware,
and populates a new
RPScreenRecorder called
CameraPreviewView, which
is a subclass of UI view.
Since it is UI view, you can
simply add it as a sub view
to your game view, and
even freely position it
so that it does not
obstruct gameplay.
You can even attach a Adjust
or Recognize It to it,
and allows you to position
it manually if you wanted.
So looking at the code here,
we're simply setting the
IsCameraEnabled property
to True during recording
of broadcast.
Then we get a CameraPreviewView
instance, then set its geometry
to position it, and
then simply add it
as a sub view to your game view.
So now that we have video
commentary, we also need
to pair that with audio.
So microphone recording was
included last year in iOS 9,
but now we are including
the ability
to mute it during
recordings as well.
So we all know broadcasters
always have something
interesting to say,
but sometimes even
they need a break.
but sometimes even
they need a break.
So Olympus Rising
makes this really easy
by including a microphone
toggle in their fly out menu.
Implementing the action
to microphone toggle
couldn't be simpler.
The existing IsMicrophoneEnabled
property
on RPScreenRecorder can
be set to true if you want
to enable the microphone, and
false when you want to mute it.
Again this can all be
done during a broadcast
or recording session.
Alright so that concludes
our session,
and I really hope
you enjoyed it.
With the powerful new broadcast
feature and now support
for Apple TV, we
can't wait to play
and watch your games live
on both iOS and tvOS.
For more information,
visit Developer.apple.com,
session 601, and during the
week, also make sure to check
out our other game
technology sessions,
what's new in GameplayKit,
SpriteKit and Game Center.
Okay, so that's it for me.
Thanks so much for attending
this session and I hope
that you enjoy the rest
of your week here at WWDC.
that you enjoy the rest
of your week here at WWDC.
Thanks so much.
[ Applause ]