WWDC2013 Session 606

Transcript

X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[ Silence ]
>> Good afternoon.
[ Applause ]
Well my name is Sam Bushell
and I work on Media Frameworks.
And that's what we're
going to be talking
about in this session.
We're going to talk about
Media Frameworks, old and new.
We're going to talk
about QuickTime and QTKit
and we're going to talk about
AV Foundation and AV Kit.
In case you haven't heard,
AV Foundation is a new media
infrastructure we've been
working on at Apple
for the last few years.
It's common on iOS and OS X.
And it's focused on modern media
formats like H.264 and AAC.
But AV Foundation was
built by engineers
with deep media experience
from building QuickTime
over many years.
Let's give a bit of
history about QuickTime.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
QuickTime was a pioneering
framework
for dealing with digital media.
There were some early developer
seeds but it first shipped
to the public in December 1991.
And over the years since
1991, there have been a lot
of QuickTime updates delivering
an enormous collection
of features.
One of our biggest individual
feature releases was QuickTime 7
which is shipped as part
of Mac OS 10.4 Tiger.
This introduced big
deal features like H.264
and the QTKit Framework.
The QTKit Framework
introduced an objective C API
that wrapped the QuickTime C API
for easier integration
into Cocoa Apps.
Four years later, we introduced
Mac OS 10.6 Snow Leopard.
And in that four years,
we had built a new media
playback pipeline as part
of building the iPhone.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We called this Core Media.
Now we weren't ready
to deliver a public API
for Core Media at the time.
But we delivered-- we
added a mode to QTKit
where you could use the
Core Media playback pipeline
to get optimized
playback of H.264 and AAC.
We called this QuickTime X.
And then in OS 10.7 Lion,
we introduced AV
Foundation as a public API.
In our framework hierarchy
AV Foundation sits lower
than the UI frameworks
of UIKit and AppKit.
This means that we can
deliver the same API
across iOS and OS X.
In the OS 10.8 Mountain Lion, we
enhanced the AV Foundation API
and we introduced the Video
Toolbox as a public API.
And this year in Mavericks,
(I'm still getting
used to saying that.)
We introduced-- we're
introducing AV Kit.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
AV Kit is the framework
where we will put APIs
that let you integrate AV
Foundation with AppKit.
So, you can see in the last
few years we have built
in a new stack of
media frameworks.
We call this the AV
Foundation family
and this is the direction
we're headed.
This is our focus in
the media systems group.
We have not been adding
new APIs to QuickTime
or QTKit for-- for
some time now.
I said that with one small
asterisk to come back to later.
So, as of Mavericks, the
QuickTime C Framework
and the QTKit Framework
are deprecated.
What this means is that
we've marked these APIs
as deprecated in
the header files.
And that means while your code
will still compile you will get
deprecation warnings like these.
But your apps will still run.
Let's emphasize that.
Your apps will still run.
While were on the topic,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
the QuickTime Movie file format
is still the primary file format
for AV Foundation and
for QuickTime Player.
It's still supported.
What we are deprecating
is the QuickTime C API
and the QuickTime, I'm sorry,
the QTKit objective C API.
The file format is
still supported.
And also on topic the movie file
format's ISO cousin MPEG4 is
also still supported.
So AV Foundation is the future
of media applications on OS X.
Let's talk a little bit about
how we built AV Foundation.
As I said, AV Foundation
was built by engineers
with deep media experience
from building QuickTime
over many years.
We wanted to build a
new platform for media
that could do things that
QuickTime couldn't do
and go places that QuickTime
couldn't do-- couldn't go.
So we built it on the same
kinds of modern foundations
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that the rest of Apple has been
moving towards, core foundation
and foundation, GCD and
Blocks and so forth.
Now QuickTime was built
on a modern foundation.
It's just that it was a
modern foundation in 1991
when System 7 was new.
So a lot of the technologies
that it was sitting on top
of have or ones that we have
moved apart, moved away from.
Where QuickTime's APIs--
QuickTime is often said
to have a lot of APIs
but if you count these up
by numbers and if you look
into the header files, a lot
of these APIs are simply
exposing the implementation
of QuickTime.
And sometimes that's a
good fit for how you want
to extend QuickTime and use
QuickTime but sometimes its not.
With AV Foundation we have
taken care to design our APIs
to be a good fit for how
clients will use them.
And so QuickTime's APIs
are monolithic and in some
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of these cases we have taken the
opportunity to re-factor them
in AV Foundation into
multiple objects.
In some cases these makes
the AV Foundation APIs much
more flexible.
Owing the QuickTime's
pre-OS X heritage,
many of its APIs
were only designed
to work on the main thread.
Now we did some retrofitting.
We did some refitting
and make many
of the APIs multi-thread
savvy in later years.
But that left the overall
rules if you had coming to it
as a first timer the overall
rules were somewhat awkward
and complicated.
AV Foundation is designed
to be deeply multithreaded.
There are two goals for this.
One is to take maximum advantage
of multi-core hardware
like we have today.
And the other is to improve
application responsiveness
by letting you take
slow, blocking operations
and move them off
the main thread.
AV Foundation is able
to take advantage
of hardware acceleration in
ways that QuickTime could not
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
such as video encoding.
And we made major design
decisions in AV Foundation
with the goal of
power efficiency.
Remember this is a framework
that was first delivered
on the iPhone.
It is a device that we tested
and run on a battery that fits
in your pocket and still
delivered long hours
of playback time.
Some of these power efficiency
design decisions could not have
been made compatible with
the QuickTime architecture.
And finally QuickTime's
integration,
deep tight integration
with legacy frameworks,
QuickDraw in particular,
means that it cannot
escape the 32-bit world.
Now you may know
that you are able
to use QTKit a 64-bit
application.
But for playback and editing
at least, what's happening
when you do that is that it's
running a 32-bit background
process to use QuickTime.
AV Foundation has designed--
been designed to be
64-bit native from day one.
So do you get the message here?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
There are lots of great
reasons why AV Foundation is the
direction that we're headed
and why it should
also be the direction
that your apps are
headed as well.
And AV Foundation supports the
media types that matter, video,
audio, closed captions
and subtitles,
chapters, and time code.
Now as I mentioned QuickTime
has a history, a vast history
with many features
added over the years.
And some of these were
breakthrough features
when they were added
in the mid 1990s.
But the world has moved on.
If you look at this list you
would think to yourself many
of theses APIs-- many of these
features would be superseded
by basic things that we take
for granted today like HTML 5.
And HTTP live streaming has
proved to be much more scalable
for delivery than RTP
Streaming ever was.
That brings us to codecs.
So QuickTime has a vast history.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
It has collected many, many
codecs over its long history,
and many of these are rather
old and have been superseded.
The categories of codecs
that are still relevant today
basically are in three groups.
Those that are used for
delivery like H.264 and AAC
and we use JPEG for
chapter images.
Codecs that are used
for production workflows
and editing, these are often
called Mezzanine codecs.
And Codecs that are used
as-- for import from captures
and device, standard
formats that we need
to be able to import from.
These three categories of
codecs are still supported
by AV Foundation.
And that leaves a number
that are left by the wayside.
And if you have really good
eyesight, you might be able
to look at this and
say well hang on some
of those aren't really what I
think it as video codecs anyhow.
And the way that QuickTime's
architecture was structured
still image formats and video
filters and effects also had
to be registered
as video decoders.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Well nowadays, we have
the image IO Framework
which delivers still
image format support
and we have designed
different ways
of integrating video
filters and effects
into the playback pipeline.
But our media is
personally important to us.
Some of the media in these
formats is irreplaceable.
And just because the
formats are out of date,
it doesn't mean we want
to orphan the content.
So we've provided a
mechanisms starting in Mavericks
to help you-- to help
you migrate the content
in these legacy containers,
legacy formats
into AV Foundation-supported
formats.
It's called QTMovieModernizer.
It's automatically run
by QuickTime Player
when it discovers a legacy codec
in a movie file you're opening.
It works with third-party
QuickTime components.
And it's provided as
a new API in Mavericks
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
so you can do the
same in your apps.
The way it works is it
produces a new copy of the movie
in an AV Foundation
supported format.
Normally this will
be H.264 in AAC.
But there are some cases
for production workflows
where you'd want to use
Apple ProRes and PCM instead.
If the content has
an alpha channel,
then the alpha channel can be
preserved in Apple ProRes 4444.
Now, it's delivered
as part of QTKit.
And this is something
we don't do very often.
We're adding a new API to
a deprecated framework.
But this is intentional because
there's a very specific message
we want to send here.
QTMovieModernizer will be
available to you exactly as long
as QTKit and those legacy
codecs are still available.
So now is a good time to
gather up your legacy media
and bring it across the bridge.
[ Pause ]
So that's our story here
about QuickTime and QTKit.
QuickTime and QTKit
are deprecated
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and we have been building a new
media framework brick-by-brick,
foundation by foundation
as it were called the AV--
the AV Foundation family.
In the rest of this talk,
we're going to talk about how
to migrate existing applications
to the AV Foundation
framework family.
And first I'll introduce
Stefan Hafeneger up to talk
about AV kit and introduce
it to you for the first time.
[ Applause ]
>> Thanks Sam.
So I see most of you sitting in
the audience here have some kind
of media playback in
your applications.
Let's see.
How many of you still
use QT Movie View?
Hands up. OK, so there's a few.
And how many of you
already use AV Foundation?
OK, so a few more.
And who of you has ever written
their own playback controls
on top of AVPlayer layer?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
OK, a few.
Yeah, I have written
a few as well.
Wouldn't it be nice
if it didn't have to--
didn't have to deal with that?
Here's something
new for all of you.
We want to make your life
as developers easier.
And that's why we are
adding AV Kit to our stand.
AV Kit is a new high
level Cocoa Framework.
Our goal is to provide
you view-level services
for media operations on top
of AV Foundation and AppKit.
AV Kit shows the AVPlayerView.
AVPlayerView is an
NSView subclass
for displaying audio-visual
media.
It comes with a set
of standardized playback
controls and behaviors.
This means, you can get
the same look and feel
that we have been using
in QuickTime Player
since Snow Leopard, now
in your applications.
And as a bonus, it takes care of
of localization, accessibility,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
high resolutions, state
restoration and so on.
And we made it really easy
for you to adopt AVPlayerView
in combination with the AV
Foundation in your applications.
So let me walk you through the
necessary steps in the demo.
All right, so in Xcode
you create a new project.
And select OS X Cocoa
Application and press Next.
The name is going to be
AVKitPlayer and we make sure
that we created a
document-based application.
Press Next and save
the document or the--
the project on the desktop.
Let me make the Window
a bit bigger here.
[ Pause ]
So the first thing
that you have to do is
to add the AV Foundation and AV
Kit Frameworks to our project.
So I'm here on the linked
frameworks and libraries.
I click on the Add button.
And then in the New sheet,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
we set first search
for AV Foundation.
Oops. I select that, press Add
and then we do the
same for AV Kit.
Now go up with the
mouse to the Info tab.
And on the document types,
first thing we do is we
remove the MyDoc extension.
Instead for identifier, we
use public.audiovisual-content
so that we can open all kinds
of audiovisual media
in our application.
Finally, we switch the
row from editor to viewer.
So that's-- NSApplication
doesn't create a document
for us automatically.
Now on the left we select
the documents xib file.
We remove the label
and then on the right,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
we search for AVPlayerView.
We're dragging an
instance of AVPlayerView
onto our document
window and resize it
so it fits the entire window.
In the inspector, we
change the control style
from default to floating.
And this would give us
the nice floating HUD
that we have in QuickTime
player.
Next, we show the
assistant editor.
And here, switch to the
document header file.
And now with the Control-click
we can add an outlet
to our document.
We call it playerView
and hit Connect.
Xcode will now complain that
AVPlayerView is not declared.
So, what's still left to do
is to add the header files.
All right, so what
have you done so far?
First, we create an OS
X document application.
Second, we aadd the
AV Foundation,
AV Kit frameworks
to our project.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Third, we'd set a-- sorry,
modify the info plist
so we can open all kinds
of audiovisual media.
And fourth, we drop an
AVPlayerView into our document
and create outlets so we can
reference it from the documents.
So let's see how much
more we have to do
to actually make it work.
[ Pause ]
So I close the assistant editor
and select the document
implementation
and we scroll all the
way down to the bottom.
We're not going to
use readFromData,
instead we implement
readFromURL.
And here we just return yes.
We want to use AV Foundation
to open the document for us.
So we scroll up to
windowControllerDidLoadNib
and implement [playerView
setPlayer:
[AVPlayer playerWithURL:
[self fileURL]]] And now,
press on Build and Run.
[ Pause ]
Go to File, Open.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Select the movie, press Open.
[Background Music] We have a
fully functional document-based
application for playing videos.
OK and it's- [applause]
thank you.
Press the Play and Pause
button to Play and Pause.
I can use this Slider
to scrub around
and we use the Scan button
to scan forward and backward.
But you can also use the
keyboards, for example space,
just stops playback
and stops playback.
We can step with
the arrow keys frame
by frame, forward and backward.
And for those of
you who are familiar
with the J/K/L navigation
in other applications
you have this as well.
[ Pause ]
All right.
Wasn't that easy?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So let's go one step further.
Let's assume your users have
some content where they want
to cut off the beginning
and/or the end of the movie,
something we call trim.
Let's see how much
work we have to do
to add these features
to our application.
So for that, I'm going to open
another version of this project.
[ Pause ]
As you can see, I already set
up a trim menu item
in the Edit menu.
So we now open the
document implementation.
The only thing we have to
do is, in the trim IBAction
to call our playerView,
beginTrimming
WithCompletionHandler
and just NULL.
But as a good Cocoa Application,
we also want to enable
and disable the trim menu item.
So for that, in
validateUserInterfaceItem
for the trim IBAction,
we implement for trim--
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
playerView canBeginTrimming.
If we now build and run, and we
open the same document as before
and go to Edit and trim, you
get the same user interface
in the QuickTime player.
So we can direct
the trim handlers
to select the shorter portion of
the video, hit the Trim button
and now we're just playing
[Background Music] that--
this portion of the video.
[ Pause ]
All right.
So two lines of code-- sorry.
Two lines of code
was all we had to do
to create a very basic
video playback application,
using AVPlayerView
and AV Foundation.
Two more lines of code and we
have trimming working as well.
Just imagine how long
it would have taken us
to create these controls
from scratch.
All this time, you can now
invest instead making your
applications even better.
[ Pause ]
So let's talk about
a few details now.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So how does AVPlayerView work
with the AV Foundation
class hierarchy?
AVPlayerView has a strong
reference to an AVPlayer object.
This object provides
the content of course.
AVPlayer itself mentions
AVPlayerItem which serves
at mid-- which serves as
the mutable data structure
for an immutable AVAsset.
This means in order to provide
content for an AVPlayerView,
you have to do the
following steps.
First, you create an
AVAsset from an NSURL.
Once you have an AVAsset, you
can create an AVPlayerItem.
We have AVPlayerItem, you can
then create an AVPlayer object.
And finally, you associate
this AVPlayer object
with an AVPlayerView.
However, if you really just
want to play the content
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of the movie file on disc, you
can do all four steps at once.
[ Pause ]
So as we showed you
earlier in the demo,
you can directly create an
AVPlayer object from an NSURL
and then pass this object
to the AVPlayerView.
Let's go back to the object
graph from earlier for a second.
I told you, the first step
in order to provide content
from AVPlayerView is to create
an AVAsset from an NSURL.
But what if you're
using AVCompositions
in your applications instead?
All I'm showing today
that AVPlayerView
would just work fine
for AVCompositions as well.
[ Pause ]
AVPlayerView let's you choose
from four different
standardized control styles.
Which one you pick for your
applications is up to you.
It really depends on the
type of application you have
and what looks best in
your user interface.
Let me walk through the
difference real quick.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The first style doesn't
show any controls,
but instead it gives you all
the gesture and keyboard events
that AVPlayerView implements.
The second style has the
controls at bottom of the view.
This is the closest match that
we provide to the QTMovieView.
The third control
style has controls
in this-- in the floating HUD.
This is exactly the same UI
that we are using
QuickTime Player today.
The last style is just
a Play/Pause button
at the center of the view.
That will also show a
circular progress indicator
during playback.
All controls automatically show
and hide upon user interaction.
You can change the control
style in interface--
builder or in code at anytime.
For the second and
third control style,
we also have Trim controls.
The screenshot you can see
on the left shows the TrimUI
for the floating
controlsStyle and the screenshot
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
on the right shows the TrimUI
for what we call inline
Trim controlsStyle.
As you can see, the only
difference is the margin
around the controls and the
graying on the background.
[ Pause ]
AVPlayerView has
dynamic controls.
This means that if your
content has chapters,
additional language
or subtitles,
it will automatically display--
adds chapter and media selection
pop-up buttons to the UI.
Some of you might have streaming
content in your applications.
AVPlayerView will automatically
switch to a different set
of playback controls
when the content type
of the current AVPlayerItem
changes to streaming.
AVPlayerView has also
API for customization.
For instance, if you
want to allow your users
to share their content
from-- with an AVPlayerView,
all you need to do is to
set one property to yes.
AVPlayerView add the share--
the OS X Share Button
to the UI takes care
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of everything for you.
This includes optimizing
the content
for the destination if needed.
So let's switch our
focus to Trimming now.
As I showed you earlier in
the demo, it is really easy
to add Trimming to your
application via AVPlayerView.
All you need to do is just
call beginTrimmingWith
CompletionHandler and
NULL as the argument.
But since that's not
going to really tell you
that if it would
actually show the TrimUI,
you should always call
canBeginTrimming first though.
This method not only returns no
when AVPlayer already shows
the UI, but it also makes sure
that the current AVPlayerItem
can actually be trimmed.
But what if you are
really interested
in what the user
did in the TrimUI.
We start off with exactly
the same code as before.
This time, we are going to
implement a completion handler.
The CompletionHandler
has just one argument
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of type AVPlayerViewTrimResult.
The result that it will return
to you will either be
AVPlayerViewTrimOKButton
or AVPlayerViewTrimCancelButton.
In the former case we apply
whatever the user chose
in the UI, in the later case
we will set the trim selection
before dismissing the UI.
So I just told you
what AVPlayerView does
when the user interacts
with the TrimUI.
I haven't told you yet what
it does under the hood though.
We're not replacing the
current AVPlayerItem
or creating AVComposition.
Instead, we use existing AV
Foundation API and functionality
for setting the reverse
and forwardPlaybackEndTime
on the AVPlayerItem.
The player controls are
constantly observing these two
values and updating
when calling me.
One important note here,
AVPlayerView does not
provide state restoration
for the AVPlayer property
because we can't guarantee
that we will be able to
restore this object for you.
This means it is
your responsibility
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to restore the content when your
users relaunch the applications.
And this, of course, includes
the reversePlaybackEndTime.
So since we are just
setting some properties
on the current AVPlayerItem
when the user interacts
with the TrimUI the
underlying AVAsset
of course it contains
the entire movie,
which means that
if you use Trimming
and the application
supports export,
you have to set the timeRange
on the AV asset export session.
So first, you get the reverse
and forwardPlaybackEndTime
from the AVPlayerItem and
then you create a CMTimeRange
and set this value
on the ExportSession.
And this is exactly
what we do for sharing.
So wrapping up.
AV Kit is a new UI level Cocoa
framework for AV Foundation.
It provides you view-level
services--
sorry, it provides you standard
playback and trim controls
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
through the AVPlayerView.
With AV Kit you're now able to
use the power of AV Foundation
without having to write your
own custom playback controls
with our AVPlayer layer.
So please consider
adopting AVPlayerView
in your applications.
This will make sure that we
can provide a consistent look
and feel, not only
here in Apple's
but across all media
playback applications in OS X.
And if you have--
still not fully sold
on AVPlayerView maybe
this will convince you.
Starting on OS X Mavericks
QuickTime Player uses AV Kit
from media playback.
The view you see in the middle
is the same AVPlayerView
that you can use in
your applications.
With that, let me call Sam
back on stage to talk to you
about moving from
QuickTime and QTKit
to AV foundation and AV Kit.
[ Applause ]
>> Thank you Stefan.
All right.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So for the rest of this
talk, we're going to talk
about what's involved in
migrating an existing QuickTime
or QTKit application over
to AV Foundation and AV Kit.
How will you get
to AV Foundation?
Well, it depends on how you use
QuickTime and how you use QTKit.
For some developers, it will
be a pretty easy change,
but for others it may
require more refactoring
and deeper thought.
We're not providing
an API for API swap.
We've taken the opportunity
to change the API model
where we think it can
make AV Foundation better.
That said, some of the more
recently developed QTKit APIs,
were actually developed
around the same time
as their AV Foundation
counterparts
and so they have a
very similar API feel.
So, we're going to go through a
bunch of API areas and for each
of these API areas
I will show you--
I'll have some reminders
of the kind of QuickTime
and QTKit APIs you would
have used and then focus
on the AV Foundations
version and some of the things
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that make that interesting.
So let's start with the basics.
In QuickTime, every AV
resource in use is represented
by capital M, movie
object and for every track,
there is a track object
and a media object.
And QTKit has classes
wrapping all of these.
In AV Foundation, AV
resources are represented
by the AVAsset class and
for every track there is
an AVAssetTrack.
What's different in AV
Foundation is that AVAsset
and AVAssetTrack are immutable.
They provide a read-only
view of the AV resource.
QuickTime and QTKit
have a number of APIs
for creating a movie
or QTMovie object.
In this simplest form they
all operate synchronously
which means that your main
thread is blocked while
QuickTime and QTKit go off and
unload the file and parse it.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
With AV Foundation,
creating an AVAsset
from a URL always finishes
very quickly and succeeds,
that's because it hasn't
actually done anything yet.
You can then use the
asynchronous property loading
API to request that AV
Foundation load values--
load some properties
asynchronously
and provide a block that'll
be called when data--
when enough data has
been read and parsed
to produce those values.
So this lets that happen off
on another background queue
and lets you keep the
main thread responsive.
[ Pause ]
In QuickTime and QTKit, the
same movie object is the one
that you would call to
start playback or to pause
or to set the time or to step.
These are called
transport controls.
With AV Foundation,
we have separated
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
out the mutable state related to
playback into playback objects.
The principle playback object
is called AVPlayer and that's
where you send rate changes
and that means play
and pause as well.
Now, AVPlayer is designed to be
able to play through a sequence
of AVAssets, automatically
advancing
when each reaches its
end time, in some cases
with gapless transitions.
So this design goal
led us to take
out the per item state
related to playback
into its own object
called AVPlayerItem.
This is where you
tell-- this is the object
that you send messages
to seek and to step.
AVPlayer's subclass
AVQueuePlayer can play
through a sequence
of AVPlayerItems
and each will advance when
it reaches its end time.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So this is an example of how
we've refactored the object
in order to give us more
flexibility in the API.
Another thing to know is
that AVPlayerItems default
when you seek is to
snap to a keyframe.
This is because keyframe is
the most efficient display
for random access and
also the most efficient
to start playback from.
But there's a variant
of seekToTime
where you can specify
a tolerance range.
If there's a keyframe inside
that tolerance range
then we'll snap to it.
If there's no keyframe inside
that tolerance range
then we'll snap--
it will seek exactly
the time you specified.
If you specified tolerance of
0 then we always seek exactly
to the time you specified.
If you've used the
QTMovieView to integrate
into a view hierarchy
then you should check
out the new AVPlayerView
class in AV Kit
that Stefan demonstrated.
One thing that's different
if you used a QTMovieView is
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that the transport control
methods are not re-exported
by the view object.
You make those calls directly
on the underlying AVPlayer.
And AV Foundation provides
the AVPlayerLayer class
for integrating into a Core
Animation layer hierarchy.
It also provides the
AVSynchronizedLayer class
for synchronizing Core
Animation layer animations
to match the time
of a playerItem
as the playerItem's time and
rate change, so will the time
and rate of that
synchronized layer.
We've demonstrated in
previous year's sessions how
to use this class for
editing applications.
But there are other
ways you could use it
that aren't editing
applications as well.
Another thing to know is that
in applications linked on
or after iOS 7 or on or
after Mavericks by default,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
AVPlayerLayer and AVPlayerView
will both honor the user's
choices for media accessibility
by-- automatically.
This may not work
in the current seed
but it will work fine
in the next seed.
We have a session coming
up tomorrow on preparing
and presenting media for
accessibility that goes
into more details about what it
means for the playback pipeline
to honor the user's media
accessibility options.
Let's move on to
authoring and editing APIs.
QuickTime has a number of high
level movie transcoding APIs
and these are all synchronous.
And QTMovie has a
writeToFile method
which wraps these
synchronous APIs.
And in QuickTime-- sorry,
in OS 10.7 we introduced
an asynchronous
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
QTExportSession API.
Well, AV Foundation's,
AVAssetExportSession
API is also asynchronous
and it's internal pipeline
is deeply multithreaded.
It's also preset
based which means
that you can offer your users
high level options like 1280
by 720, and let AV Foundation
choose a well tuned bit-rate
for the encoding.
If you need more control
or if you need access
to the media frames--
the data that's going--
going through the pipeline then
you can use the combination
of the AVAssetReader and
AVAssetWriter classes.
With QuickTime, the APIs that
you'd use for reading media data
from a file were
different depending
on whether you wanted the data
be decoded or not decoded.
With AV Foundation,
AVAssetReader provides
features for both of these.
It depends on how
you configure it.
When you create the
AVAssetReader output
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to decode data-- to
retrieve data from a track,
if you pass an output settings
dictionary then we will decode
data into the format
you request.
If you pass a nil outputSettings
dictionary then we won't
decode it.
Now, I should note
that the QuickTime APIs
for reading video
data are synchronous.
They don't begin loading
and decoding video data
until you request it.
By contrast, AVAssetReaders
pipeline inside is
deeply multithreaded.
The file reading,
the video decoding,
and the audio decoding
all happen asynchronously
and in parallel.
This means that when you-- when
you pull a new media sample
out of the queues at the end
the latency when you make
that request is very, very low.
It has probably already
decoded it
in the background while you were
working on the previous frame.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The QuickTime APIs for writing
media files are also synchronous
and the same with QTKit.
AV Foundation provides
the AVAssetWriter API
and its pipeline is, you guessed
it, deeply multithreaded.
And you can configure it to
encode data, encode the video
for you-- encode the audio
for you or you can use it
to write already prepared
data to a movie file.
AVAssetWriter also provides
you much more control
over how you encode.
For some developers they find
that this control is
actually too much control
and they want preset access.
They'd like to be
able to use presets
like AVAssetExportSession.
Well, that's what we've
done in Mavericks,
we've added
AVOutputSettingsAssistant
which lets you use
presets as a starting point
for picking good output
settings for AVAssetWriter.
So, AVOutputSettingsAssistant
is new
in Mavericks and new in iOS 7.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Also in the session tomorrow on
preparing and presenting media
for accessibility we'll go
into more detail about how
to use AVAssetWriter to
construct subtitle tracks
and alternate audio tracks.
But let me go into more detail
about the internal
pipeline of AVAssetWriter.
Like I said it's deeply
multithreaded which means
that the video encoding
and the audio encoding,
and the file writing can
all happen asynchronously
and in parallel with each other.
But let's think about
how we construct movies,
in order to construct a movie
for efficient playback we want
to alternate, we want to
interleave, audio and video
and any of the track types.
This means that the
file writing queue,
the file writing process inside
AVAssetWriter will alternate
between the different
tracks, writing a portion
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of media data from each in turn.
Well, this means that sometimes,
if that file writing code runs
out of data on one track,
then pretty soon it will
stop consuming media data
on the other tracks until there
is more data on that track
that it's waiting for.
In fact, at some point it's
going to need to be able to say
to you at the API level, "Stop.
I'm not ready for more.
I don't want more data on these
tracks until I have more data
on that track," or for
you to say, "Actually,
I'm completely done giving
you data on that other track."
So, there is a flow control API
built into AVAssetWriter input
where you can query to say,
"Is this input ready
for more media data?"
And there's also an API
utility where you can ask it
to call a block over and
over and over just as long
as it's ready for more input.
If you wanted to get access to
video frames during playback,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
for example, to integrate them
into a custom OpenGL rendering,
QuickTime and QTKit provided
a way that you could set
up a visual context to
retrieve those video frames.
AV Foundation's API
for doing this is
called AVPlayerItemVideoOutput.
And similarly, there
was also a way,
that you could set
up an audio context.
And the audio context was
a way that you could tap
into the audio waveform as it
went past, either for analysis
or even to modify the waveform
to provide some kind
of audio effect.
AV Foundation in Mavericks
provides this API as well.
It's called an audio
processing tap and it's an API
that you install an
object onto an AV audio mix
and then you can install that AV
audio mix on to an AVPlayerItem
or on to an
AVAssetExportSession,
or onto an
AVAssetReaderAudioMixOutput.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
(It's like Dr. Seuss.)
Sometimes, you just want
to get a still image
out of a video file,
QTKit provided a utility
method for doing this.
AV Foundation provides
a whole class
and this class has both a
synchronous one-shot API
and an asynchronous batch API.
Editing. So, all three
of these, QuickTime,
QTKit and AV Foundation - all
provide APIs for inserting,
deleting, and scaling segments
expressed as time ranges
and these are really high level,
powerful APIs that you can use
to deliver high level user
focused editing experiences.
With QuickTime and QTKit,
these are methods on the movie
and the QTMovie objects.
With AV Foundation,
AVAsset is immutable.
So it can't be there.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Instead, we have subclass of
AVAsset called AVComposition
and it has immutable
and mutable variants.
Similarly, AVAssetTrack
has AVCompositionTrack
and there's an
AVMutableCompositionTrack
as well.
But it is important to know
that the API model is actually
different in AV Foundation.
This is a bit of an
advanced topic here.
But in all three
cases, QuickTime, QTKit,
and AV Foundation,
these editing APIs work
by manipulating a data
structure called an edit list.
An edit list is part of a track
inside the corresponding objects
and it is a data structure
that says play A for B seconds
and then play C for
D seconds, and so on.
Well, with QuickTime and QTKit,
to insert a segment of one movie
into another requires a bunch
of sample table copying.
This is because in this API, a
movie's edit list can only refer
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to that own movie's
sample tables.
AV Foundation's AVComposition's
edit lists do not require
that the sample table
be in the same asset.
So, it isn't necessary for
the sample table copying
to have occurred and that
makes it a bit more efficient
when you're doing the editing.
It also means that if
you insert a bunch of--
insert a bunch of clips and then
you delete them, you're not left
with leftover sample table
references that you don't need.
Another way of thinking
about this is that QuickTime
and AV Foundation
have different ways
that they do their references.
AV Foundation is using a
segment-level file reference
and QuickTime and QTKit are
using a sample-level reference.
If you have been using
the QTKit metadata API,
then here's some good news.
AV Foundation's AVMetadataItem
API is very similar.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
In fact, if you look up there,
you might be hard pressed
to find a difference.
Well, one little difference
that's on the slide is
that it's using Objective-C
property,
Objective-C 2 property, API's as
a way of describing the methods.
That's not really
so interesting.
The more interesting thing is
that metadata is no
longer loaded eagerly.
So, just like those
other properties
that you can use the
asynchronous key value loading
API, you use the
same API to request
that the metadata be
loaded and it can be loaded
on the background thread and
keep your main thread free
up for user responsiveness.
Let's move on to capture APIs.
[ Pause ]
The QTKit and AV Foundation
capture APIs are also broadly
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
very similar.
In both cases, you have a
capture session object and for
that capture session
object you add inputs
and outputs, and previews.
Now, with QTKit and AV
Foundation, in both of those,
there is a simple mode of the
API in which you automatically--
sorry, in which the session
automatically helps you build
the connections.
When you add the
inputs and the outputs,
it automatically builds
capture connections
by matching media types.
So, it links up all of the video
inputs to outputs and it links
up all of the audio
inputs to audio outputs.
But with AV Foundation there
is also an alternative more
advanced form of the API, and
in this more advanced form,
you use a different method and
you say, "No connections please.
I don't want you to build the
connections automatically.
I will make explicit calls
to set them up myself."
And this is a better choice if
you have multiple video inputs,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
multiple cameras, or if
you have multiple previews,
or if you have multiple
file outputs
because you can construct
exactly the graph that you want
by constructing those
connections yourself.
So, going through the inputs
and outputs, most of the inputs
and outputs are the same but
there are some extensions
and some improvements.
AV Foundation has a CaptureInput
object for capturing
from the screen and we've
done some work in Mavericks
to improve performance
of screen capture
when the cursor is visible
in the captured frames.
[ Pause ]
In QTKit when you added
a video preview layer,
it would automatically
create a video preview output
and that output object
was how you configured
certain properties.
Well, we no longer--
in AV Foundation,
we don't have the video preview
output object, we've folded all
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of those configuration APIs
into the video preview layer.
So, they're just one
stop for you to access
to get those configured.
And the video data output
and the audio data output
can produce both uncompressed
or compressed data,
and if available
on the hardware you're using,
the video encoding will
be hardware accelerated.
And there's also an audio file
output to write to an audio file
like a CAF file and
there's an output
for writing a still
image, for example,
when you take the photograph.
Another thing to know
about the output object is
that whenever there's
a delegate,
object that you can install,
you can also specify what
dispatch queue your delegate
callback will be called on.
The device discovery APIs you
can see here is very similar.
In both cases, there's an API to
get all devices, there's an API
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to get all the devices
with particular media type.
The default device
with the media type,
and if you already
know a unique ID,
you can get that device as well.
What's different with
AV Foundation is a piece
of final deprecation with
QTKit in a 32-bit app,
the old ancient QuickTime
sequence grabber video digitizer
components were grandfathered
into the device input list.
AV Foundation does not
grandfather those old sequence
grabber components.
It only supports the
modern DAL and HAL devices.
And AV Foundation provides
that video preview layer
like we discussed and
the interfaces that used
to be configured through the
video preview output object
and now ones you access
through the preview layer.
There's no direct analog
to the QTKit capture view
but you can take that layer
and put inside a view just
like you would with any
other core animation layer.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So we've gone through a
bunch of high level things.
Let's dig down into
some low level stuff,
beginning with time.
We, in media systems, are of
the opinion that the right way
to represent time in a media
system is as a rational number.
Floating point numbers will
almost always introduce some
kind of rounding error because
the numbers never have--
never happen to be expressed--
exactly expressible as
floating point numbers.
And when you add this up by
the millions, they can lead
up to measurable drift.
So, we use rational numbers.
And in order to support
really long media files,
you need to use a 64-bit time
value and a 32-bit time scale.
The time value is our
name for the numerator
and the time scale is our
name for the denominator.
Now, QuickTime had
a 64-bit numerator,
32-bit denominator time
object called the TimeRecord.
But, in the very early 1990s,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
doing 64-bit math
was really awkward.
And so, there were shortcuts
where you'd pass a time value
that was a just a
32-bit time value.
And these shortcuts looked
like they were good
conveniences 'cause, you know,
most movies were that
short, that's fine.
But unfortunately, they
were also shortcuts
that were taken inside
QuickTime and that meant
that you couldn't
really take advantage
of that full 64-bit size.
QTKit has a QTTime which is a
struct for the 64-bit numerator
and 32-bit denominator
used in many of its APIs.
It's great, but it's
still using QuickTime
for playback and editing.
And so, it was kind of hamstrung
by that limitation
in the pipeline.
But with AV Foundation
just like QTKit,
we have a struct called CMTime
which has a 64-bit numerator,
a 32-bit denominator, and
a bunch of other cool stuff
that you can look
in the header file.
And just like QTKit, there's
a time range a CMTime range
expressed as two times-- as two
CMTimes, a start and a duration.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And there's also a
CMTimeMapping which is expressed
as two time ranges, a
source and a target,
and that's used in
some editing APIs.
But the really good news is
there's no 32-bit shortcut
which means we didn't use
a 32-bit shortcut either
and you can use that
full 64-bit size.
The APIs, the objects
that represent time
when it's moving
are very similar.
In both cases, there's
a clock object
that represents an external
source of moving time
which is not under
program control
and then there's a timebase
that you can set the rate
and set the time and so forth.
And the timebase's time
comes from its master
which could be a clock
or another timebase.
That's a model that we
like and we've continued.
QuickTime, in some places
QTKit, would use a--
non-opaque data structure called
the SampleDescriptionHandle
to describe compressed data.
Originally, it was read
and written directly
from movie files
and that was great
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
when everything was big-endian
and the file is big-endian,
and the Mac is big-endian,
woo-hoo, but then we ported
to Intel processors and we had
to endian flip it and we had
to have rather complicated
rules for how you endian flip it
and some pieces stay big-endian
in memory and that's kind
of weird to deal with as well.
So, we've learned that lesson
and now we have a rather
nice FormatDescription object
which has a clean API.
And we also had a rather
nice SampleBuffer object
for holding individual samples.
It retains the format
description.
It carries timing information.
The data that you've
references may--
it doesn't have to contiguous
and it can even just
be a promise it's going
to be delivered later.
And it has a way to
attach key value pairs
of supplemental information
for other descriptive purposes.
If you need to encode or
decode video frames directly,
the AV Foundation offers
the Video Toolbox APIs
like the enhanced Image
Compression Manager APIs
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that were introduced
in QuickTime 7,
these use core video
pixel buffers
for the uncompressed images.
They use core media sample
buffers for the compressed data
and there is an additional
interface
for transferring an image
from one pixel buffer
to another pixel buffer.
If you are still using
QuickTime's graphics importers
and graphics exporters for
still image support, bless you.
I worked really hard on those,
but it is time to wake up
and smell-- smell the 64-bit
native uniformly thread safe
coffee delivered by
the Image I/O Framework
and it's CGImageSource and
CGImageDestination APIs.
So there's the tour.
There's the travel guide.
I've actually introduced you
to quite a large number of APIs
but we've scratched the surface.
There are more AV Foundation
APIs that you can find
in Headers and Documentation.
In summary, we are deprecating
the QuickTime C framework
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and the QTKit Objective-C
framework in Maverick.
This is a reminder for you
to make the transition.
The deprecation warnings
will appear on your screen
but you'll still
be able to compile
and your apps will still run.
Everybody, your apps
will still run.
So, AV Foundation is the
stack of media frameworks
that we are building and
working on and focusing
on in the media systems group.
And we're building frameworks
with significant
architectural advances
over QuickTime across the board.
They're better.
That said, we're not deprecating
the QuickTime movie file format.
It is still supported as
our primary file format
in AV Foundation and
in QuickTime player.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We're introducing
QTMovieModernizer in Mavericks
to help bring your media into
AV Foundation supported formats.
We have introduced a great
new view integration class
in AVPlayerView as
part of AV Kit.
Finally, we know that
developers have had a long
and rich history
developing with QuickTime
over its last 22 years.
And we know there are
probably some of you
who are using QuickTime in ways
that we haven't yet anticipated
in AV Foundation's family.
If you can't figure out how
to bring your QuickTime API
app forward to AV Foundation,
we want to hear from you.
We're interested
in your feedback.
You can come and see us
this week in the lab.
You can write up enhancement
requests using Apple's bug
reporting mechanism, and you
can send email to John Geleynse,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Apple's Director of
Media Evangelism.
His email address is
on the next slide.
But we are not finished
with AV Foundation.
We-- we'll make it
better with every release
and we're interested
in your feedback.
There's the email
address for John Geleynse.
There's two resources
online that I want
to draw your attention to,
one is the AV Foundation
Programming Guide
and the other is
called Tech Note 2300.
It's all about moving
QuickTime and--
oh, it's all about moving
QTKit code to AV Foundation.
There's also an AV Foundation
zone inside the Apple
Developer Forums.
You can get help from other
AV Foundation developers,
and sometimes if you're lucky,
from AV Foundation engineers.
And it's also searchable.
Sometimes the question
that you're asking has
already been asked.
We have three other
sessions from media systems
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and AV Foundation
folks this week.
There's the preparing
and presenting media
for accessibility session
which we'll go into more detail
about what it means
honor a user's
accessibility preferences.
There is, "What's new
in camera capture?"
Which will focus
principally on new features
in AV Foundation
capture on iOS 7.
And there's also very exciting
session on advanced editing
with AV Foundation which
will introduce a new way
of integrating your
code directly
into our playback pipeline
to do fancy video effects
and filters limited only
by your imagination.
That's very exciting
feature to me.
Thanks for coming up.
Bye, bye.
[ Applause ]