WWDC2015 Session 508

Transcript

X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[Applause]
>> DOUG WYATT: Good morning.
I'm Doug Wyatt from the
Core Audio team and I would
like to show you something
new we have been working
on called Audio Unit Extensions.
This is a new technology in
iOS 9 and OS X El Capitan.
About Audio Units: we've
had had this technology
in our operating systems since
the beginning OS X and iOS.
The operating system
includes a great number
of built-in units ranging
from I/O units and mixers,
a lot of different effects
ranging to software sampler.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
a lot of different effects
ranging to software sampler.
We use these internal
Audio Units in many
of our higher-level
APIs, for example,
the media playback stack.
But Audio Units are also a
widely adopted third-party
plug-in format on OS X.
There are literally thousands
of third-party Audio Units
in the market out there.
Now, Audio Unit extensions
for the first time bring
us a full plug-in model
on both OS X and iOS.
It is built on top of the
app extension technology,
which means if you are writing
plug-ins you can package them
into apps, and those apps can
be sold on the App Stores.
[Applause]
As part of this technology,
we have modernized the API and
yet at the same time
maintained compatibility,
and in this session I will go
through details of this new API,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and in this session I will go
through details of this new API,
which we are calling the
version 3 Audio Unit API.
It's based on an Objective-C
class called AUAudioUnit that's
in the Audio Unit framework.
And as an Objective-C class,
of course, it plays nicely
with Swift, as we will see.
In this session, we are also
going to look at a number
of classes in the
AVFoundation framework.
We have AV Audio Unit
component manager
and AV Audio Unit component.
These are used to located the
audio components on the system.
Those appear for the
first time in iOS 9.
They also exist on Yosemite.
And we will also be using
AVAudioEngine in some
of our example code we
will be showing today,
in particular the
AVAudioUnit class
and AVAudioUnitEffect class.
Those have been available
since last year's OS releases.
So about compatibility now.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So about compatibility now.
This is how things
look now in OS X.
We have our existing
version 2 Audio Unit hosts
and existing version 2
Audio Unit implementations.
The hosts start their
communication
with audio component
instance new,
and our implementations
are built
on audio component
factory functions.
We have a new set of APIs here,
so we will have new hosts
using those new APIs.
And new Audio Units implemented
using those new APIs.
Hosts will communicate with
the class AU Audio Unit.
New version 3 Audio Units
will subclass AU Audio Unit.
So that's two separate APIs.
What are we going to
do to be compatible?
We have built bridges
between these two APIs.
So thanks to these
bridges, we will find
that new version 3 hosts should
be almost completely compatible
with existing version
2 Audio Units.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And conversely, existing version
2 hosts would only need minor
source changes to work with
new version 3 Audio Units.
And I will detail those API
changes in a little bit.
So now I would like
to give you a demo
of a new Audio Unit working
in an only slightly modified
version of Logic Pro.
I have a little session here,
it has a drum loop built in.
And here I'm going to apply
an Audio Unit to this track.
So here are all of the
Apple built-in Audio Units.
And here I have a new demo
Audio Unit called v3 Distortion.
So I can open this Audio Unit.
I can find the preset I like,
and we can hear Logic playing
through this Audio Unit.
[Music]
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[Music]
There it's dry.
Completely distorted.
Now, if I go to activity
monitor here,
we can see that this
distortion Audio Unit is running
in a separate process,
AU v3 distortion.
It's consuming a
little bit of CPU.
It has some threads running.
Now, suppose this Audio Unit
has a bug in it, and it crashes.
Well, I can simulate that
here in activity monitor.
I can force quit it.
And notice in Logic,
the view went blank
but the music kept playing.
[Applause]
So here is a diagram of what
we were just looking at.
That's a slightly modified
version of Logic Pro,
but it's still basically
communicating using the existing
version 2 API, which is bridged
to AU Audio Unit and in turn,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
version 2 API, which is bridged
to AU Audio Unit and in turn,
in that separate
extension service process,
we saw the distortion units
AU Audio Unit subclass running
along with its custom
View Controller.
In the Logic process, there
is also a View Controller,
and you see how these
are bridged
across the process boundary.
Now, I'd like to talk
about hosting Audio Units
and I will show you an example
that uses the version 3 APIs.
We have sample code called
Audio Unit v3 Example.
I checked a couple of hours
ago, but it hadn't appeared yet.
I hope it comes out today.
In this sample code project,
you will see there are a number
of targets and one of
them is called AU Host.
Now, this application is fairly
simple and straightforward,
but it shows how to find and
open Audio Units that are
on the system, how to
connect them together
into a rendering chain, how
to select Audio Unit presets,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
into a rendering chain, how
to select Audio Unit presets,
and how to open an Audio
Unit's custom view.
So in the AU host app, we have
something called simple play
engine, which is a Swift
class that uses AVAudioEngine.
It uses an AV audio
player node connected
to an AV Audio Unit effect.
That AV Audio Unit effect
in turn exposed an
underlying AU Audio Unit,
which is the maiden class of
the version 3 Audio Unit API.
We have the player
to the effect,
to the mixer, to the output.
That's how the simple
play engine makes sound.
We will also see how to use the
AV Audio Unit component manager
class to select from the
AV Audio Unit components
on the system and use
that to control what kind
of AV Audio Unit
effect gets selected.
So let's get into a
little bit of code,
but first there is a very
fundamental data structure here
when working with Audio Units.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
when working with Audio Units.
We have the audio
component description
and its first three fields:
the component type, type,
subtype, and manufacturer.
That tuple uniquely identifies
an Audio Unit in the system.
The flags are also important.
They are partially populated
by the audio component,
and there are new ones
populated by the system.
We will describe some
of those as we go along.
The important thing
here is this is the key
that identifies the plug-in.
So to find Audio Unit
components on the system,
the first thing we do is create
an audio component description
that contains a wildcard.
Here we say the component
type is effect.
That's not a wildcard, but
we have component subtype
and manufacturer of zero.
Those are wildcards, so we have
built a component description
here that identifies any effect.
And then we can take that any
effect component description
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And then we can take that any
effect component description
and pass it to AV Audio
Unit component manager,
and it will give us
back all of the effects
on the system matching
that wildcard.
So here we get an array of AV
Audio Unit component objects,
and those contain things
like the name, tags,
also the audio component
description of that unit.
So here we have got an
array of components.
We can pass that back to the UI,
and in turn the UI can call this
method in the simple play engine
to select one of these
previously vended
effect components.
So here it gives us a component.
We are going to fetch the
audio component description
out of that, pass it to an
internal method, and the guts
of that internal method is here.
We are going to call a new
class method of AV Audio Unit.
And this method asks it to
create an instance based
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And this method asks it to
create an instance based
on the component
description we have here.
Now, this is an asynchronous
function, meaning it's going
to go off and start
instantiating it,
and then it's going
to call this closure
when it actually has
instantiated the Audio Unit
and we are ready to use it.
So here we are in our callback.
This is Swift closure syntax.
We have our AV Audio Unit.
And then we can attach
it to our engine.
We have stored it into a
member variable, the effect.
And now we have an AV Audio
Unit know that's the effect.
We are going to patch
that into the engine.
We will disconnect the
effect from the main mixer,
then connect from the
player to the effect.
And then from the effect
to the main mixer node.
So now we have got an
effect in our play engine.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And now we can store the
actual AU Audio Unit.
That's the plug-in, and
here we can do all kinds
of interesting things
like manipulate the component's
effect, presets, and parameters.
For instance, here, we will just
get the list of factory presets.
And this too can
populate a field
in the table view --
rather, in the UI.
So the user can choose
the factory preset.
And finally, I would
like to show you how
in the app I will
show you in a minute,
we can get the Audio Unit's
custom view and embed it
into the host application's
view.
Here we are in the View
Controller of the host,
so we are going to ask the play
engine, give me your Audio Unit,
and then we are going
to ask the Audio Unit
for a View Controller.
When it's done with that,
it will call us back
with a View Controller that we
can embed in the host's view.
Okay. I would like to bring up
my colleague Michael Hopkins now
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Okay. I would like to bring up
my colleague Michael Hopkins now
to show you this app
actually running now.
[Applause]
>> MICHAEL HOPKINS: Thank
you very much, Doug.
I'm delighted to have
this opportunity today
to show you this
AVAudioEngine-based v3 host
application running
on an iPad here.
As you can see, I'm
going to launch
that by tapping the
icon for the host.
And on the left-hand side of
the screen we have a list of all
of the effects Audio Units
that are present on the system.
And this includes both
the built-in Apple audio
component-based effects as well
as several new extension-based
v3 Audio Units I
installed myself.
At the top of the screen, I have
a Play button that I can tap
to toggle the playback
of a drum loop.
Now, let's see how I can
apply some effect nodes
and add them to the graph.
First, I will play
this with no effect
and then I will add
a couple effects
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and then I will add
a couple effects
so you can hear that working.
[Music]
With the high pass filter,
it's filtering out almost all
of the sounds of the cymbals
and other higher frequencies.
A delay, which is a little
bit hard to hear in this room.
And I'm going to go
ahead and stop that.
So now I would like to show you
for the first time an
extension-based Audio Unit
running on this iPad.
That is the distortion demo.
When I select that, now
you can see the list of all
of the factory presets that
the Audio Unit is publishing.
These include some
drum-specific ones as well
as some really crazy, wild
effects like alien chatter.
Now, as Doug mentioned ,
v3 Audio Unit can have
a custom view on iOS.
And I'm going to show you that.
I'm going to go ahead
and tap the View button.
And what we have done is we
have loaded that View Controller
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And what we have done is we
have loaded that View Controller
from the Audio Unit
and I have installed it
as a child View Controller
within our application context.
So for the first time, we
have a built in Audio Unit
with a UI running in our host.
We have a large slider, excuse
me, a large knob that I can use
to control the amount
of distortion.
And let me go ahead
and play that for you
so you can hear that in action.
[Music]
It's really a lot of fun.
It's an amazing experience
to be able to have
that Multi-Touch UI working
fluidly in a host application
without having to go through
all of the hassle of switching
out to another application,
doing some tweaks,
switching back to your host,
starting recording,
switching back.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
starting recording,
switching back.
Now, you won't have
to do that ever again.
[Applause]
Thank you.
And I'd also like
to point out further
that this is the same Audio Unit
that you saw running in Logic
in Doug's earlier demo.
In fact, the source code for
the Audio Unit is identical.
No changes were required.
The drawing code is also
very similar because I chose
to write this using
Core Animation
so that API is almost
fully portable.
The only changes
that were necessary
to bring this Audio Unit to
iOS are in the event model,
whereas we have had to use
the touch events in UIKit
versus the AppKit mouse
events on the desktop.
So really you guys
have an opportunity
to with only a few changes
to publish an Audio Unit both
on the desktop and on iOS.
Thank you very much.
[Applause]
Back to you, Doug.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
>> DOUG WYATT: Thank
you, Michael.
So I would like to talk
about using Audio Units
in your host applications
in situations
where you are not
using AVAudioEngine.
We have a similar method
on the AU Audio Unit class
to asynchronously
create an instance
of the component description.
You see that there.
We also, for those of you
with existing version 2 hosts,
a minimal translation path is
to start using audio
component instantiate.
We will talk about that
in detail in a bit.
Now, I would like to
talk about the subject
of extension service processes
versus plug-ins loaded
into host processes.
Now, as anybody who has worked
with Audio Units is aware,
of course, with our
existing plug-in model,
the plug-ins are always loaded
into the host's progress,
and this remains true
for version 3 hosts.
If it's a version 2 existing
plug-in, and that might be one
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
If it's a version 2 existing
plug-in, and that might be one
of the Apple built-in
ones on iOS
or it could be a third-party
one on OS X, but in any case
if it's a version 2 Audio Unit,
regardless of any other factor,
it's always in the
host's process.
Now, version 3 Audio Units have
a slightly more complicated
story here.
By default, version 3
Audio Units are loaded
into a separate extension
service process.
And this is the diagram
we saw before with Logic.
This is true, again,
whether it's a version 2 host
or a version 3 host.
Now, on OS X only it is
possible for the plug-in
to be loaded directly
into the host's process.
Now, for this to happen,
both parties have to opt in.
The host when instantiating
the Audio Unit has
to pass this option to any
of the asynchronous
creation methods we just saw,
and you see the name of that
new flag there called Load
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and you see the name of that
new flag there called Load
in Process.
And the Audio Unit also has
to be packaged specially
and consent to this with
a plist entry called Audio
Component Bundle.
So if both parties do opt in,
then the framework will
actually load the plug-in
into the host's process.
So the host will be
communicating directly
with the plug-in's AU
Audio Unit subclass.
Now, as a host author, why
would you want to do this?
There is a tradeoff
here between safety
and performance is the reason.
Of course, it's a security risk
to be loading third-party code
into your app, and if it
crashes inside your app,
then users might
blame you instead
of the misbehaving plug-in.
But on the other hand, we
have performance reasons
where you may want to load
plug-ins into your process
if you are a host, because
there is some overhead
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
if you are a host, because
there is some overhead
to communicating with
that separate extension
service process.
And we have measured that
as being on the order
of 40 seconds microseconds
per render cycle ,
o you can do the math to figure
out how significant that is
in the context of your host.
You have some number
of out-of-process plug-ins you
might be communicating with,
so you have to add that up.
And there is also the factor
of how much audio you are
asking them to render.
For example, if you are
rendering at a very low latency
of 32 frames, that's a 1
millisecond render interval,
so overhead of 40 microseconds
could be significant
at 5.5 percent.
So that's your tradeoff
if you are a host author.
I mentioned earlier
that existing version 2 Audio
Unit hosts need a few changes
to work with version
3 Audio Units,
and here is what has to change.
I mentioned the audio
component description flags,
and in there, the
component flags.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and in there, the
component flags.
There is a new flag called
Requires Async Instantiation.
That's set for most if not
all new version 3 Audio Units,
so if you see that flag set
in the component description,
you have to use the new Audio
Component Instantiate method
instead of Audio
Component Instance New.
Now, similarly in an
existing v ersion 2 host,
if you want to access an
Audio Unit's View Controller,
then you also need to use a new
asynchronous method to do that.
There is a new property,
Request View Controller.
It is also asynchronous.
You can read about
the details of that
in Audio Unit properties.h.
So about these asynchronous
methods.
You can use the new methods
with version 2 units,
but you must use them
with version 3 units
when the flags are set.
And the reasoning here,
well, the big sort
of externally facing reason is
that it helps responsiveness.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of externally facing reason is
that it helps responsiveness.
If it's going to take half
a second to instantiate
that Audio Unit, well, if
you unblock the main thread,
your host applications, meters,
or other animations will
keep drawing smoothly.
Now, especially when
updating existing code --
and this was the
first thing I did
when testing internal test code
-- it's tempting to sit there
and wait on the main thread
for the asynchronous
operation to complete.
Well, don't do that, because not
only will you block any graphics
you are doing, but you will also
block some underlying operations
in the framework that
are actually required
for the Audio Unit
to be instantiated.
So you will deadlock if you
block on the main thread.
Don't do that.
Now, I would like to
switch gears from talking
about hosting Audio Units
to creating Audio Units
with the new version 3 API.
First, a few words
about app extensions
since the new Audio Unit model
is based on app extensions.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
since the new Audio Unit model
is based on app extensions.
App extensions are bundles with
a file type extension of .appex.
Xcode will build them into
an app's plug-ins directory,
and we saw how they get
loaded by the system
into separate extension
service processes.
You can read all about the nuts
and bolts of app extensions
in the app extension
programming guide.
Now, our new sample code
project, Audio Unit v3 Example,
contains a sample Audio
Unit implementation called
Filter Demo.
Filter Demo, when you look
at that sample project,
has three targets.
It has what we call
the containing app,
and what it contains is the app
extension as well as a framework
where a lot of its
common code exists.
Both the app and the extension
link against this framework.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Both the app and the extension
link against this framework.
Now, inside this framework,
we have two main classes.
There is AU v3 Filter Demo,
that's the AU Audio
Unit subclass,
and the Filter Demo
View Controller
that controls the custom
view of the Audio Unit.
Now, what's cool about
doing things this way is
that while we are developing our
signal processing and view code,
we can do this all in the
context of our own app
so we are debugging not in a
separate SPC service process,
but we are debugging
and developing our code
right there interactively
in our own app.
We also let our app
look like something
when the user opens it.
It's not just a plug-in
for somebody else.
And we are not duplicating
any code
to be able to accomplish that.
There is one extra bonus here
that on OS X, if we want to,
we can designate this
framework as being the bundle
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
we can designate this
framework as being the bundle
that a host process
can load into itself.
So let's look at
the app extension.
It has an info plist
with important entries.
The NSExtensionPointIdentifier
tells the system what kind
of extension it is, the main
storyboard tells the system,
when you launch my
extension service process,
open the storyboard.
And finally, there is an
Audio Components array
that tells the system, here are
the audio component descriptions
that I am registering.
Just a quick reminder here, in
your storyboard, you will want
to be sure to specify
your custom class.
You may need to specify the
module if you are building it
into a separate framework
like we are here.
Then the extension actually
has no code in it other
than this little
bit of dummy code
to keep it from being empty.
We have to link against
the Filter Demo framework.
All of the good stuff
is in there,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
All of the good stuff
is in there,
and here we just have a global
variable referring to it.
Let's move onto the
framework now.
So the main class
in the framework is the
Filter Demo View Controller.
In extension terminology,
this is the extension's
principal class.
Whenever the extension
is created or loaded,
the system will create
an instance
of that principal class.
And it's got two main jobs.
It in turn creates the AU
Audio Unit subclass and,
as a View Controller as you
would expect, it creates
and manages the plug-ins
custom view.
Here is the class declaration
for the Filter Demo
View Controller.
It derives from AU
View Controller,
which is either an NS or UI
View Controller basically,
and it also implements a
protocol called AU Audio
Unit factory.
That's a simple protocol and
implements exactly one method,
Create Audio Unit with
Component Description.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Create Audio Unit with
Component Description.
And the job of this method is
to create the AU
Audio Unit subclass.
And here it is, the
AU v3 Filter Demo.
Now, let's look at that
AU Audio Unit subclass.
So for reasons we will
get into in a bit,
these are actually embedded
C++ classes or objects.
The filter DSP kernel is
where all of the math happens.
We will listen to it later.
It's a little more interesting
than looking at its code.
We have some code here
dealing with the buses.
This is an effect.
It has one input and one output,
and our base class is going
to want us to provide
arrays of buses
so we have numbers
to support that.
And we have something
called a parameter tree.
We will see what that
is in just a second.
Here is the initializer.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So the first thing we do
is initialize our input
and output buses, and then
we wrap them in bus arrays.
And these arrays each
contain a single bus.
And now we are looking
at parameters.
So every parameter is an object,
and you can think of this object
as kind of the bridge
between your implementation
and the host.
In the middle, there is
the parameter object.
This is a simple
low-pass filter,
so it's only got two parameters,
a cutoff frequency
and residence.
Every parameter has
an identifier,
so here we are saying
it's cutoff.
It has a localizalbe name.
We are being bad and
not localizing it here.
It has an address.
We will talk about
that in a bit.
Arrange, and some units, and
flags which you will recognize
as being almost identical
to what we do
with version 2 Audio Units.
So here we have created
our first parameter.
We will create our second
one almost identically.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We will create our second
one almost identically.
And then finally we can
create our parameter tree,
passing an array of
those two parameters.
Now, we have our parameter tree,
and we want to wire it up so
that it's connected
to our DSP code.
And the way we do this
is install a block
into the parameter tree called
the Implementer Value Observer.
So this block will get
called any time somebody,
whether it's the host or our
own view, changes a parameter,
And so in response to that
change, we will simply set
that new value on
our filter DSP kernel
so that it takes
immediate audible effect.
Now, in the other
direction, there are times
when the tree needs to refresh
its value based on what we have
in our signal processing.
That's what this block does.
It fetches the current
value from the DSP
and returns it to the tree.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and returns it to the tree.
Next, this is an
important override method.
If you're familiar with the
version 2 Audio Unit API,
this was called Audio
Unit Initialize,
which is not a good name choice
in the Objective-C world.
So we decided to make
it very specific.
What was initialize time
is really prepare to render
and allocate the resources that
are associated with rendering.
So there are things like
buffers, DSP state, and so on.
So the first thing we do here
is called the base class method.
Then we can ask our input
bus to allocate some memory
for audio input to the plug-in,
and we can initialize our signal
processing code here based
on the current channel count and
sample rate of the output bus.
So entirely parallel, we
have a method that's called
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So entirely parallel, we
have a method that's called
to deallocate render resources.
And here, too, we
call the base class
and basically undo whatever
we did when allocating.
So the process of rendering
works through a block
that gets called every render
cycle, but we are asked
to provide this block
before starting to render.
Here we are going to
capture our C++ members
into local variables
that are pointers.
Now, the reason for this
is that we are going
to be running our block for
rendering in a real-time context
where it's not safe to access
any Objective-C objects
because the runtime could block
and cause an audio glitch.
So, again, we are just going
to capture our C++
member variables.
And then we can return
the block.
It returns AU Audio Unit status.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
It returns AU Audio Unit status.
And if you are familiar
with the version 2 API,
the parameters are
largely the same.
There is a time stamp, a
number of sample frames,
an output audio buffer list,
and here is something new called
the real-time event list head.
I will talk about that
in detail, but it relates
to scheduled parameters
and MIDI events.
And finally, the
Pull Input block.
This is how the host
tells us, the implementer
of the Audio Unit,
where to get input from.
So in the guts of
the input block,
the first thing we will do
is pass that Pull Input block
to our input C++ object
and ask that input object
to fetch the audio input
for this render cycle.
Then we are going to do some
housekeeping with buffers.
We will send them down to
the DSP state, and finally,
we ask the DSP state to process
the audio for this render cycle.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
we ask the DSP state to process
the audio for this render cycle.
It's already been
informed of the buffers,
and we are just going to give it
a time stamp, the frame count,
and the linked list
of real-time events.
So that's the guts
of this Audio Unit,
there is not a whole
lot of code.
There is a lot more code
that does the actual
signal processing,
but as I mentioned before,
it's better to listen
to that than to look at it.
So I would like to bring
Michael Hopkins back
up to show us the
AU v3 Filter Demo.
>> MICHAEL HOPKINS:
Thank you, Doug.
[Applause]
I'm going to go ahead and
start with the app container
that contains the extension.
You will see first on the screen
in the left-hand side
is our Filter Demo,
which we have distributed
as sample code.
To the right of it is the
distortion demo application
that I showed you earlier.
I will go ahead and
launch the Filter Demo.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
I will go ahead and
launch the Filter Demo.
Now, at the top of the screen,
you will notice the
two parameters
that Doug talked about.
We have the cutoff in
the residence parameter,
and these are represented
in our UI
with a slider and a text field.
And this portion of the
UI is actually contained
in the application, whereas the
larger area in the main screen
with the graph is
actually our embedded view
from the Audio Unit.
I can go ahead and change
the value of the parameters
by dragging the slider.
And what's happening here is
the application is changing the
value of that parameter.
And the view is listening for
changes to that parameter,
and then it's updating.
As you will see,
that update is live.
Conversely, I can
interact directly
with our embedded Audio
Unit view by tapping
in the graph and dragging.
And you will notice that as
I drag that with my finger,
the application is
receiving notifications
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
the application is
receiving notifications
that the parameters have changed
and it's updating in turn.
But that's kind of a boring
demo without any audio going
through it, wouldn't you say?
Let's go ahead and
take a listen to that.
[Music]
I could do this all day.
You got time?
Now, it's really, really
cool to just, the fluidity.
Thank you.
Just how fun it is to use your
fingers to just be able to play
with that in a Multi-Touch UI.
It's phenomenal.
Another thing that's
cool about this is
because we have designed the
user interface in such a way
that it can adapt to any
size that it's embedded in,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that it can adapt to any
size that it's embedded in,
we can actually take this iPad
and we can rotate it sideways,
and you can see that now the
user interface is updated.
And from a portrait view to a
landscape view, and vice versa.
Now, we are doing that because
we are supporting Auto Layout
and we are looking at size
classes, so that's really great.
But what happens when we go
and we take this and we put it
into our host app, which
has a much smaller amount
of screen real estate
dedicated to plug-ins?
So I will go ahead and
switch back, and we are going
to open the host there.
I'm going to get rid of the
beautiful distortion demo
and embed our Filter Demo view.
Tap on View to load that.
And now you can see
that that's being loaded
in a constrained vertical space
and a very wide horizontal space
yet it still works
as you would expect,
and none of the labels overlap.
It still works exactly
as we would expect.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
It still works exactly
as we would expect.
So this is a fantastic
new technology,
and we are so excited to be able
to finally bring it to you guys
and I can't wait to see what
do in your own iOS apps.
Thank you.
[Applause]
>> DOUG WYATT: Thank
you, Michael.
Just a few words now about
the containing app in general.
It is a vehicle for your
plug-in and helps you
with rapid iteration and
development, but you can think
about putting some extra
things in that containing app.
We saw with the Filter Demo that
it has the simple play engine,
you may for whatever reason
want a more complex play engine
of some sort.
This is also a good
place, the containing app,
to try putting creative
touch controller.
There may not be room
in a plug-in view
for your full touch controller.
Maybe there is, but you might
think about having extra,
an extra-large version
or something
in your containing app.
The app is also a good place
for any documentation or help
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The app is also a good place
for any documentation or help
that would make the
plug-in view a little dense.
A few final words about
creating an app extension here.
So if you are going to build
a framework to be loaded
in process on OS X, despite what
we are doing here with Swift,
we can't recommend that
you do this on OS X
because the Swift API
is subject to change.
If you build your plug-in
against one version
of the Swift runtime and you are
loaded into a host that happens
to be using another version,
there could be collisions,
and that would be bad.
We realize when you look at the
sample code here and you try
to build your own plug-ins,
there is a lot, you know,
there is three related targets
that have to be built properly.
It's a little bit complicated.
We do plan in Xcode's template,
but for now you can copy
liberally from the Filter Demo.
Okay. Now I would like
to talk in general
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Okay. Now I would like
to talk in general
about the modernized AU Audio
Unit API from both the host
and the implementation sides.
I would like to compare the
way that properties are handled
in version 2 versus version 3.
In version 2 Audio Unit
API, we have scope-
and element-based properties.
A large number of properties
are in the global scope
so you have a bunch of
code where you type.
K Audio Unit scope
global, element 0.
And it's painful,
especially from Swift,
where we have property values
that are void pointers.
You end up typing unsafe mutable
pointer all over the place,
and that makes my head hurt.
We have these functions
with long argument lists.
And by comparison in
the version 3 API, well,
properties are properties.
We use a dot syntax in
Objective-C and Swift,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We use a dot syntax in
Objective-C and Swift,
so you can write
AU.maximum frames to render.
We also implement our
classes to be key-value coding
and key-value observing
compliant so you can use value
for key, and add
observer for key path.
We also added a special KVO
method to the bus array,
add observer to all
buses, so you don't have
to simultaneously be watching
for buses to come and go just
so that you can add
KVO observers on them.
That can be a painful
cycle to chase.
Speaking of buses, they
are full-fledged objects
in the new API.
We have the AU Audio
Unit bus array.
The AU Audio Unit has
an array of input buses,
an array of output buses.
And the buses have
two major properties.
They have a format and a name.
The format is manipulated
by the host.
We are able to reject
formats that we don't like.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We are able to reject
formats that we don't like.
We use the same formats
in version 3 Audio
Units as version 2.
Let's look at parameters.
We have some of the
same problems here
in the version 2
API with parameters
as we did with properties.
We have these unwieldy
scope element ID tuples.
And furthermore, in some very
complex AUs we just didn't have
enough bits.
We have these functions with
long argument lists again,
and we also have a complicated
AU event listener API.
In the version 3 API, I
hinted at this earlier
with the parameter tree
and the Filter Demo.
Well, it is a full tree.
Parameters can be grouped,
and here we have an example
of a simple analog
synthesizer emulation.
It has groups for oscillator,
filter, and amplifier.
The filter and amplifier have
envelope groups beneath them.
And the most brightly
colored boxes beneath are
the parameters.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So the waveform octave,
filter cutoff and resonance,
and the envelope attack,
sustain, and release.
So these boxes are all nodes,
whether they are groups
or parameters, and every node
in the parameter tree has
a unique and permanent ID.
This is like a C identifier.
So using these unique
IDs, we can use KVC to go
and find a parameter
we are looking for,
such as oscillator.wave
or filter.envelope.attack.
And this would be
a lot more flexible
for these very complex
audio units
that have very large
parameter trees.
Now, you will notice
that parameters have
numeric addresses
and that they are 64 bits,
but we do have to treat them
as transient in any situation
where we aren't the one who made
up that addressing scheme.
So that means if I'm a
host application and I want
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So that means if I'm a
host application and I want
to record some parameter
automation, I should record
that automation using
the key value path --
I'm sorry, the key path of that
parameter and not its address.
I alluded to this earlier.
The AU parameter object is
the focus of communication
for parameter values
between hosts and views,
and on the other side, the
Audio Unit implementations.
Now, from the host's
point of view,
the parameter object has
properties like its value,
also a minimum and
maximum value, and so on.
So we can set and get parameter
values using dot notation
as you would expect.
Now, we can also set
values in such a way
to prevent a feedback
loop, which is useful both
for performance and for
keeping the UI smooth.
We don't want to be
getting notifications
that are slightly different
from what we are doing
as we move a slider.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
from what we are doing
as we move a slider.
So the set value method
accomplishes that.
And that token is obtained from
a method to add an observer
to a parameter, or its tree,
or a group in the tree.
And when we do that, then
we can get called back
from the parameter.
That's what we see
at the bottom there.
There is the block called
the AU parameter observer,
and it passes us the
address and the new value
of the parameter that changed.
As for the implementation, we
saw this in the Filter Demo.
It has the implementer
value observer
and value provider blocks.
Now, in Filter Demo,
it installed these
blocks on the tree.
It's also possible to install
them at any level of the tree,
even on individual parameters.
I would also like to show
off what we have done
with parameter scheduling
because I think this
is a big improvement
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
over the version 2
API in this area.
We have the host and the
implementation dealing
with things somewhat separately,
but here we have the AU Audio
Unit base class doing some help,
it's helping us implement this.
So the host can obtain from the
AU Audio Unit a block called the
schedule parameter block.
And at render time, it can call
this block to change parameters
in sample-accurate way.
So the first argument to do
schedule is a sample time,
the parameter value
can ramp over time
if the Audio Unit has
advertised it as being rampable.
For example, the
Apple Mixer does this.
And the last two
parameters, of course,
are function parameters -- are
the address of the parameter
to be changed and the
new parameter value.
Now, things are a
little bit different
on the implementation side.
We don't just get a
pass-through call from the host.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We don't just get a
pass-through call from the host.
Instead, the base class is going
to fetch the internal
render block, which we saw
in the Filter Demo,
and it's going to pass
that render block
the real-time events
that pertain only to
that render cycle.
So the base class is maintaining
the full schedule of all
of the pending scheduled
parameter changes
and just parceling out
the relevant pieces of it
to the Audio Unit
at rendering time.
So that's parameter scheduling,
and we have done the exact
same thing with MIDI events.
The host fetches a block
from the Audio Unit
before starting to render.
It calls that block
at render time.
Now, you will notice here we
have added a function argument
called cable where,
in the version 2 Audio Unit API
there is only one MIDI cable
with 16 channels, now we
have 256 virtual MIDI cables.
So if you have an Audio Unit
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So if you have an Audio Unit
that wants huge sample
banks, you can do that.
You can address them all
on virtual MIDI cables.
On the implementation
side for MIDI events,
this is exactly the same as
for scheduled parameters.
The base class AU Audio Unit is
maintaining internal schedule
and passing events to
the internal render block
through the real-time event list
only during the render cycle
during which they are
supposed to take effect.
So we think this is
a big improvement.
It saves the implementer
a lot of work.
Now, about rendering in general,
we are still using a pull model,
meaning that an output unit
pulls a mixer, pulls an effect,
pulls another effect,
pulls the player.
Audio flows back down
through the chain.
One difference here is
in the version 2 API,
the Audio Unit needs
to maintain some state.
It needs to have a notion of
whether it's getting its input
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
It needs to have a notion of
whether it's getting its input
from another Audio Unit
upstream or a function callback.
Now, in the version 3
API, it's much simpler,
the AU doesn't have to
maintain that state.
This callback, as we
saw in the Filter Demo,
comes from the host, and
it is passed during every
render cycle.
But otherwise, the APIs are
pretty much functionally
identical, and this lets
us bridge very efficiently
between them.
Now, if your host is calling AU
Audio Unit directly to render
as opposed to using AU
graph or AVAudioEngine,
here you will want to call
allocate render resources
as usual and then hold
onto the render block.
You can call the
render block to render.
It looks very similar to
the internal render block.
It's worth reviewing
here some rules
about the audio buffer lists
that appear at render time.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
about the audio buffer lists
that appear at render time.
Now, the host provides an
output audio buffer list,
and in that audio buffer list
the M data pointer can be null.
The Audio Unit must replace this
with an internally owned buffer
and at the same time,
the AU has to promise
that that buffer
will remain valid
until the next render cycle.
This is all, by the
way, exactly the same
as with version 2 Audio Units,
I'm just reemphasizing it
because it's important.
Now, in the render block, we
have some rules, similar rules
but not the same,
about input buffers.
The host provides that
input block, the AU calls it
for input, and when the AU
calls that block for input,
it has to supply valid
audio buffer lists
with non-null M data
pointers to that block.
Now, the host is allowed to
replace those pointers to memory
that it owns and can
promise to keep valid
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that it owns and can
promise to keep valid
until the next render cycle or
deallocate render resources,
and all of this accomplishes
an important goal,
which is to absolutely
minimize copying.
Okay. Here is the scary
slide for those of you
who are writing code to
run in rendering context.
So audio rendering
almost always happens
in a real-time thread context.
And this is a restrictive
environment
because we can't
allocate memory,
which means that we really
shouldn't even be calling
dispatch async, for instance.
And in fact we can't make any
call at all which might block,
for example, taking a mutex
or waiting on a semaphore.
The reason is if we do block and
we block for any length of time,
then the audio rendering thread
in the system will
miss its deadline,
and the user will
experience that as a glitch.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So we have to be very careful
when both using and calling or,
I'm sorry, both using
and implementing
these render blocks.
So you will see in the Filter
Demo how we went to some lengths
to not capture our Self object
or any other Objective-C
object for that matter.
In that block, we avoid
the Objective-C runtime
because it's inherently unsafe.
It can take blocks.
Unfortunately, the Swift
run-time is exactly the
same way.
So this is why in the Filter
Demo you'll see we have C++
objects and we capture
pointers to those C++ objects.
Now, if you are allergic
to C++, you are free
to do the same thing
in plain vanilla C,
although I'm not sure why
you would want to do that.
Enough scary stuff.
I would like to bring
up Alec Little now
to show Audio Unit extensions
in Apple Music creation apps.
[Applause]
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[Applause]
>> ALEC LITTLE: Thanks, Doug.
I'm Alec. I work on the music
creation applications for Apple,
things like GarageBand
and Logic.
We are excited about the
new Audio Unit extensions.
We think they will add real
power and creative possibilities
for developers and users.
So I wanted to talk a little bit
about what some of
our plans are.
So first of all, we plan
to support the Audio Unit
extension, of course,
in all of our main applications.
So that's GarageBand iOS,
GarageBand Mac, Logic Pro X,
Logic Pro 10, and Mainstage.
So what I thought we would do
today is look at some examples,
some pretty pictures, if you
will, from GarageBand iOS.
And this is just
preliminary stuff,
but I think it will
give you an idea of kind
of what we are planning
to do as a host
to support Audio
Unit extensions.
So first of all, we are going
to be supporting AU instruments.
So the example I'm going to be
giving is about how we're going
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So the example I'm going to be
giving is about how we're going
to implement those
AU instruments.
So first of all, just a
little graphic to explain
about what we are going to
do, something simple here,
but GarageBand is
going to request
from the View Controller
a custom UI dimension
that we will talk
about in a second.
Pass MIDI events over to
the Audio Unit and then
of course receive audio
back over the audio bus.
On to the promised pictures.
So GarageBand, our main
launch screen launches
into what we call our
touch instrument carousel.
We have all of our touch
instruments here: keyboards,
drums, smart guitars,
all of those things.
Then if you see on the left
there is this container,
and if GarageBand is seeing
that there are Audio Unit
instruments installed
on the device, we will
show that container.
I can swipe over
to that container,
and that's where my
Audio Units live.
If I tap on it, then
we will see all
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
If I tap on it, then
we will see all
of the Audio Unit instruments
installed on the device.
Now, if I tap on one
of those instruments,
we will show a big gray
box and a keyboard -- no.
We will show your
custom UI up there
in that nice embedded
view inside GarageBand,
and I think that's the coolest
part of this whole thing is
that we get to show
the actual identity
of your Audio Unit
inside our host there.
And we will be providing our
standard GarageBand keyboard
for you to be able to play that.
We will be recording the MIDI
and receiving the audio back.
So that just kind of brings
up a pretty obvious point.
When you are providing
these custom UIs,
make sure you are not putting
any sort of custom, you know,
MIDI controller-type
device there
because we won't be capturing
your MIDI in GarageBand.
Just a quick look what we
plan to do on the phone.
Is, again, we have a lot more
limited screen real estate
there, so there is a button
in the top-right corner,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
there, so there is a button
in the top-right corner,
which pulls up the
controls view,
and there is the custom UI.
So all of the control there,
and a little bit of room
for the user to still play on
the keyboard down at the bottom.
So here is the most important
slide probably from me today,
and this is the dimensions
we will be requesting
from the View Controller.
So you want to have your
stuff look good in GarageBand,
pay attention to
those dimensions,
and we will do something
pretty cool together.
So, again, we think it's going
to be really exciting to be able
to see and have the
users be able to play
with the actual interface
of your Audio Units
right inside GarageBand,
and we are really excited to
work with you guys to come
up with really cool stuff!
[Applause]
>> DOUG WYATT: Thanks, Alec.
So I imagine you may have
questions at this point.
I would like to try to
anticipate a few of them.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
I would like to try to
anticipate a few of them.
What about inter-app
audio on iOS?
This is a couple
of years old now.
And there is a number of
apps that have supported it.
Well, from our point of view,
this uses a small subset
of the version 2 API, and it
doesn't support a bunch of thing
that people have asked us for,
like parameter support,
presets, and so on.
And we get these requests
and I think, well,
we should have a
full plug-in model,
which is what we have now.
So while we are not
deprecating inter-app audio,
we do see the way forward to add
all of these missing features is
through Audio Unit extensions.
Now on OS X, you
may be wondering
about the compatibility story
if you have existing
hosts and Audio Units.
The bridges should save
you from a lot of pain.
They are compatible, and we
have gone to a lot of work
to make things work
as well as we can.
So I would recommend that you
update to version 3 when you can
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So I would recommend that you
update to version 3 when you can
or need to for some feature.
Maybe you want to redo the
way you handle MIDI events
or scheduled parameters,
for example.
We do have a shortcut
for porting.
It's called AU Audio
Unit v2 Bridge.
This is an AU Audio Unit
subclass that is implemented
on top of a version 2 AU, so you
might be able to start with that
and evolve to a more fully
native implementation.
And as Michael mentioned
earlier,
version 3 Audio Units are
largely cross-platform
between iOS and OS X.
The signal processing code
in AU Audio Unit should be
absolutely fully portable
because there are no
UI implementations,
or dependencies, rather.
AU View Controller derives from
either UI or NSViewController
so you get a little bit of
insulation, but you will
at some point run into
platform-specific UI.
We are running out of
time, and there is way more
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We are running out of
time, and there is way more
than I could possibly go
through in an hour here.
At this point I would like to
refer you to your header files
in the Audio Unit framework.
You do need to link AudioToolbox
for historical reasons.
The main header file is AU Audio
Unit.h but there are others,
AU View Controllers in the
Core Audio kit framework,
and as we mentioned, there
is the AVFoundation framework
with AU Audio Unit component.h.
We have good HeaderDoc in all
of these, so I would like to
urge you to check them out.
Finally, if you would like to
use our Audio Unit's logo --
there is a white version, too --
you can go check out
this link for a license.
And that brings us to the end.
So we have seen how we now have
for the first time a full
plug-in model for audio on iOS,
and it's the same on OS X.
And, again, it lets you sell
Audio Units in both the iOS
and OS X App Stores by
packaging your Audio Units
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and OS X App Stores by
packaging your Audio Units
as app extensions.
We looked at the simple host
applications that are possible
with AVAudioEngine, and that's
illustrated in our new sample,
Audio Unit v3 Example.
So I would like to encourage
you to write bugs as you work
through the sample code,
read the documentation,
and work on the great AU hosts
and implementations
that I know you will.
So for more information, this
was our session yesterday,
so thank you very much!
[Applause]