Transcript
>> Good morning.
My name is William Stewart and I manage the Core Audio
group and we are doing three sessions this morning
in this room on audio, primarily on iPhone.
We're also covering some general audio topics that
are relevant for the desktop as well as iPhone.
The set of frameworks and services that we provide are
quite extensive and cover a number of different areas
and we'll be going through some of these in the talk.
The Media Player API is just a general kind of remote access
for the iPod application and the media library and the iPod.
And the iPod itself uses the same sets of APIs and
frameworks that we're discussing with you today.
So it's all, you know, what Apple uses itself
to implement its features is, of course,
the same things that you get to use as developers.
OpenAL is an industry standard, I
guess you'd call it, for doing games
and we support that on the platform for game audio.
AV Foundation made its debut last year with some
very simple AV audio player and recorder objects
and this year it's become quite an extensive
framework with a collection of video.
And there's a whole series of sessions on the new classes in
AV Foundation but there is some specific audio functionality
in this framework as well and Allan
will be going through that in a moment.
And all of these technologies are really built
on a collection of services that are rendered
through Audio Toolbox and that's really the primary
services that my team delivers to the platform.
That includes services for reading and writing
audio files, for converting data formats,
for using audio units that provide
processing, mixing, all kinds of things --
basically the collection of tools
that you need in order to do audio.
So that's a very general introduction.
There will be some more data overviews
through the sessions this morning.
The sessions that we have is this one and one of the things
that will be covered here is how your
application integrates with the rest of iPhone OS.
So it's primarily audio session and managing the
resources of the platform and how you can best use them.
What we thought we'd do in the second
session is take a step back from talking --
rather than sort of focusing specifically on APIs, we
thought we'd rather take it from a different angle.
And so what we're doing in that session is
looking at: what are the fundamentals of audio?
If you're talking about digital audio, not just on
our platform but on any platform, what does that mean?
What is Linear PCM?
What is AAC?
How are these things different?
How do we express them in our APIs?
But really more like: what is the
fundamental features of these things?
And our APIs are very fundamentally shaped by what audio
looks like and as a data format in some of the constraints
in terms of time and resolution and everything that
we deal with by dealing with this media format.
And then the last session, Audio Development for iPhone OS,
is really taking a more detailed
look at how to use audio units.
So what do audio units look like for your application?
How do you interact with them?
And we're also taking a little bit of a forward-looking
stance at that and looking at some of the general ways
that you can deal with more complicated
processing demands with IU graphs and so forth.
So that's enough of me talking.
I'll get Allan to come up and he'll
begin his discussion on AV Foundation.
Thank you.
[ Applause ]
>> Allan Schaffer: Great.
So thank you, Bill and good morning everyone.
The AV Foundation has a number of high-level classes
that you can use for audio playback and recording.
And this is where we're going to spend
most of the time in this session.
So I'm going to be talking about the audio player, which
lets you play back audio from a file or from data in memory.
I'll talk about the recorder, which lets you record audio,
capture from the microphone and record that to a file.
And then I'll talk about the audio session, which,
as Bill said is going to let you manage the
audio behavior of your application on the device.
There's a fourth class that actually
I'm not going to be covering
in this session but it's worth taking a look at as well.
We covered it yesterday in the AV Foundation
sessions for video and that is the new stream player.
Now, a lot of the new functionality in AV
Foundation has been geared towards a lot
of very expressive video functionality.
And so that class is part of all of
that but it can also be used for audio,
to either play audio from a local
file or to stream it over a network.
So I'm just going to jump straight in.
Let's talk about the AVAudioPlayer.
And this is really a very simple
class for you to use to play a sound.
It supports a variety of different file formats and the
ones that are supported by the audio file service is API.
So that's things like caf files, m4as, mp3s, and so on.
And the class provides a number of
just basic playback operations --
to play a sound, stop, pause, move
the playhead around, and so on.
And with this object, if you want to play multiple
sounds simultaneously, you can do that as well --
what you do is just create multiple instances of the
object and have each one controlling the different sounds.
There's also a number of properties that I'll go
through in just a moment but things like volume control,
you can enable metering, you can have the sound be looping
as it's played back, and the few new features in iOS 4 --
the object now supports stereo panning from left to right.
It also synchronizes playback if you have
multiple instances playing simultaneously.
So right away, I'll just jump into some API here.
To get started with this class to instantiate an object,
you just -- we'll call init with contents of URL.
The URL needs to be a local file that's in the sandbox
for your application or you can create it from an NSData.
And one quick side note before I go on: All of the code
snippets that are on the slides in this talk are available
on the attendee website so you don't
need to worry about writing them down.
You can just go and download them right after
the talk or go ahead and do it right now.
Now, there's a number of properties on the
AVAudioPlayer that let you control playback.
And you can either set these up before you begin playing or
with a number of them, actually, you can just change them
as the player is playing a sound,
so you can change the volume --
here I'm setting it to 100% of the current output volume.
You can change the panning -- here I
have it set all the way to the left.
If I had it to 1.0, then it would
be all the way to the right.
The number of loops here is something also you can control.
So 0 means no loops, -1 means loop indefinitely,
or you can have a specific number of times
that the audio will loop back after
it's played through once.
You can have direct control over the playhead
as well with the current time property.
And so if you want to implement something where
you are scrubbing around in an audio file,
you would just be changing the value of this property.
Or if you want to reset the playhead
to the beginning, you set it to zero.
And there's a delegate as well
that we use for notifications.
And then other properties, some that I mentioned --
so to be able to enable metering of the playback,
you can find out of the duration of the audio
file that you're playing, the number of channels,
and the state of the player as it's playing your sounds.
Now, the playback controls here are really
simple, there's just these four controls
that you'll be using all the time with this object.
PrepareToPlay is actually probably
one of the more important ones.
This is going to get the player ready to play
a sound for you with absolutely minimal lag.
So what this will do is allocate
the buffers that the player is going
to use internally and prime those buffers with your data.
And that way, when you go to invoke the play
method, it can happen nearly instantaneously.
So the play method just starts playing your sound now.
And so -- if you later on you pause or stop the sound, the
play method will resume from the point that it leaves off.
And that's maybe an important note to make is that if
you had expected, after you stopped playing a sound,
for the playhead to go back to the
beginning, that actually isn't the behavior.
The playhead stays where it was, just like a tape deck.
And so if you want to go back to the beginning,
you would reset the current time to zero.
Pause is going to pause the playback but with
Pause, the player stays ready to resume again;
the cues and the buffers are still
going to be allocated and ready to go.
And that's the difference between Pause and Stop.
With Stop, the cues are disposed
of and the buffer is disposed of.
So if you want to restart playing after you have
stopped, after you've invoked the stop method,
you would probably call prepareToPlay
and then later on call play.
Another element of the player class
are some delegate methods.
So these will be invoked when certain events happen.
And probably the one that's very important for
you to implement is when the player is finished
and that's called audioPlayerDidFinishPlayingsuccessfully.
And in that method you might clean up, you might change the
state of your interface, and take care of other things just
to indicate to the user, "Okay,
the sound is no longer playing."
Then there's a number of other
delegates that you can implement.
If there's a decoding error in the file that you
played back or if interruptions began or ended --
interruptions are things like a
phone call came in for example,
and I'll talk about interruptions a
lot more towards the end of the talk.
So let's just put this together and look at playing a sound.
So here in this method, I'm being passed in the
URL of a local file; I create my player object;
I next set the passing appointer to that URL; I set up
a delegate for any of the notifications that I may want
to have fired later; prepare the
player for playback and hit Play.
So all of this is really -- this is just to show that
this is a very simple class but a great way for you
to get started with audio playback in your app.
So that's the player.
Let me now go just about as quickly through the recorder.
Another simple class -- this lets you record audio to a
file and its behavior actually is that it will either record
until you stop it by calling it stop method or you
can set it up to record for a specific duration.
And it supports a variety of different encoding formats,
I've listed a bunch here -- AAC,
ALAC, Linear PCM, and so on.
AAC is interesting though because we have hardware
support for the AAC encoder on certain platforms --
the second generation and third generation iPod
touch, we have hardware support for it on the iPad,
the iPhone 3GS, and of course on the iPhone 4.
Now with the AVAudioRecorder, the API here is really
just the mirror image of what you saw with the player,
so I won't go through it in quite as much detail.
You initialize the recorder with a URL to a local file.
One difference, though, is this next
parameter: the settings dictionary.
I'll cover that on the next slide
so I'll come right back to that.
There's a number of recording controls that
you can manage, so prepareToRecord, Record,
or Record for a particular duration, then Pause and Stop,
and sort of your predictable properties that you can get
about the state of the object and
the state of the recording.
Now about that the settings dictionary.
So when you're recording, you need to specify exactly what
format you want to record into, what sample rate to use,
the number of channels, and then perhaps
format specific settings as well,
like for Linear PCM you'd specify
the bit depth, the endian-ness.
For certain encoded formats, you might
specify the quality or the bit rate and so on.
So let's take a look at that.
Now this looks like a lot of code but
really, it's actually very simple.
All I'm doing here is setting up a dictionary containing
key value pairs for each of those encoding settings.
So on the left here, I'm setting my format to AAC,
the rate to 44100, the number of channels to two,
the bit rate to 128K, and the audio quality to the maximum.
So all of that is just being packed into an array and I
give that array now and create a dictionary from that.
And then that dictionary is what I pass
when I initialize the audio recorder.
And so now it's all ready to go.
I can call its methods to prepareToRecord and start
recording or record for a particular amount of time.
Okay, so I know that was pretty quick through
those objects but they're really just very simple.
The thing is, though, that they're very feature-rich so
they do suit a lot of the basic needs of audio developers
who just want to play and record some sounds.
If you saw the Quest demo on Monday
or yesterday, for example,
we're using this API to play all
of the game background soundtrack.
And so, you know, it's just a very,
very capable API for doing that.
And really, it's recommended as the
starting point for you in most cases.
You know, it's the good starting point unless your
requirements go into more complicated uses for audio.
So if you need access to audio samples, for example for
processing, then you might use the audio unit's API,
which Murray will be covering in the third session.
If you need to do spatial 3D positioning of audio
sources in a game, then OpenAL is perfect for that.
And so you would probably choose that API in that instance.
And if you need to do network streaming, then
you might use the new AV Player or you might go
into the audio file streaming services API.
Okay, where I want to go next, though,
is into audio session management.
And this is really an important topic for developers
to all get absolutely right in their applications
and that's why we put so much focus on it.
And really, I'm going to dedicate
the rest of this talk to this topic.
So the idea here is that this is how you can manage
the behavior of the sounds in your application
and make them behave according to both the expectations of
the user for the kind of application that you're writing
and to be consistent with either built-in applications
or other applications of the same time of app as yours.
What you're going to do with this API is to categorize
your application into one of six possible categories
and then your app's audio is going to follow the
behaviors that are defined for that category.
Then this is also the API that will let you
manage some of the shared resources on the device
and do things like mix with background audio.
It's how you'll interact with interruptions if
they occur, say again, if a phone call comes in
and it's how you can handle changes in the routing if
the user were to, say, plug in or unplug a headset.
Now there's actually two APIs here
that are relevant to what we're going
to be talking about: a high-level API and a low-level API.
So the high-level API is the AV audio session class.
It's an Objective-C class, part
of the AV Foundation framework.
And really, it wraps up all of the
most commonly used functionality
that you need to manage with the audio session.
Then there's also a lower-level API called Audio
Session Services and that's part of the Audio Toolbox.
And that's really all of the implementation
that we expose to you.
C-based, a lower level and has a bit more functionality.
But what's interesting to note is that it
is possible and quite okay for you to mix
and match between the high-level and low-level APIs.
In fact, what's quite typical is you might set up your audio
session using the high-level API and then maybe just drop
into the low-level API to set some overrides or
other things that aren't exposed at the high level.
So there's five basic tasks that we're going to go
through for the remainder of the session here to talk
about with AVAudioSession: We're going to set up the
session and configure its delegate; we will choose --
very carefully choose and set -- and audio session
category; we'll go active and we'll talk about that;
and the things that are new there in relation to iOS 4;
then I'll talk about how we handle
interruptions and handle route changes.
So first setting up the session -- very easy.
The audio session instance is just a
singleton object for your application.
So you just retrieve a handle to that.
You'll set up a delegate for any notifications
that might occur, like an interruption.
And this is the place where you might
request certain preferred hardware settings.
Just for example, in this case I'm
requesting a sample rate of 44100.
But now the second part is really -- this is
the most important part of using this API.
And it's not a line of code, it's a choice
that you have to make for your application.
You will choose instead a category based
on the role that audio plays in your app
and the kind of application that you're writing.
And there's six possible categories: playback;
record; play and record; audio processing;
and then two more -- ambient and solo ambient.
So let me show you how these differ.
So I have the categories listed here on the left.
Their intended usage will be a column that we
build out as I talk about this and then a number
of the behaviors are listed in the table as well.
And it's things like whether each
category obeys the ringer switch, meaning,
is your audio silenced if your user
flips the ringer switch to silent?
Is your audio silenced if the user locks their screen?
Does this category allow your sounds
to be mixed with others?
Does it use input or does it use output?
And is it allowed to play in the background if your
application is transitioned into the background?
So let's have a look.
Okay, first with playback.
This is probably the most straightforward of all of the
categories because it's intended for such an exact purpose.
This is the category that you should choose if your
application is an audio player or a video player.
So if that's the primary purpose of your application is to
output media, then you would choose the playback category.
And you can see that the behaviors here, in a way,
are very similar to what you see with
the iPod application on the device.
This does not -- applications that are using this category
are not affected by the state of the ringer switch;
they can continue to play if the user locks the screen.
Now MixWithOthers is not enabled
by default by saying optional here,
it is possible here for you to optionally turn it on.
This is an output category.
And the last one.
With this category, it will allow your application to
continue to play audio through a background transition
if you have set up the audio key for the
UI background modes in your info plist.
Now Record is very similar.
But of course, its intended usage is for audio recorders,
applications that are doing voice
capture, that sort of thing.
The behaviors, though, are right
across the board mostly the same.
Obviously this is an input category
that uses input instead of output.
But it will also survive through a transition
in the background if you have set that key.
Play and record.
Well, this one is essentially combining those first two.
So this is intended for applications that are
doing voice over IP or voice chat types of apps.
You can optionally mix with others,
just like with the playback category.
But with this one, this is using
input and output simultaneously
or enabling you to do that, so both of those are shown.
And if you choose this category also, your
application can go into the background.
One important difference with this category,
though, compared to what we've seen, say,
with the playback category is the default audio route.
So with the playback category, by default
your output will go through the speaker.
With the play and record category, your
output will go on a phone to the receiver,
which is the speaker you hold up to
your ear when you're on the phone.
Then the audio processing category.
All right, so this one is used for offline conversion
of audio file formats, offline processing of audio data,
and you can see that actually many
of the behaviors are similar
but it's not using either input or output for any sounds.
All it's doing is doing this processing in memory.
Now, one thing about -- a special note about this,
about how I say that it's allowed in the background.
So yes, if you set up your application to use this
category, the processing can continue in the background.
But unlike the previous three, just setting this category
alone does not enable your application to transition
into the background and keep running; you would have to
use one of the other ways of going into the background.
So for example, maybe you would just be
asking for extra time to do processing
and that's how you would transition into the background.
Now, these next two are very similar to each other,
so I'm just going to put them up simultaneously
so you can see the difference: ambient and solo ambient.
But the purpose of these two categories is very
different from the previous four that we talked about.
The top four are really intended for applications
whose main purpose is very audio-centric, right?
A playback app.
You know, an audio player, a voice recorder, a VoIP app or
something doing audio conversion, so very audio-centric kind
of purpose of the way that application uses audio.
These other two are really intended
for much broader purpose kinds of apps.
So games and productivity apps or utility apps
would probably choose these latter two categories
and the reason is because of the behaviors that
they enforce are consistent with the behaviors
that users expect for that kind of application.
So what you can see is, well, these
both obey the ringer switch.
That means the user is playing your
game or using your to-do list app
or whatever your app happens to
be that's using in this category.
If they're doing that and they hit the ringer switch, well,
the audio will be silenced, which
is exactly what they expect.
And again, it's because the purpose -- the usage of audio
-- is not critical to the purpose of the application.
It's perfectly expected that you can play a game
with the sound turned off, or many games at least.
It's expected that you can use your
productivity apps like Mail or Safari and so on
and have the sound turned off in those as well.
They will both obey the ScreenLock as well, that means
that audio playback will stop if the user locks his screen.
I'm going to jump over.
They both are output categories and neither
of these enable your application to transition
and continue to play audio in the background.
But the difference between them
actually is this MixWithOthers parameter.
And so I'm going to be talking about that in a
little more detail, really, just coming up next.
But it has to do with whether your application
needs access to a hardware codec or not.
With MixWithOthers with ambient that means you would
use that for applications that don't require access
to a hardware codec, don't need to use it.
Excuse me.
With solo ambient, that's the category
you would choose if your, you know,
game or productivity app does require
access to a hardware codec for decoding.
Okay, so here is where we are setting category.
So we've gone through the table now, we've made
our choice for the specific kind of application
or applications that you guys are writing.
And in this case, we're choosing the ambient category.
We go on to the next part here
and set our session to be active.
So to do that, all we do is call set
active, tell it yes, we are now active.
And once we're active, now we can play sounds or
record sounds if we have chosen the record category
and go on to set up our audio APIs,
handle interruptions, and so on.
Like we have now asserted that we want to make
use of the audio functionality on the device.
But let me go back now and talk about going active.
So it would be typical for most applications
to just go active when they start up
and stay that way for the remainder of the app.
But there's a few classes of applications that should not
do that and it's because of the interaction that they have
with other audio that might be playing in the background.
So with a voice recorder application, for
example, or a VoIP app or a Turn-by-Turn app --
well, all of those should have different behaviors related
to music that might be playing on the iPod when they start
up or there may be an email notification
sound and so on that might happen.
So with those kinds of applications, you want to
be a little more clever about when you go active.
And the story is that you should
only go active in those kinds
of applications while you're actually doing those things.
So in a recorder app, you would only
go active once you actually start
to record, not just when the application starts up.
And you would go inactive as soon as you're done recording.
On a VoIP app, you would only go active while you're
on the call and then go inactive when you're done.
And in a Turn-by-Turn app, well, there's some
specific behavior that we would want there,
which is that let's say the user had been playing music
in the background when they ran your app and now it's time
for you to announce the next turn, "turn left."
What we want to happen there is for the iPod music
to be ducked -- for it to lower its volume --
you make your announcement and then you go
inactive to bring the iPod music back up again.
The same would be true if it was audio playing from
a third-party application in the background as well.
So let me just show you in specific
how you might go about that.
I want to focus on the VoIP app and the Turn-by-Turn app.
So again I said, with the VoIP app, you would
go active when it's time to start the call --
and this is going interrupt sounds that
might be playing in the background --
and then go inactive when the call is over.
And what we can do, there's actually a new
method in iOS 4.0 called setActivewithFlags.
And what that can do is if you set that flag to
notify others on deactivation, then that other,
the background process that was playing
audio can be notified when the call is over.
And it will be told, "Ah, okay, the call is done.
You can go and resume your audio now,"
with Turn-by-Turn navigation types of apps.
As I said, there's a couple of things you want to do here
so that the other audio gets ducked while you
make your turn announcement and then comes back.
And this is in three steps here.
So this is the first step, the setup.
The first thing that we're going to do, of course, is just
choose the right category to put our application into.
So the main purpose of a Turn-by-Turn app
would be to announce these instructions.
So we would need that to happen regardless of the state
of the ringer switch, the screen locking and so on.
So it will choose the playback category.
Now next, though, we're going to
set an override on that category
and this is the part where I said was optional before.
We're going to set this override to enable our
category -- override our category -- to MixWithOthers.
So "others" being, say, the background music
coming off of another app or maybe from the iPod.
And then third in this step is that we're
going to enable other mixable audio --
or say that the other mixable audio
should duck when we go active.
And so that's what's going to lower its volume when we go
active and then when we go inactive, it will come back.
So here are those two parts.
So when it's time now for us to make the
Turn-by-Turn announcement, we'll go active,
we have some audio player ready to go with the
sound of that announcement, and we play that.
And then whenever that is done, when
it's finished making the announcement --
for example, if we were using the AVAudioPlayer,
we could do that through the delegate
-- that's when we'll go inactive again.
But now I've been talking about mixing with background
audio so let me go into a little more detail about that.
And there's really a convergence
of a few different topics here.
So your application might be playing a variety of
sounds, it might be taking advantage of a hardware codec,
it may be using a software codec, or just
playing straight through with the mixer.
Another app might be running in the
background and also playing sounds.
So it might be the iPod application is running
or it could be a third-party application
now that's running in the background.
And so we have to sort of arbitrate:
Well, what's the user going to hear?
And it's going to depend on things that
are going on in both of the applications.
It will depend on what category
both of the applications have set.
And related to that, it will depend on whether either
one of those have enabled MixingWithOthers if they happen
to choose the playback or play and record category.
So all of this is ending up to define something that I call
"mixable" or "nonmixable" as a state of your application.
If your application -- now by default, the only mixable
kind of application is one that chooses the ambient category
but if you choose the playback or play and record and
override it to MixWithOthers, then those become mixable.
Otherwise, everything but ambient would be nonmixable.
So let me show you this just in a
picture what's going to happen here.
So let's say that this is your app, the
foreground app, and you're playing in the AAC file
or some of you might be playing an mp3 file.
But just for the sake of discussion, I'm going to use AAC --
but just bear in mind that the same
things would apply if that were the case.
Now, if you had put your application into a category that
is nonmixable, then you will be able to take advantage
of the hardware codec to playback that compressed track.
It will go into the mixer and out to the playback hardware.
Now, if you had chosen a mixable
category, then what will happen is
that actually your sound is going to be decoded in software.
So we have mp3 and AAC and a number of other
software decoders and those will just run on the CPU
to code your audio, have that go in through
the mixer and out to the playback hardware.
But okay, so this is the basics and you guys
are probably already familiar with this part
but now what happens if there's a background application?
The thing is, it's the exact same story.
So a background application -- let's say that it's
playing a music soundtrack of its own, it has an mp3 file.
If it chooses a nonmixable category, then it will
be able to take advantage of the hardware codec
and its sounds will play through the mixer,
they will be mixed with your music
soundtrack, and out to the playback hardware.
The same thing again.
If they choose a mixable category -- if this is the case --
and you're mixable, too, then both of these will be decoded
in software and play out through the playback hardware.
But there's one case that you need to be thinking
about: What if you've chosen a nonmixable category
for your application, meaning you're asking to use the
hardware codec and there's something else that's going
to be running the background, maybe
when your application started,
there was already something there,
what's going to happen with it?
Well, the result will depend on the
category that the background app chose.
So if they chose a mixable category,
then they'll get a software codec
and both of these are going to be mixed together.
So even though you have chosen a nonmixable
category for your app, since they have decided
to choose a mixable category, essentially
they're playing nice.
They're saying, "Okay, I can pretty much mix with anything."
And so both of these sounds will be heard; they'll be
mixed in the CPU and sent out to the playback hardware.
But if they've chosen a nonmixable category also,
then the sounds from the background app are going
to be silenced when your application goes active.
Those are the different cases now
for mixing with background audio.
Now, what's interesting, though, is that there's actually
a way for you to detect in advance what might happen
and therefore for you to decide maybe of a different
category, depending on whether something is already there.
And you may have seen this in some apps that are
doing something like this: they'll say, "Hey,
do you want to play the game sounds, like, do you want
the game music soundtrack to play in the background
or do you want the iPod or a third-party
app soundtrack to play in the background?"
And it depends, the user either says yes or no.
If they say yes, well, then a couple of things happen.
If they say yes, this is a game, so we would usually
either choose the ambient or solo ambient categories.
So in this case, they say yes, it means,
"Okay, then that means they want my game music
and my game music is an mp3 file or an AAC
file, so it's best to use the hardware codec.
So I want the hardware codec."
So I'll use solo ambient and I'll play my game soundtrack.
And if the user says no, well, then I'm
not going to use the hardware codec.
Maybe all I'm going to do is play sort of
the incidental sounds in my game, you know,
the bullet sounds or just a momentary sounds.
But maybe those are something that would be okay to
decode in software or they may be Linear PCM, WAV,
or AAIF files that can just be mixed directly.
So in that case, I'll choose ambient,
saying, "Hey, I'm fine to mix with others
and there's something else playing in the background.
So I won't play my soundtrack, I'll
just play my incidental sounds."
So all of this is fine.
This is a perfectly good way -- logic -- to use for defining
your app but one part that just might be not be necessary is
to leave this choice as something that the user has to
figure out when they first start the application up.
Instead, you can just detect this programmatically.
So there's an audio session property
called OtherAudioIsPlaying.
And it will come back you know, 1 or 0.
And you can use this to decide whether or not
you play your game music soundtrack and really,
you'll use this to decide what category to use,
whether or not to enable MixingWithOthers or not.
And so I show you here just how
you can get at this information.
So AudioSessionGetProperty, I pass that
token above and I get back the result.
Now one change that's important to note in iOS 4 is
the behavior where your application may be suspended.
So prior to iOS 4, it would be very typical to
just put this in the beginning of your application.
Maybe application did finish launching,
you would check this,
you'd have a value that was valid for
the entire run of your application.
But now something that might happen with, say,
a game is that user might come into your game,
start it up, and then realize, "Oh, you know what?
I want to listen to my first person shooter
soundtrack instead of the game sounds."
And they suspend your app, they go over the iPod, they start
up their playlist, and then they come back into your game.
So you don't want to have already
made your decision about this,
you want to recheck it every time your
application comes back from being suspended.
So check again in applicationDidBecomeActive.
Okay. One more thing, we've been talking
a lot about the behavior about mixing
and how to detect that when your session goes active.
We talked about the behavior with, like, recorders and
VoIP apps and Turn-by-Turn apps, when they go active.
There's one more thing about going
active in a tip or a change that occurred
that I want to just bring to your attention.
It's a behavior change with the MPMoviePlayerController.
So some of you guys might be using this to
playback video; very simple class for that.
But its behavior has changed in
relation to your audio session category.
So prior to iPhone iOS 3.2, the movie player had
its own session -- playback was its category.
And so that would interrupt your session
potentially, it might silence other audio
because that playback by default is a nonmixable category.
And so this could have a number of effects.
Well, now in iPhone iOS 3.2 and above -- so iPad
and then all the devices that support iOS 4 --
the movie player controller now uses your audio
session; it just inherits whatever setting you made.
And so this is actually nice.
It's something now where the movie player's
behavior is now made consistent with your app.
But of course, you have to be just aware of this
change in case you were relying on the movie player
to silence other sounds or do something that was
kind of a side effect just of you playing the video.
Now, if you want to go with the default
behavior -- sorry the new behavior --
then you do nothing, it's just the behavior's changed.
But if you want to revert back to the
way it was prior to iPhone iOS 3.2,
there's a property on the movie player
object if you set that to false.
Use application audio session, you set that to false, then
the movie player will go back to choosing its own category.
But all right.
So folks, we have gone through
three of the five basic tasks.
We've done setting the session in delegate, we talked
about making the right choices as far as your category,
and then going active and all of the
effects that going active can have.
So two more things to talk about and
that's interruptions and route changes.
So let's go into interruptions.
The thing to understand here is that your
application's audio might be interrupted at any time.
So it could be interrupted by a phone call, a
clock alarm, if you're running in the background,
it could be interrupted by a foreground application.
So what will happen if you are interrupted is that your
session is just made inactive and whatever you were doing
with audio is stopped -- if you were playing, it's no
longer playing; if you were recording, then that is stopped.
And it just -- this just happens to you -- it's not a
request that this is about to happen, it's just done.
So what you can do -- there's certain steps,
though, that you can take in reaction to that.
Of course, you might update your user
interface to reflect that this has happened.
But more than anything else, you're interested
in what happens after the interruption has ended.
So if the user has declined the phone
call and comes back into your app,
you're interested in getting your audio restarted again.
And there's a number of other cases as well
where, you know, you just need to be --
all that you're really wanting to do is get
back up and running after the interruption.
So there's a few different things that you need to do.
So the first is let's say that we've set up our audio
session and we've implemented these two delegate methods.
Begin interruption and then this one is new
in iOS 4, it's endInterruptionWithFlags.
The old delegate is still available
as well, just endInterruption.
But this lets you get a little bit more fine-grain control.
I'll get to it in a second.
So when the interruption begins, that
means the phone call has come in.
You know, as I say, the playback has
stopped, you are already inactive
and so what you really should just do is change
the state of your user interface to reflect that.
If you're a game, you would probably
go to your pause screen.
If you're an audio player, well, you would
change your playback icon from whatever it was
to something to say, "Okay, restart again."
Let's say that the user declines the phone
call and is now back in your application.
Well, now the interruption has ended and there's this
flag that can be passed into you that will tell you,
based on various characteristics of the interruption
whether or not your session should resume, okay?
Whether you should start playback,
restart playback, or restart recording.
And assuming that that's the case, meaning if you're past
the flag -- AVAudioSessionInterruptionFlag should resume --
then now you can just fire everything back up.
So you will set your session as active again, you
can update your user interface, and resume playback.
So that actually, though, is just the general case.
There's actually a few instances depending on if
you might be using one of the lower-level audio APIs
from the Audio Toolbox or using
OpenAL that you need to take care of.
So let's start with OpenAL.
Now OpenAL has this concept of a context that is analogous
to the position of the listener in the OpenAL's world.
And the context does not survive through an interruption,
does not stay current through an interruption.
So what you need to actually do is when the
interruption begins, if you are using OpenAL,
then at that point you will want to invalidate the context.
So you do that by calling alcMakeContextCurrent and
just passing it nil -- that invalidates the context.
Then when interruption is over, you can end that
interruption and pass it -- well, excuse me --
you implement the endInterruptionWithFlags delegate method,
you check to see if the flag says that you should resume,
and if so, then you can set yourself as active and now
is when you make your OpenAL context current again.
Another case that you have to take a few extra steps is if
you're using the AudioQueue API and an interruption occurs.
So if you're using this API, well, then first of
all, when the interruption happens you probably want
to save your playback head or the recording position just
in case your application actually
gets quit or suspended for later.
But now the decision about how you restart will depend
on whether your application is using a hardware codec
or a software codec for whatever it's
doing -- its playback or recording.
If you're using a hardware codec, then that
attachment cannot survive through the interruption.
So you will dispose of the currently playing
AudioQueue when the interruption begins
and then when the interruption is
over, you will create it back again
and start it again with a new queue whenever you're ready.
Now if you're using a software codec, then it's simpler,
there's no need to dispose and restart -- excuse me --
dispose and recreate the queue, all you have to do
is restart it, for example here with AudioQueueStart.
This actually, I just -- one side note that I put up here.
This can actually get a little intricate though.
So we wrote a technical Q&A that has some snippets of
code that you can take a look at and it's up there,
number 1558, up on the developer website.
And the last topic here is routing.
So the behavior that users expect in
terms of the audio system routing is
that whatever gesture they have made most
recently is taken as an expression of their intent
for where the audio should be routed or saying it
way more simply, "whatever happened last, wins."
So if the user plugs in the headset, then we
take that as an expression of their intent --
that they want the audio to now
be routed out through the headset.
Or if they were using the microphone, they
plugged in a headset with a microphone,
then they want the audio input to be
taken from the headset microphone.
And now usually along with that, there's some behavior
as far as whether the audio should continue or not.
When they plug in a headset, we want the
audio to continue to go -- if it was output --
to just keep playing without pausing at all.
But when they unplug the headset, okay,
that's also taken as an intentional gesture
from the user to change the routing on the device.
We'll route back to whenever it was previously, probably the
speaker and in that case generally we want audio playback
to pause so that they unplug their headset, it
doesn't just start blaring out through the speaker
at them before they have a chance to take care of it.
But okay, so these are the behaviors that users expect
and so there are ways for you to respond to route changes
that will let you implement this in your app.
So really there's just three topics I'm going to mention
here: so it's possible for you to query the current route;
it's possible for you to listen for changes and then in
response to those changes, you might either keep playing
or stop playing as I just said; and it's
also possible in limited cases for you
to redirect where the output is currently going.
So first of all, just getting the current route.
So this is just another case where
you can call AudioSessionGetProperty,
pass it this token that I have
at the top of the audio route.
So this is going to give you back CFStringRef,
just with the name of the current route.
And so you can see here that I'm outputting that to the log.
But it will just tell you, "Okay, the current route is
the speaker, the headphone, the receiver," and so on.
Now, more importantly, though, is that you probably
actually would rather be listening for changes to the route
than to just know where it is now and output
that to the log because you want to react
to those changes accordingly in your app.
And what you do here is -- now this is down in
that lower level API, as I had mentioned earlier --
you're going to set up a C-based callback
using AudioSessionAddPropertyListener.
And basically you will be registering
for notifications of a route change.
And you pass it this token,
AudioSessionProperty_AudioRouteChange.
And then your callback is going to be told the reason
why the route changed, like the user unplugged something
or plugged something in and you'll also be
informed what the route had been previously.
And of course, through what I'd just shown you,
you can also find out what the route is now.
So here's the code, just to get this started.
We're setting up a property listener here.
AudioSessionAddPropertyListener with the
audio route change, we set up our C function
and you can optionally pass it some data as well.
Now what that C function is going to look
like, this is what you would have written.
The final parameter to this function
is this void* with some data.
And that data can be cast to a CFDictionaryRef.
And inside that dictionary is where
you're going find the information
about what the old route was and the reason why it changed.
So that's all I'm doing here, getting those values out.
And then you can act accordingly in your app.
Now, the third one that I mentioned
is that in certain limited cases,
it is also possible for you to redirect the output.
Now we actually limit this in most
cases because for most of the instances
where the output should be rerouted,
we want to leave that up to the user.
If the user has plugged in or unplugged the headset,
then we take as an intentional gesture
from the user to change the route.
And we really don't allow for third-party
apps to interfere with that.
But there's this one case where it may be necessary for
an application to change the route itself and that's
if you're using the category play and record.
Now you might remember from earlier in the talk how I
mentioned that play and record by default will output
to the receiver, that speaker you hold
up to your ear when you're on the phone.
Well, let's say that you're writing a VoIP app and so you're
going to be implementing something where normally that's
where the output would go but you
also want the option of rerouting
out to the main speaker for like a speakerphone mode.
So that's really what this allows you to do: You can set
a property to override the audio route and in this case,
if you were in the play and record category
and you were already outputting the receiver,
you could redirect that out to the main speaker.
But okay, folks, so actually that takes us
through our five topics on the agenda here.
As I said, we set up the session, we chose
a category, and choose that very wisely
for the role that audio plays in your application.
We made the session active and saw all the
effects that that might have with relation
to background audio and mixing and so on.
Then I talked a little bit about handling
interruptions and route changes here in the end.
I just wanted to mention a couple -- as Bill had said,
some sessions coming up so the next session is going to be
about fundamentals about digital audio,
it'll be right here in this room.
And the third session in this room as well is
going to go deep into the use of audio units,
which are awesome for writing applications
that need to do more intense audio processing.
Here is actually my contact information: I'm Allan Schaffer,
so my email address or you can contact Eryk Vershen
who is our media technologies evangelist if you
want to get in touch with us after the show.
And a couple more notes: The audio session programming
guide has a lot of great information that goes
into a little more detail than what I just
spoke about, so be sure to check that out
and really make the right category choice for your app.
And then finally, we have the Apple Developer Forum.
So if you have questions about audio and want to talk
about just among yourselves, among other developers
and along with us, check out the dev forums.
So thank you very much.
[ Applause ]