WWDC2003 Session 407

Transcript

Kind: captions
Language: en
I'm Craig lengthen music and audio
partnership manager at Apple and today's
session you're going to learn my Mac
os10 audio truly does rock so what we're
going to cover today we're going to go
over some developer market opportunities
we'd Apple we're excited about our
advancements in the media music and
audio space recently and we'd like you
to be able to share in our success the
market is really large and I'm going to
go over a couple of slides to show you
just how large that market is and
hopefully show you some new applications
to some new opportunities that you can
tap into with your applications we're
going to cover some audio device driver
changes in Panther and we're going to go
over the core audio architecture design
objective finally we're going to dive
right into the api's and show you how to
get started whether you're a game
developer a music and audio developer or
you just want to input output or process
music or audio with your application
you're going to worry about the api's
that you need to get started today so
market the following is a slide of the
estimated number of music and audio
creators in the United States today at
the top end of the spectrum is the pro
market there's approximately 50,000
aspiring sorry 50,000 professionals who
are doing music and audio day in and day
out making a living at it 100 percent of
those individuals are using accusing a
computer in their creation process now
in the middle end of the pyramid is the
aspiring professional market these are
individuals who aspire to be at the top
of the pyramid they aren't quite making
a living at it but they spend a lot of
money on their gear and it's a very
large market however only about thirty
percent of those use a computer in their
music and audio creation process and at
the bottom end of the pyramid you'll see
the creative individuals fifty five
point nine million individuals in the
United States who are creatively
inclined to do music and audio in some
fashion or another these could be guitar
players people who are just learning to
play musical instrument students DJ's
hobbyists etc but only about five
percent of those individuals currently
use a computer in their music and audio
creation
set and one of the things we want you to
be thinking about in today's session is
how you can be using our core Audio API
services to create applications that are
easy to use and are going to show these
individuals how they can use a computer
in their music and audio creation
process this represents about twenty six
percent of the United States population
that's one in four individuals is your
market it's a really big market and
there's a lot of opportunities there for
you to tap into so let's drill down into
it just a little bit further the purple
wedge that you see up here is represents
a guitar player now there's about twenty
eight million guitar players in the
United States a lot of guitar players 15
million keyboard players nine million
DJs and remixers and producers and the
rest of the pie chart breaks down via
brass woodwind percussion orchestral etc
and again the point I want to drive home
here is that not a lot of these
individuals are even aware that they can
use a computer in their music and audio
creation process it's never been easier
than today to create a music and audio
application on the Mac using our core
audio api services you're going to find
out today just how easy it is so what
were some of our design objectives in
our core audio api services well two
things ease of use and performance we
wanted simplified user configuration for
the end user and streamlined development
for the developer performance that was
built with the professional in mind in a
lot of cases it was built with direct
feedback from the professional music and
audio developer and musician communities
it's the most feature-rich audio
platform out there core adios
multi-channel and high resolution with
sample rates up to 24 bit 96k and beyond
it's extremely low latency and we have
native MIDI support built-in if you've
ever looked in your utilities folder you
might have seen this little keyboard
icon up here which represents audio MIDI
setup you can configure your entire
audio and midi studio with this utility
very cool stuff we have class driver
support plug-and-play support for all of
your hardware spec compliant USB and
firewire audio MIDI devices we highly
encourage all the hardware developers in
the audience to really look at making
your hardware devices that compliant
it's really important it cuts down in
your development time and cost and it
makes it much much easier on the end
user the plug-and-play device support
you just plug it into Mac OS 10 and your
device will work no need to worry about
device drivers and finally audio units
and audio codecs Apple platform
standards which extend the capabilities
of the operating system via DSP plugins
virtual instruments and audio codecs
making it much much easier in your
development now we're going to get into
audio units and audio codecs and in a
lot more detail and bill session coming
up here in just a few minutes but one of
the biggest questions i get from
developers about audio units is well who
uses this the following slides should
really speak for itself the a really
large development community has already
sprung up around audio units and it's
getting bigger every day there's a lot
of individuals out there who are very
excited about the work that bill
Stewart's team has done and he's going
to get up here in just a second and get
into a little more detail so I'd like to
bring up Nick Thompson he's going to
talk about audio device drivers in Mac
OS 10
[Applause]
hi my name's Nick Thompson and a man is
the audio driver team in Apple hardware
division I wanted to kind of give you an
update on kind of talk about the drivers
that are in Darwin and how that
structured and also give you a little
bit of an update on what we've done for
the power mac g5 and changes that you
might need to make for your product so
and then I wanted to cover a class
drivers to a USB and firewire and point
you to some resources for when you're
writing your own drivers so looking at
the structure kind of a block diagram of
kernel-based drivers on Mac OS 10
they're all pretty much based on our
audio family when you look at the
built-in drivers you'll see there's a
basically a superclass Apple onboard
audio which encapsulate SMO STUV the
common chip features fries chips and
then we write plugins for free chip
specific to the platform so when you're
looking at the code in Darwin that's
kind of the structure of it and we also
developed cluster oversee USB and
firewire and the source code for the USC
driver is in Darwin currently the source
code for the firewire driver isn't Enzo
and currently and we're planning on
getting that in there after Panzer so
the permit g5 there's a couple of new
things that you've probably already
heard about I think the biggest one is
this is the first computer from Apple
where's built in digital i/o for audio
and we're really excited about this
we've also made some improvements the
analog section it's basically similar to
previous computers but we added support
for 24 bad data as well as 16-bit data
and also added support for different
sampling frequency so previously is
basically CD quality output now you can
have 44 148 32k at different depths and
I want to talk a little bit about some
of the changes that are needed for
devices and device drivers on the new
computer platform the really cool thing
about the digital section is that we do
clock recovery on input this means that
you can do better curat copies of
original source material which if you're
a musician really matters to you the
other thing as I mentioned is that both
on the analog and digital parts we've
got support for new sampling frequencies
and different sample sizes the connector
that we use is basically a ham it's a
hustling connector and the specs that
cover their serve up there is a 608
7-footer 617 it's basically the standard
friction lock connector that you'll see
on most consumer ed devices that have
optical connectors on them so the back
panel you'll see the two connectors
third and fourth down from the top of
the obstacle connectors we put a little
door on there so the you you don't see
the red light we connector basically
with we're testing with every piece of
consumer a big year that we've confined
understand the cables Justice standard
tough line connector that you'll be
familiar with if you come across a
across optical gear before wanted to
talk a little bit about ac3 support
obviously because we support this
lampert it's possible to either stream
and PCM data across sir or encoded
stream there's an important thing to
know about encoded streams you can
either output data from all answers and
PCM data and it's mixed together in the
same way that the analog section is
today or you can have one single ac3
encoded stream and so if you have say a
DVD player around it's important to know
that you won't hear system alerts if
you're in this mode which we termed hog
mode
if you're developing pci cards are some
changes that you need to be aware of the
biggest one is that we're no longer
supporting 5 volt signal England cards
and you can tell whether a card is key
for 5 volt signaling there's basically a
notch at the back end of the card if it
has both notches removed this is 3.3
volt universal code so you're okay we're
encouraging you to visit the
compatibility lab and make sure if you
have your cards that they work in the
new computer and the g five loaves are
downstairs there's also some changes
that you'd want to make an internal
based device driver basically we need
you to make your driver 64 bit safe when
we've seen a number of drivers from
third parties that currently don't work
if you have any questions about this
come find me we're happy to work with
developers and make machines available
if you can come to cupertino so you can
get your stuff working but basically
there's a bunch of macro is a kind of
ambiguous right now they've been
replaced with specific macros for the
word sites that you're using you also
need to remove any assumptions you have
in your driver about logical and
physical address is being the same when
we were pulling the Apple onboard audio
drivers we we basically came across
places where we were making these
assumptions and we weren't working so
you know we had a call in their physical
to kV which maps physical addresses the
colonel virtual you need to basically
work around these issues and then
finally you also need to make sure that
you prepare memory for Al if you do this
the diet entries created and the world
is good if you don't do this memory
isn't what you think it is talk about
the class drive is a bit first of us see
basically using our driver is going to
save your development costs so you
should try and make sure that you know
if you're developing a USB device that
can fluently USB audio swag we've we've
basically implemented support for
everything we've seen so far and I think
the message of this slide is you know if
you have a device that you're working on
with a mexi unit for example let us know
and we'll make sure that the drive
word for that we're also working on the
USB device 2 point 0 specification which
is kind of different to USB 2.0 it gets
a bit confusing here but this is
basically the specification for USB
audio devices on both full speed and
high speed USB we're going to be making
sure that we provide support for those
new devices in the cloth driver if you
are working on a USB 2.0 audio device
let us know we're interested in taking a
look at it and making sure that we work
with your hardware this is going to be a
big year for firewire thing we're
working with a number of developers
doing for our products again using the
standard space driver is great for those
of us in you it introduces your
development costs and for us it makes it
easier for customers when they buy a
device that can plug it into the mac and
it should just work basically there's
kind of two flavors of firewire devices
out there at the moment there's a vc
devices and we've seen silicon from
bridge quahogs a semiconductor for those
and there's a malign devices from a
variety of manufacturers including
yamaha Korg but sorry personas because
while at the gym the list is actually
growing and we're aiming to basically
support any silicon that anyone comes up
with us basically closed compliant and
Panthers we've made some changes the
driver that we shipped in Jaguar was
essentially a driver for speakers we're
now adding music subunit support so
we're providing input and output support
different numbers of input and output
streams support for MIDI we're also
working on lowering the latency and
Jessa this is really important to us for
a variety of reasons the other thing
that we're doing is if you have a
network of devices such as an online
network or a network of speakers we're
making some changes in how they present
it when it's a took a little bit about
what happens when you hot plug a device
so basically when you hop plug your
device that will show up we'll create a
unit for it and we'll start building the
state that we need to a builder device
reference will
an engine reference and start linking
that up and then we'll start building
stream references for input and output
start building stream rest of the output
stream and stream rest of the input
stream what happens when you unplug in a
new device right now is that will create
a new engine which is inefficient
because you're going to start getting
interrupts on each engine so what we're
going to do in Panther is basically
start linking the whole thing together
so the inside network is presented as a
single device so a new device is hot
plug will actually create a new stream
rat for the input that the output will
be the same output device so the weight
of this is one can wind up getting
presented as you'll see two input
devices with four streams on them in a
single output device with eight strings
on it so the goal here is essentially to
present a network as a single block
device which is the intention of amylin
networks but it's also for speakers are
much more intelligent way of doing
things finally I wanted to point you
some developer resources this driver
coding in Darwin you know she'd
definitely check out oil audio family if
you're doing a kernel based driver
should also really consider whether your
driver needs to be in the colonel it's
possible to hook into core audio and
write a driver that sits in user land
and that's often a better way of doing
things the PCI cards you do want to be
in the kernel for fire one maybe you
want to be in the kernel maybe you don't
you need to think about it on a device
specific basis we've also got sample
code in the SDK for the phantom audio
driver which is a great place to start
and we're looking at hopefully by the
transit time frame getting out of sample
PCI driver for a couple of devices
because a number of developers have been
asking for that and as always you should
check out the audio developer pages
developer.apple.com flash audio so
that's it the drivers I want to
introduce Bill Stewart and to talk about
Claudia
[Applause]
okay so thank you all for coming so what
is claudio and i can tell you something
about it when we started on this we had
a number of goals and we'll go through
some of those but basically Claudio is
an API that applications can use to
access audio and many services we talk
to you about access to devices we
present a unified API regardless of the
device dealing with with a audio or midi
we wanted to provide extensibility
mechanisms for doing software processing
and that covers both audio units as well
as codecs and we also want to provide
some general audio or sequencing sorts
of services that could be used in games
could be used in in audio music
applications and just to basically give
you a set of services that can satisfy
the kinds of things that you need to do
so that's what we're going to cover in
the session hope you can realize that
and this is probably a good kind of
explanation of what this looks like too
many of you now it's like a bunch of
sort of things that are kind of all over
the place and what I want to try and do
today is by the end of this session get
to a point where you can see this
diagram and have some sense of how this
might all fit together and it's not just
a bunch of sort of weird name some of
which start with a you and some of them
with audio and some of them have midi in
them all kinds of confusing terms okay
so there's four frameworks in the OS
that you need to look out for headers
and for linking against the succour di
framework call midi audio unit and audio
toolbox and then there's different
modules that load dynamically Audio
Units codecs and MIDI drivers this is
specifically like user level space then
of course you've got the colonel drivers
from I guess or
as Craig discussed earlier when we first
started with core do we wanted to set a
high standard for ourselves to reach I
think this was a very important initial
step to take for us because it's
extremely difficult if you aim low and
then have to scale up it's a lot easier
if you if your aim as high as you need
to aim and then you can appropriately
scale back so we really looked at what
was required from the pro audio market
and this is not to say that this is a
set of api's that can only be used in
musical applications or or pro audio
sort of workstations and things that
really can scale down to games and to
lower sample rates and and lower quality
abstractions and so forth but we wanted
to start at this point so we had
requirements of latency and jitter that
were very important to us both for
medium for audio we want to obviously no
restriction on sample rate which was a
problem with previous versions of the OS
multi-channel awareness throughout the
system not just with devices but also
with codecs with audio units with the
whole way that we think about audio in
the system and we didn't want to be
limited to just some small subset of how
we can represent audio data we wanted to
have rich abstractions that can be
applied throughout the system and the
session after this one will go into some
more detail about how we represent data
in the different subsystems of core do
so let's start at the bottom and we'll
kind of work our way up so the I'm not
going to spend a lot of time in the
cordilla hell part this is very much a
low level interface to devices there's a
lot or a lot of abstraction here for the
devices but there's also a lot of
specifics of I state that you need to
manage and
if you are in a situation where you
really need to interact very intimately
with the device this is the API that you
use this is in the Quartier framework we
affectionately call it the hell the
hardware abstraction layer and you will
get all of the characteristics of
devices published here configurations
system device preferences management of
the device status and all that kind of
thing and for MIDI we've got the
committee framework and this is really
the api's that are published through
here for transporting a MIDI data
through the system both in and out of
the system through drivers as well as
inter application in Jaguars committee
we have a concept of virtual sources and
destinations and we found that a lot of
developers from Mac OS 9 and the many
services that are ms and 3 mitigate that
one of the things that have been missing
from the way the system is used is an
IAC bus which is an indoor applications
bus and this is basically just a driver
that looks it's a software that looks
like a driver that actually gives you a
way to kind of connect MIDI between
different apps and it looks like you're
dealing with drivers rather than sort of
having to do extra work to look for
virtual sources and destinations so this
will be a new feature in Panther and the
other thing of course with committee is
needing to configure device and publish
device characteristics and so forth I'm
going to go to the demo machine and just
very briefly walk through the audio MIDI
setup applications just as a way to kind
of give you some sense of how all this
is put together so this is the Audio tab
we're going to we've got the mark the
unicorn 896 box here we use different
devices each year with you imagic and
delta m audio devices which always use
the motor one this year this top part of
the section here is
system settings and these are the the
default input and output they're
typically the devices that are used by
apps like iTunes by quicktime player by
games and the user can typically see in
the sound press i can see like these are
all the output devices I've got and they
can choose which one they want to use as
their default output and another thing
we introduced in our in Mac OS 10 was it
was a distinction between the device
that you would use for playing audio on
and the device that you would use for
things like sis beeps and throw forth so
the system output device is the device
where your beach go and the bottom part
here is is a device for City
configuration you can see for 896 that
it has different clock sources and you
have also in core do you have this
concept that a device can have different
streams and this is the way that the
device can publish its capability so on
896 we have two streams we have one
stream and here's the first stream and
it has a channel capabilities and this
would be like the analog a channel iOS
and the a channel digital i/os on the
device and the second stream would just
give us two channels and that would be
the spit if input or output and you can
see here I've got like sample rate
options for the the streams and
typically a device will run as the same
sample rate across the different streams
one of the optimizations that we didn't
do enough work on in Jaguar that we've
rectified this in panto is to turn
streams on and off you can think of a
stream as kind of a logical unit on a
device and you can turn a stream off and
that can tell the driver to not do work
on that stream and that can be done on
an application and application basis and
that can make the load of running the
system a lot less than if you're just
running a device that may have 50 i/o
streams another thing that we're doing
our for Panther is
trying to address the problem of how to
configure your setup for surround
systems different speaker orders and so
forth and on your Panther CD you'll see
there's a this panel for the speaker
configuration now you won't see this if
all you're looking at is the device with
two channels because it's just two
channels and everyone pretty unpretty
much understands left and right but if
you have a device like 896 that or the
metric halo devices or pci cards or some
of the USB devices with more than two
channels you'll see this speaker
configuration utility and i can
basically tell the system that these are
the on my device the way that I've got
this wired up in the studio or in the
home whatever is that one channel one is
going to be this speaker channel two is
that speaker and you can set these up
for different types of of surrounds now
you can't have all these different
surrounds active at the same time so
really you choose the surround speakers
that you've got in your particular
location and then you basically say well
okay I've got less run here and I can
make a sound and I can get that sound
out there we're going to change that to
a pink noise other than a sine wave
sound I can get the center cell and
lifestyle so forth and that can you know
make the whole configuration for the
user a little bit simpler and then we
will do mapping in the output units to
remap channels and they'll probably
needs to be some API changes for that
we're trying to look at how we could
push this into the existing output unit
but we're not sure at this point if we
can do that without actually doing the
wrong thing and we'll be publishing some
details about this later on and there's
some more details about how we represent
this stuff in the audio format session
if i go for the MIDI section now
I've got a one USB device here from
Roland it's a mpu 64 and it has four
midi ins and outs I've got a studio set
up here and look let's not say I'll
leave that over there early software so
so one of the things I wanted to talk
about with this app is a little bit of
confusion about exactly what it's meant
to be doing and this really reflects as
the audio side does this both of these
apps if you're developing and you're
familiar with the API they really
reflect the a lot of the structures that
are in the API themselves and so you can
sort of see this is like this is what
the user sees but also as a developer we
can just with a couple of grafting
concepts here we can understand what
you're seeing from an API point of view
so this is a driver this driver has what
is in MIDI is called our MIDI entities
and it has four midi entities which are
these pairs of in and out ports and the
in and out port submitting our endpoints
and each MIDI endpoint takes a full MIDI
stream so that's 16 channels and you're
able to you know talk to whatever is at
the end of that stream will get the data
from the end of that stream so this this
configuration here has for MIDI entities
and each MIDI entity has an end points
were in and out and what we're doing
here is describing the driver now what
is out here is just three devices that
I've added by just doing this add device
thing and I can add a new external
device and i'm not actually creating any
it's not like i'm going to really create
like a whole patch flow here all i'm
doing is describing to the system what
I've got plugged into what the system
knows about which is the actual driver
you can't really do through connections
here in the sense that that all like
just automatically route you have to
wrap these with your cables you're
really just describing to the system two
keyboards or the
control surfaces or the modules that you
have on your system and you can save
different configurations and if i unplug
the USB device the driver will go
offline and you'll see that reflected in
AMS and what the what can be done here
is so on rather than sort of showing
this is port four of us be you know SMP
you 64 you can actually present by query
and call middies api's you can actually
present my synth module as the
destination for that to the user now the
thing that's missing from this that I'm
sure many of you have dealt with OEMs
and free midi is some way to describe
what is actually in the device itself if
it's a roland you you 20 or you 220
thing what patches does it has and what
is the current patch banks that I've had
what are the capabilities of the device
we've worked with the many manufacturers
associations to describe a document
format for that using XML the the spec
has been ratified and has passed through
all of the MMA processes for doing that
we wanted to do this through a body
rather than as an apple sort of only
initiative just to make this something
that could be broadly adopted both by
manufacturers and other computer
companies besides ourselves so that for
the user there's one data format the the
documents not available yet from the MMA
psyched I will be soon as we're going
through the final stage of actually
making the documents are clearer about
what it's contained so that there's you
know it's an easy-to-understand document
you'll be able to also these XML files
yourself to describe custom patches and
we will hope that they'll be websites
available and manufacturers will publish
them for their devices and so then the
user can see my sense module and it's a
rolling since module yada yada and it's
got these patches on it and just like
with OEMs and free mini on Mac OS 9 and
it should be
as easy and experience for the user on
10 we can go back to slides please right
so that's the bottom let's get into
audio units so dear units we have a logo
and that's the logo after took some some
effort i can tell you so so audio units
and i'm going to say that these are
generalized plugin format and by
generalized i mean that it has very many
different uses it's not just for effects
it's not just for software instruments
and what we'll do today's later on as a
demo for some other what air units and i
just want to give an overview generally
if what audio units are the types of
audio units that you can have and
they're kind of categories and their
functionality so how does it work an
audio unit uses the component manager
and we use the component manager because
it has some nice features about it has
discovery mechanisms find next component
you can just make this call and you can
specify the type of audio unit that
you're looking for and we have different
types once you find that the audio unit
that you want and you can either be
doing this programmatically or you can
present menus based on what you've
discovered to the user then you open
that component and once you've opened
two component you get back what's called
a component instance so you can think of
a component in object-oriented terms as
a class specifies an API specifies what
that a member of that class can do and
the audio unit itself is the component
instant is like an object an instance of
the class and so audio unit as a typist
is type desk to a component instance
so you've opened your audio unit so what
do you need to do to it well it can be
as simple and complex as the type of
audio unit you're dealing with what you
want to do with it and the first steps
really for an audio unit is to look at
the properties that it has and
properties really represent in in this
kind of sense of the property mechanism
it's the state of the audio unit and so
you it is a general mechanism it's
extensible we define property types you
can get information about the property
how big is the property can I read the
property or write the property or both I
can use the property making mechanism to
find out the state that the unit comes
up in and I can change the state of it
and so forth and it's really the way
that you manage that fundamental state
of an audio unit and then once you've
sort of set up the state of the unit
then we have a second phase which is
initialization we split these up because
there's often a lot of things you might
want to discover about what an audio
unit can do particularly if you're in
like a hosting environment before you
initialize does it make it able to
render and able to do its job and so
audio unit initializes this concept of
of allocation in order to actually
operate and once an order unity is
initialized and it's considered to be in
a state to be usable and that is the the
one call that you do really to use it is
audio unit render and there's a you'll
so I'm not going to go into the specific
details of the API so that that there's
a lot of arguments that you can pass to
this and flags and so forth some move
fairly well documented in the headers
and you can ask questions on the list
now we're ready to go we're going to
call a unit render that where we're
going to get inputs are the form you can
get input for an audio unit from two
different places we wanted to have with
audio units an idea that we can connect
them up into process and grass all we
can use them independently either just
one off
or maybe two or three and then I want to
provide data to it but I don't want to
have to be an audio units to provide
data to another audio unit so you can
also have a callback function or you can
have the connections for there's two
ways that you can provide data to an
audio unit and at this point we're
talking about audio data it's an audio
unit and so when you call audio unit
render on an audio unit it's going to go
if it wants input and some of their
units may not want input and we'll have
a look at those in a minute it'll call
it its input proc or its connection for
a simple data when that returns it's now
got input data to process that processes
that data and then you get that back in
the Buffaloes that you provide audio
unit render and you're done well are we
done no not quite because one of the
things you want to do when you're
processing audios you want to be able to
tweak it you want to be able to set
delay times differently you may want to
be able to set frequencies if you're
talking about volumes going to change
volumes and so forth so all of these are
considered as real time operations on
the audio unit things that you can do to
the audio unit while you're in the
process of rendering it and we abstract
that into what we call parameters an
audio unit using the property mechanism
publishes a list of the parameters that
it allows the user to manipulate it
publishes things like what's the range
is a parameter what kind of values does
it have and the DB or Hertz or maybe
just a generic 021 sort of parameter
there's a whole bunch of different types
of parameters that an audio unit may
publish and it really discs and we've
seen some third-party units that have a
couple of hundred parameters a lot of
our units may be fairly simple they made
just two or three parameters it really
depends on what the unit is doing and
what the developer of the unit wants you
to be able to manipulate so we have
effects units and that's really kind of
the meat of a lot of where we think the
third parties will be developing units
in Jaguar
we ship various filters there's high-low
pass band pass high shelf and low shelf
filters we ship a reverb unit and the
reverb unit has quality settings that
you can adjust the quality really
determines how much CPU that the unit's
going to take at runtime and it actually
affects the quality of the rendering and
so things like games and stuff they may
be less concerned about a very high
quality and more concerned about the
load that the unit's going to take and
we have a digital delay you know we have
a peak limiter and Panther was added a
multiband compressor unit it's a full
band compressor and it's so pretty nice
actually and you know we're not going to
publish a couple of hundred audio units
as accompanying ourselves this is for us
the big part of audio units is what
developers are going to do with is this
is where it can get very interesting and
very bizarre sometimes so to sort of
summarize that we create one we've got
the state management we've got the
resource allocation we've got rendering
and we've got the control of the
rendering well is that all we need no
typically if you if you used logic or if
you've used digital perform or any of
these for the hosting environment and
type applications you want to present
some kind of view to the audio unit so
the user can interact with them we've
published a generic view which will just
query the parameters that the unit tells
us and we will assemble like a bunch of
sliders and that's you know it's not bad
but it's probably not as as good as what
you can do if you really understand your
DSP and so a developer of the naughty
yoona can publish a custom view and some
of the views if you've seen these are
pretty creative and pretty interesting
and jagua we only had the ability to
publish carbon views in panto adding
support for cocoa so you can publish a
class that implements a cocoa protocol
and it can be discovered from
Asking naudia unit for its cocoa view
and the cut a carbon app can put a cocoa
you I up in a separate window and the
cocoa lap can put a carbon you I up in a
separate window so there's not a lot of
extra work on the hosting side to do to
deal with both Aziz and we think that
probably a lot of developers would be
very interested in the cocoa you are so
that's there now and the other side of
this is communication and control
mechanisms in Jaguar we have a parameter
listener API it's in audio toolbox audio
unit utilities dot H and it is an API
called au parameter listener laws gets
hold off when I'm wandering around but I
just can't stand still so I'm going to
keep wandering excusing so the parameter
listener when when we were looking at
this to design this there's two ways
that you can do this you can have a UI
that or an asset is going to you know 30
times a second or whatever it's going to
pause for the values of the parameters
see if anything's changed and it seemed
to us a lesson elegant way of dealing
with this so we decided to do a
notification service and the
notification services aimed at allowing
anybody who wants to know about a
parameter changing on an audio unit to
be able to listen to that parameter to
see that that parameter is changing them
to react appropriately and then when
they want to set a value of the
parameter if they just use the standard
audio unit set parameter call that's not
going to invoke the notification you
need to use to a parameter set for which
is in this in this header file and that
basically then we'll tell the notifier
the notification mechanism that hey if
you've got anyone who's listening to
this you better tell them about it we've
decided to extend this a little bit in
Panther and to include the ability to
notify state changes from the a audio
unit property and so you can do both
parameter chain
and property changes using this
notification mechanism now one of the
important things about audio units with
all of this kind of stuff is that we're
never going to know everything that we
need to know about every possible audio
unit that every developers ever going to
write and so there are mechanisms in
place to be able to share private data
with private property IDs and all this
kind of thing between your audio units
and your views or this kind of stuff and
so this mechanism can be used to to
communicate that there's a need to go
and get some state that may be in the
audio unit if you're the view and it can
be done in the thread safe manner so you
can call this sort of API from your
render sphere as well as from a UI
thread the other weakness that I think
we had in Jaguar with the carbon you is
that we had this idea if you're doing
automation that you need to know like
start and end state as you're doing
automation and we sort of put that into
the view and that really wasn't the
right place for that and it kind of
restricted the sum of the use of it
where the audio unit may know that it's
a start and end state not to view and so
we added to the Panther services for
this this idea of the beginning end of a
parameter ramp state so you could
imagine if you've got a control surface
when the user touches the control some
of these control surfaces a sense it is
to actual touch so you could touch the
control and then that would be a signal
to say hey I'm bout to start ramping
this parameter and the UI could reflect
that the users types that control with
the changing button in the UI and by
putting this into the controller this
means that we can also support this with
Coco you eyes as well as carbon you eyes
without having to add additional logic
to the Coco you I so I think this is a
very nice addition to this and ever said
before this is real-time thread safe and
will continue to work very hard to
ensure that that remains true
okay you all asleep yet was nice I'm
actually still with me okay so we've got
a fax and this is really I think we're a
lot of the work is done for third
parties but also another area that a lot
of third parties do work in these
instruments and software synthesizers
and the idea with this of course was is
that you're not going to be processing
audio although some sense will let you
have audio input into their synthesis
algorithms but also the main part here
is that Bobby will be MIDI data coming
either from a host apple from an
external keyboard and so we have an
extension to the basic audio unit API to
a musical instruments and get my clicker
to work and we call this a music device
that's in the music device dot H file
when this just adds semantics for
starting and stopping notes and for the
controller's notes we've got a couple of
api's there that really just talk
committee so you've got you know music
device send MIDI and you just take the
the two or three by midi for channel and
then you've got another API for dealing
with the sis X and the other sorts of
extended midi messages we also added an
additional API for the music devices
because we didn't want to just talk midi
as a software architecture we don't
really have to abide by the same
limitations as hardware does with with
the MIDI stuff so we have an extended
API where you have the ability to have a
software send have more than just 16
channels we call them groups in in our
parlance so you can have an arbitrary
number of groups you can actually group
notes into different instruments playing
on the same group so i could have three
notes that are playing on different
sounds and they're all playing on the
same group and i can just send control
messages to that group the extended
protocol works in a similar kind of way
to MIDI if you kind of break nidhi down
in a suit
way and you can express all of the
control semantics of meeting in the
extended protocol that it gives you some
flexibility you can specify pitch using
a fractional floating point numbers 60.3
60.5 will give you the half time between
C and C sharp and away you go so this is
a something that we're seeing a lot of
two parties using the music to buy stuff
there's some very interesting
third-party cents out there already one
of the things that we were working on
this is a to address some concerns were
raised by ways who are a very large
developer of audio processing applica
plugins and they wanted to have a way to
do offline rendering offline rendering
is typically done when you want to
actually process the file of data and
you want to look at the whole contents
of the file not just in the real time
all all of the audio unit development
that we've done up until this this
particular unit is really aimed at
working in real time and so it has
constraints about having to work in real
time you need to report like latency you
need to report how long it takes you to
process sounds in terms of the time
delay between input and output we've got
an offline unit you need to be able to
look at the data all of the data that
you're going to be asked to process
before you process so there's to render
phases with an offline unit there's an
analyze phase and then there's a render
fo so do you think of reversing the
sample order in a file you need to
actually like start at the end and work
work your way back if you think of an
offline unit that may normalize you need
to look at the whole audio data before
you start to process it so you can do
the normalization and there's not really
any additional ati changes for this
there's a couple of different flags
there's a couple of properties that are
specific for an offline unit there's a
new audio unit type a you o el audio
unit offline it's not in your Panther
headers
because we're still revising discussing
this with waves and some other companies
and will be that this will be published
in Panther and we're getting pretty
close and we can always if people in
sedate and contact me and I'll send them
the spec in the code that we've got at
the moment and we will ship a reverse
annoyer unit that does reversal in the
SDK at some point as an example and
they'll be code there as well to have
you host these offline units okay so
when I said generalized audio unit we're
still sort of like in the the general
field at the moment about normal types
of audio units now let's look at some
abnormal types of audio units with and
one of those types is mixer units so in
in jago we had two mixes a stereo mixer
that takes mono or stereo inputs and
gives you a next single stereo output
and in jago as well we had a 3d mixer
the 3d mixer will take multiple inputs
it will have a single output and will be
in either 24 or 5 channel output and the
four channel is like a quad setup which
is pretty much what we've got in the
room today and 50 where we don't
actually do the 3d stuff into the lse
channel we just do the 55 channels and
what you can do with this 3d mixer is
quite a lot you can pan and localize
within a 3d space you have a number of
on-off options and I'll get Chris Rogers
to come up and we're just going to give
you a fairly quick demo of the CD mix
[Applause]
well thanks still setting me up there
last year I gave a more complete
demonstration of the 3d mixer but some
things have changed since last year
because developers have made some
requests of the mixer so we put some new
features into the 3d mixer and we can
have a look right here what I have right
here is a simple little app simple
little user interface on to the 3d mixer
and it has a number of different
controls for choosing the type of
rendering for the source in this demo
there are going to be three sources or
it can be up to three sources and you
can choose equal power panning a simple
spherical head model an HR TF model and
these first three are for rendering the
stereo output and then the last two
sounds field and vector based those can
be used for stereo quad or 50 then over
here we have some checkboxes that let
certain fixture features of the mixer be
enabled or disabled for individual
sources down here we have master volume
control and here this is kind of an
obscure slider that controls distance
attenuation that is when sources when
sound sources get farther away they get
quieter but how much quieter do they get
say if they're ten meters away or 100
meters away how much quieter did they
get this can control what that curve is
that the follow up and that's a feature
that developers have been asking for so
down here in this part of the display we
have have some meters which the 3d mr.
now supports lives metering you can
meter the input levels and output levels
both RMS power and peak levels
metering is something that we put into a
couple of our other units and later on
today we'll see the new new audio unit
matrix mixer also supports this which is
kind of interesting so maybe I should
just bring a source in and see
ok
so maybe I should simplify this a little
bit
and you have turned all my stuff is off
okay here we go so I'm using
vector-based hanging right here and I
turned these these sources off so well
not listening to those we're just
listening to this blue one here but if I
put this dot right here then we're
essentially just in the center to center
channel
hold on we have the helicopter coming
I'm sorry there it is let me turn him
off okay I think now maybe our meters
will I'll show this a little better ok
so I'm about straight ahead here I could
be coming out of the center speaker and
the channel ordering down here is left
right surround left right center so ok
now we have left and rear rear left we
have probably surround left around right
ok now let me change this sound
I'll use this end this sound right here
okay okay turn ok it looked like we're
running a little bit short on time so
I'll try to wrap it up here basically
the new features that are the most
important are the ability to turn offer
on individual features here and that
affects the performance that you're
going to get this distance filter is a
low-pass filter that makes sounds Sun
more muted is for getting further away
and some developers had some comments
about that that they didn't necessarily
want their sounds to get more muted so
there's a way that that can be turned on
or off and any of these any of these
characteristics can be turned on or off
separately and if art performance goes
we've made some optimizations to the
mixer and to give you an idea of the
kind of performance that you can get for
in for an individual source using equal
power panning of stereo I think I'm
pretty modest machine like an 800
megahertz g4 you can get a single equal
power source at I think it's point one
eight percent of the cpu and h RTF which
would be our high quality stereo panning
mode that's at about point five five
percent of the cpu / source so little
tiny bit more than half a percent now I
have a we'll just move on
okay go back to the slides thank you
Chris the other demo that we work in the
show but we're running a bit short is a
very speed unit and that's also new in
panto and you're able to make have your
fam come into the various speed and it
can go faster or slower and go you know
Connors oh what's the chipmunk effect so
show you that later on if you want to be
in a lab tomorrow from one o'clock till
six o'clock the QuickTime lab and we can
give you a demo there if you like ok so
in Panther as well we have a matrix
mixer matrix mixer is a very powerful
beast and we'll be going to some detail
about that in the next session all
mixers have metering capabilities in
Panther so it can be you'll see some of
that in the next demo next session the
other type of audio unit that we have is
a converting unit will be talking about
the audio converter in the next session
as well and this brings some of the
functionality of the audio converter
into the audio unit world essentially
all of the conversion operations to deal
with PCM audio so sample rates bit
depths channel mapping and so forth and
all of this is configured with
properties and actually there's very
little configuration of it for you to do
because just describing to the audio
unit what you input and your Apple
format is is enough to tell the
converter what work that you do and the
converter unit functionality is that is
included is a part of the output unit
the output units interface to a device
there's no additional latency there's no
runtime cost to you for dealing with the
output units and you can manage how much
work the output unit does for you by
seeing the difference between the format
of the device which is on the on the
output unit and the format that you
provided in Jaguar we only did output an
hour output units to
input so that's why they're rabbit is
output units now in Panther and the
output units will do output as they do
in Jaguar on bus 1 and element 1 or bus
bus 0 mean on album element Marnell bus
one you can do input and so what does
this look like it looks like this so I'm
going to use the sliding I they hate us
using these things that I love it so
here's your output unit on the output
side here is your device output this is
on bus 0 and then this is either a call
back or a connection that you make to
the audio unit and this is if you're
using this unit this is what you're
doing today in Panther you can also see
if there is input available there's a
property to query that and then the
devices input is actually on the input
side of the output unit confused yet I
took me a while and then the application
actually calls audio unit render for bus
1 and that's where it gets the input
from the device and it can do the
conversion for you as well so if your
device is 20 channels and you only 14
channels and it will remove those 20
channels and just give you the four
channels and you can tell the output
unit which four channels that you want
from the device including rate
conversion all this sort of thing and if
your device just has input or just a and
you need to know when to get input then
you can get a call back and it will tell
you hey the inputs ready go and get it
and so that's a new in pampa okay so
Audio Units we talked about connection
we talked about wanting to be able to
connect all these up and we have a
structure in the audio toolbox api's
called au grass and a aggress manages
the connections between these units
it'll manage the state is it has a very
abstract representation of the graph and
then the graph itself has a couple of
different states you open the graph and
that'll basically open the audio units
that you've described as being a part of
your graph then you can initialize them
and then you can actually start the
graph running
and you can update the state of the
graph while is changing you can make or
break connections you can have a bunch
of audio units sitting off to the side
that maybe one channel one chain into a
mixer or something and then you can just
connect that chain in players down
disconnected and the graph will will do
all of this it will manage the different
threads of the thread that you're doing
the rendering in the threads are you
doing the state management from and
it'll just make this a lot simpler to do
then you can write your own code but if
you going to do this it's a good idea to
look at another API in the toolbox is
the music sequence API music sequences
of a collection of events and are
there's really two different types of
events and we have two different types
of tracks they're still just basically a
track but a music sequence has some
concept of cancer music sequences turn
talk about their timing in terms of
beeps and so the tempo describes how
many beats per second if you want to
deal in seconds you can just make a
sequence with the tempo advantage of 60
beats per minute and then you can deal
in seconds and then you've got any
number of events tracks and the event
shacks can take any number of different
types of events you can have MIDI events
you can have user events where you can
put your own data in there and you can
have parameter event so you can talk
directly to particular parameters on
different audio units and of course the
connection is that you have a sequence
and you connect that up to a graph and
we'll show you that in a minute you can
create a sequence from a MIDI file or
you can create a sequence
programmatically you can once created a
sequence you can save the sequence off
to a MIDI file will only save the MIDI
type of dance that are in that sequence
to the MIDI file obviously we won't do
the other ones and you can iterate over
the events you can loop mute solo tracks
when you're playing them and you can
copy-paste insert edit them and these
can be edited while the sequence is
actually being played in real time
and you can target the tracks that are
in a sequence to do two different
destinations you can target them to a
specific audio unit that's in a graph
that you've attached to the sequence or
you can target a sequences track to a
MIDI endpoint so you could be sending
primer sequence that you're playing
directly out to a MIDI device and if you
want to play it then you've got to have
some object to play it and that's the
music player music player has very
simple start and stop semantics it has
pre rolling so you can particularly with
software's is very important so you can
actually pre-roll chase your events to
whatever the time that you set one of
the things that we added in Panther is
the ability to scale the playback rate
so that you can if you were wanting to
do some work to synchronize this to some
external time source or you just want to
play the events back faster or slower
you can do this with this additional API
so if we go back to what we started with
connecting it all together this was the
kind of the mess of the api's and then
if we sort of put some blocks around it
this is how we kind of see it all
fitting in I'm going to go back to my
laserwriter here laser pointer so here's
the audio units here's the howl output
to buy his the MIDI device the audio
units are sort of connected with these
orange bars you don't really see this
connection here that's really a part of
the output unit and the state that it
manages in this case here which is very
similar to the 3d mix if you if you
remove this filter and you think of the
demo the christed then really what he's
doing is sort of this part of of this
whole kit and kaboodle so we're using
the audio file API to read the file data
from the disk the converter to convert
that into a format that we can use for
the 3d mixer and then the that just goes
into the as an input to the 3d mixer and
in the demo that Chris was running we
had three inputs into that mixer one
thing we didn't chose is this side of it
so you can have a music sequences
collection of tracks you can address the
track events to
different nodes in the grassland you can
make the association between the
sequence and the graph and then the play
actually owns a sequence there's player
plays of sequence which starts the graph
or if it's just talking to many then it
starts a MIDI scheduler and and the way
you go and of course you don't excuse me
you don't need to use all of this you
can just use this vid or you can just
talk to the lower levels of the system
where you can just use audio units
without using their graphs and so it's a
very modular architecture with lots of
components there lots of complexity to
get lost in we are working on
documentation it has me to know and we
are going to do some more work on some
sample code and we'll see in the next
session are Doug's one of the things
Doug's been working on which will be in
the SDK and so that's pretty much all
lid we provide those sort of services
I'm not going to read that you can read
it yourself and yeah don't panic you
don't have to know all of these things
you really need to understand well what
is it that your app needs to do and then
what is the appropriate API that you
need to do to do that and you can just
ignore the stuff that you don't need to
do ok so there the Four frameworks
Claudio for drivers call media committee
the Audio Units really that framework
just publishes the API for the
extendable audio units and codecs and
then the toolbox for formats files when
talking a lot about the format stuff in
the next session wrap up roadmap there's
some of the sessions we have the audio
and quicktime session tomorrow which is
a going to be very interesting session i
think if you go to feedback forum for
our asses after then and nick and craig
and myself and others will be there who
to contact and for more information
we've got the choreo or the audio
technologies for Mac os10 website that
ipod app electrons flash audio
you