WWDC2003 Session 407

Transcript

Kind: captions Language: en I'm Craig lengthen music and audio partnership manager at Apple and today's session you're going to learn my Mac os10 audio truly does rock so what we're going to cover today we're going to go over some developer market opportunities we'd Apple we're excited about our advancements in the media music and audio space recently and we'd like you to be able to share in our success the market is really large and I'm going to go over a couple of slides to show you just how large that market is and hopefully show you some new applications to some new opportunities that you can tap into with your applications we're going to cover some audio device driver changes in Panther and we're going to go over the core audio architecture design objective finally we're going to dive right into the api's and show you how to get started whether you're a game developer a music and audio developer or you just want to input output or process music or audio with your application you're going to worry about the api's that you need to get started today so market the following is a slide of the estimated number of music and audio creators in the United States today at the top end of the spectrum is the pro market there's approximately 50,000 aspiring sorry 50,000 professionals who are doing music and audio day in and day out making a living at it 100 percent of those individuals are using accusing a computer in their creation process now in the middle end of the pyramid is the aspiring professional market these are individuals who aspire to be at the top of the pyramid they aren't quite making a living at it but they spend a lot of money on their gear and it's a very large market however only about thirty percent of those use a computer in their music and audio creation process and at the bottom end of the pyramid you'll see the creative individuals fifty five point nine million individuals in the United States who are creatively inclined to do music and audio in some fashion or another these could be guitar players people who are just learning to play musical instrument students DJ's hobbyists etc but only about five percent of those individuals currently use a computer in their music and audio creation set and one of the things we want you to be thinking about in today's session is how you can be using our core Audio API services to create applications that are easy to use and are going to show these individuals how they can use a computer in their music and audio creation process this represents about twenty six percent of the United States population that's one in four individuals is your market it's a really big market and there's a lot of opportunities there for you to tap into so let's drill down into it just a little bit further the purple wedge that you see up here is represents a guitar player now there's about twenty eight million guitar players in the United States a lot of guitar players 15 million keyboard players nine million DJs and remixers and producers and the rest of the pie chart breaks down via brass woodwind percussion orchestral etc and again the point I want to drive home here is that not a lot of these individuals are even aware that they can use a computer in their music and audio creation process it's never been easier than today to create a music and audio application on the Mac using our core audio api services you're going to find out today just how easy it is so what were some of our design objectives in our core audio api services well two things ease of use and performance we wanted simplified user configuration for the end user and streamlined development for the developer performance that was built with the professional in mind in a lot of cases it was built with direct feedback from the professional music and audio developer and musician communities it's the most feature-rich audio platform out there core adios multi-channel and high resolution with sample rates up to 24 bit 96k and beyond it's extremely low latency and we have native MIDI support built-in if you've ever looked in your utilities folder you might have seen this little keyboard icon up here which represents audio MIDI setup you can configure your entire audio and midi studio with this utility very cool stuff we have class driver support plug-and-play support for all of your hardware spec compliant USB and firewire audio MIDI devices we highly encourage all the hardware developers in the audience to really look at making your hardware devices that compliant it's really important it cuts down in your development time and cost and it makes it much much easier on the end user the plug-and-play device support you just plug it into Mac OS 10 and your device will work no need to worry about device drivers and finally audio units and audio codecs Apple platform standards which extend the capabilities of the operating system via DSP plugins virtual instruments and audio codecs making it much much easier in your development now we're going to get into audio units and audio codecs and in a lot more detail and bill session coming up here in just a few minutes but one of the biggest questions i get from developers about audio units is well who uses this the following slides should really speak for itself the a really large development community has already sprung up around audio units and it's getting bigger every day there's a lot of individuals out there who are very excited about the work that bill Stewart's team has done and he's going to get up here in just a second and get into a little more detail so I'd like to bring up Nick Thompson he's going to talk about audio device drivers in Mac OS 10 [Applause] hi my name's Nick Thompson and a man is the audio driver team in Apple hardware division I wanted to kind of give you an update on kind of talk about the drivers that are in Darwin and how that structured and also give you a little bit of an update on what we've done for the power mac g5 and changes that you might need to make for your product so and then I wanted to cover a class drivers to a USB and firewire and point you to some resources for when you're writing your own drivers so looking at the structure kind of a block diagram of kernel-based drivers on Mac OS 10 they're all pretty much based on our audio family when you look at the built-in drivers you'll see there's a basically a superclass Apple onboard audio which encapsulate SMO STUV the common chip features fries chips and then we write plugins for free chip specific to the platform so when you're looking at the code in Darwin that's kind of the structure of it and we also developed cluster oversee USB and firewire and the source code for the USC driver is in Darwin currently the source code for the firewire driver isn't Enzo and currently and we're planning on getting that in there after Panzer so the permit g5 there's a couple of new things that you've probably already heard about I think the biggest one is this is the first computer from Apple where's built in digital i/o for audio and we're really excited about this we've also made some improvements the analog section it's basically similar to previous computers but we added support for 24 bad data as well as 16-bit data and also added support for different sampling frequency so previously is basically CD quality output now you can have 44 148 32k at different depths and I want to talk a little bit about some of the changes that are needed for devices and device drivers on the new computer platform the really cool thing about the digital section is that we do clock recovery on input this means that you can do better curat copies of original source material which if you're a musician really matters to you the other thing as I mentioned is that both on the analog and digital parts we've got support for new sampling frequencies and different sample sizes the connector that we use is basically a ham it's a hustling connector and the specs that cover their serve up there is a 608 7-footer 617 it's basically the standard friction lock connector that you'll see on most consumer ed devices that have optical connectors on them so the back panel you'll see the two connectors third and fourth down from the top of the obstacle connectors we put a little door on there so the you you don't see the red light we connector basically with we're testing with every piece of consumer a big year that we've confined understand the cables Justice standard tough line connector that you'll be familiar with if you come across a across optical gear before wanted to talk a little bit about ac3 support obviously because we support this lampert it's possible to either stream and PCM data across sir or encoded stream there's an important thing to know about encoded streams you can either output data from all answers and PCM data and it's mixed together in the same way that the analog section is today or you can have one single ac3 encoded stream and so if you have say a DVD player around it's important to know that you won't hear system alerts if you're in this mode which we termed hog mode if you're developing pci cards are some changes that you need to be aware of the biggest one is that we're no longer supporting 5 volt signal England cards and you can tell whether a card is key for 5 volt signaling there's basically a notch at the back end of the card if it has both notches removed this is 3.3 volt universal code so you're okay we're encouraging you to visit the compatibility lab and make sure if you have your cards that they work in the new computer and the g five loaves are downstairs there's also some changes that you'd want to make an internal based device driver basically we need you to make your driver 64 bit safe when we've seen a number of drivers from third parties that currently don't work if you have any questions about this come find me we're happy to work with developers and make machines available if you can come to cupertino so you can get your stuff working but basically there's a bunch of macro is a kind of ambiguous right now they've been replaced with specific macros for the word sites that you're using you also need to remove any assumptions you have in your driver about logical and physical address is being the same when we were pulling the Apple onboard audio drivers we we basically came across places where we were making these assumptions and we weren't working so you know we had a call in their physical to kV which maps physical addresses the colonel virtual you need to basically work around these issues and then finally you also need to make sure that you prepare memory for Al if you do this the diet entries created and the world is good if you don't do this memory isn't what you think it is talk about the class drive is a bit first of us see basically using our driver is going to save your development costs so you should try and make sure that you know if you're developing a USB device that can fluently USB audio swag we've we've basically implemented support for everything we've seen so far and I think the message of this slide is you know if you have a device that you're working on with a mexi unit for example let us know and we'll make sure that the drive word for that we're also working on the USB device 2 point 0 specification which is kind of different to USB 2.0 it gets a bit confusing here but this is basically the specification for USB audio devices on both full speed and high speed USB we're going to be making sure that we provide support for those new devices in the cloth driver if you are working on a USB 2.0 audio device let us know we're interested in taking a look at it and making sure that we work with your hardware this is going to be a big year for firewire thing we're working with a number of developers doing for our products again using the standard space driver is great for those of us in you it introduces your development costs and for us it makes it easier for customers when they buy a device that can plug it into the mac and it should just work basically there's kind of two flavors of firewire devices out there at the moment there's a vc devices and we've seen silicon from bridge quahogs a semiconductor for those and there's a malign devices from a variety of manufacturers including yamaha Korg but sorry personas because while at the gym the list is actually growing and we're aiming to basically support any silicon that anyone comes up with us basically closed compliant and Panthers we've made some changes the driver that we shipped in Jaguar was essentially a driver for speakers we're now adding music subunit support so we're providing input and output support different numbers of input and output streams support for MIDI we're also working on lowering the latency and Jessa this is really important to us for a variety of reasons the other thing that we're doing is if you have a network of devices such as an online network or a network of speakers we're making some changes in how they present it when it's a took a little bit about what happens when you hot plug a device so basically when you hop plug your device that will show up we'll create a unit for it and we'll start building the state that we need to a builder device reference will an engine reference and start linking that up and then we'll start building stream references for input and output start building stream rest of the output stream and stream rest of the input stream what happens when you unplug in a new device right now is that will create a new engine which is inefficient because you're going to start getting interrupts on each engine so what we're going to do in Panther is basically start linking the whole thing together so the inside network is presented as a single device so a new device is hot plug will actually create a new stream rat for the input that the output will be the same output device so the weight of this is one can wind up getting presented as you'll see two input devices with four streams on them in a single output device with eight strings on it so the goal here is essentially to present a network as a single block device which is the intention of amylin networks but it's also for speakers are much more intelligent way of doing things finally I wanted to point you some developer resources this driver coding in Darwin you know she'd definitely check out oil audio family if you're doing a kernel based driver should also really consider whether your driver needs to be in the colonel it's possible to hook into core audio and write a driver that sits in user land and that's often a better way of doing things the PCI cards you do want to be in the kernel for fire one maybe you want to be in the kernel maybe you don't you need to think about it on a device specific basis we've also got sample code in the SDK for the phantom audio driver which is a great place to start and we're looking at hopefully by the transit time frame getting out of sample PCI driver for a couple of devices because a number of developers have been asking for that and as always you should check out the audio developer pages developer.apple.com flash audio so that's it the drivers I want to introduce Bill Stewart and to talk about Claudia [Applause] okay so thank you all for coming so what is claudio and i can tell you something about it when we started on this we had a number of goals and we'll go through some of those but basically Claudio is an API that applications can use to access audio and many services we talk to you about access to devices we present a unified API regardless of the device dealing with with a audio or midi we wanted to provide extensibility mechanisms for doing software processing and that covers both audio units as well as codecs and we also want to provide some general audio or sequencing sorts of services that could be used in games could be used in in audio music applications and just to basically give you a set of services that can satisfy the kinds of things that you need to do so that's what we're going to cover in the session hope you can realize that and this is probably a good kind of explanation of what this looks like too many of you now it's like a bunch of sort of things that are kind of all over the place and what I want to try and do today is by the end of this session get to a point where you can see this diagram and have some sense of how this might all fit together and it's not just a bunch of sort of weird name some of which start with a you and some of them with audio and some of them have midi in them all kinds of confusing terms okay so there's four frameworks in the OS that you need to look out for headers and for linking against the succour di framework call midi audio unit and audio toolbox and then there's different modules that load dynamically Audio Units codecs and MIDI drivers this is specifically like user level space then of course you've got the colonel drivers from I guess or as Craig discussed earlier when we first started with core do we wanted to set a high standard for ourselves to reach I think this was a very important initial step to take for us because it's extremely difficult if you aim low and then have to scale up it's a lot easier if you if your aim as high as you need to aim and then you can appropriately scale back so we really looked at what was required from the pro audio market and this is not to say that this is a set of api's that can only be used in musical applications or or pro audio sort of workstations and things that really can scale down to games and to lower sample rates and and lower quality abstractions and so forth but we wanted to start at this point so we had requirements of latency and jitter that were very important to us both for medium for audio we want to obviously no restriction on sample rate which was a problem with previous versions of the OS multi-channel awareness throughout the system not just with devices but also with codecs with audio units with the whole way that we think about audio in the system and we didn't want to be limited to just some small subset of how we can represent audio data we wanted to have rich abstractions that can be applied throughout the system and the session after this one will go into some more detail about how we represent data in the different subsystems of core do so let's start at the bottom and we'll kind of work our way up so the I'm not going to spend a lot of time in the cordilla hell part this is very much a low level interface to devices there's a lot or a lot of abstraction here for the devices but there's also a lot of specifics of I state that you need to manage and if you are in a situation where you really need to interact very intimately with the device this is the API that you use this is in the Quartier framework we affectionately call it the hell the hardware abstraction layer and you will get all of the characteristics of devices published here configurations system device preferences management of the device status and all that kind of thing and for MIDI we've got the committee framework and this is really the api's that are published through here for transporting a MIDI data through the system both in and out of the system through drivers as well as inter application in Jaguars committee we have a concept of virtual sources and destinations and we found that a lot of developers from Mac OS 9 and the many services that are ms and 3 mitigate that one of the things that have been missing from the way the system is used is an IAC bus which is an indoor applications bus and this is basically just a driver that looks it's a software that looks like a driver that actually gives you a way to kind of connect MIDI between different apps and it looks like you're dealing with drivers rather than sort of having to do extra work to look for virtual sources and destinations so this will be a new feature in Panther and the other thing of course with committee is needing to configure device and publish device characteristics and so forth I'm going to go to the demo machine and just very briefly walk through the audio MIDI setup applications just as a way to kind of give you some sense of how all this is put together so this is the Audio tab we're going to we've got the mark the unicorn 896 box here we use different devices each year with you imagic and delta m audio devices which always use the motor one this year this top part of the section here is system settings and these are the the default input and output they're typically the devices that are used by apps like iTunes by quicktime player by games and the user can typically see in the sound press i can see like these are all the output devices I've got and they can choose which one they want to use as their default output and another thing we introduced in our in Mac OS 10 was it was a distinction between the device that you would use for playing audio on and the device that you would use for things like sis beeps and throw forth so the system output device is the device where your beach go and the bottom part here is is a device for City configuration you can see for 896 that it has different clock sources and you have also in core do you have this concept that a device can have different streams and this is the way that the device can publish its capability so on 896 we have two streams we have one stream and here's the first stream and it has a channel capabilities and this would be like the analog a channel iOS and the a channel digital i/os on the device and the second stream would just give us two channels and that would be the spit if input or output and you can see here I've got like sample rate options for the the streams and typically a device will run as the same sample rate across the different streams one of the optimizations that we didn't do enough work on in Jaguar that we've rectified this in panto is to turn streams on and off you can think of a stream as kind of a logical unit on a device and you can turn a stream off and that can tell the driver to not do work on that stream and that can be done on an application and application basis and that can make the load of running the system a lot less than if you're just running a device that may have 50 i/o streams another thing that we're doing our for Panther is trying to address the problem of how to configure your setup for surround systems different speaker orders and so forth and on your Panther CD you'll see there's a this panel for the speaker configuration now you won't see this if all you're looking at is the device with two channels because it's just two channels and everyone pretty unpretty much understands left and right but if you have a device like 896 that or the metric halo devices or pci cards or some of the USB devices with more than two channels you'll see this speaker configuration utility and i can basically tell the system that these are the on my device the way that I've got this wired up in the studio or in the home whatever is that one channel one is going to be this speaker channel two is that speaker and you can set these up for different types of of surrounds now you can't have all these different surrounds active at the same time so really you choose the surround speakers that you've got in your particular location and then you basically say well okay I've got less run here and I can make a sound and I can get that sound out there we're going to change that to a pink noise other than a sine wave sound I can get the center cell and lifestyle so forth and that can you know make the whole configuration for the user a little bit simpler and then we will do mapping in the output units to remap channels and they'll probably needs to be some API changes for that we're trying to look at how we could push this into the existing output unit but we're not sure at this point if we can do that without actually doing the wrong thing and we'll be publishing some details about this later on and there's some more details about how we represent this stuff in the audio format session if i go for the MIDI section now I've got a one USB device here from Roland it's a mpu 64 and it has four midi ins and outs I've got a studio set up here and look let's not say I'll leave that over there early software so so one of the things I wanted to talk about with this app is a little bit of confusion about exactly what it's meant to be doing and this really reflects as the audio side does this both of these apps if you're developing and you're familiar with the API they really reflect the a lot of the structures that are in the API themselves and so you can sort of see this is like this is what the user sees but also as a developer we can just with a couple of grafting concepts here we can understand what you're seeing from an API point of view so this is a driver this driver has what is in MIDI is called our MIDI entities and it has four midi entities which are these pairs of in and out ports and the in and out port submitting our endpoints and each MIDI endpoint takes a full MIDI stream so that's 16 channels and you're able to you know talk to whatever is at the end of that stream will get the data from the end of that stream so this this configuration here has for MIDI entities and each MIDI entity has an end points were in and out and what we're doing here is describing the driver now what is out here is just three devices that I've added by just doing this add device thing and I can add a new external device and i'm not actually creating any it's not like i'm going to really create like a whole patch flow here all i'm doing is describing to the system what I've got plugged into what the system knows about which is the actual driver you can't really do through connections here in the sense that that all like just automatically route you have to wrap these with your cables you're really just describing to the system two keyboards or the control surfaces or the modules that you have on your system and you can save different configurations and if i unplug the USB device the driver will go offline and you'll see that reflected in AMS and what the what can be done here is so on rather than sort of showing this is port four of us be you know SMP you 64 you can actually present by query and call middies api's you can actually present my synth module as the destination for that to the user now the thing that's missing from this that I'm sure many of you have dealt with OEMs and free midi is some way to describe what is actually in the device itself if it's a roland you you 20 or you 220 thing what patches does it has and what is the current patch banks that I've had what are the capabilities of the device we've worked with the many manufacturers associations to describe a document format for that using XML the the spec has been ratified and has passed through all of the MMA processes for doing that we wanted to do this through a body rather than as an apple sort of only initiative just to make this something that could be broadly adopted both by manufacturers and other computer companies besides ourselves so that for the user there's one data format the the documents not available yet from the MMA psyched I will be soon as we're going through the final stage of actually making the documents are clearer about what it's contained so that there's you know it's an easy-to-understand document you'll be able to also these XML files yourself to describe custom patches and we will hope that they'll be websites available and manufacturers will publish them for their devices and so then the user can see my sense module and it's a rolling since module yada yada and it's got these patches on it and just like with OEMs and free mini on Mac OS 9 and it should be as easy and experience for the user on 10 we can go back to slides please right so that's the bottom let's get into audio units so dear units we have a logo and that's the logo after took some some effort i can tell you so so audio units and i'm going to say that these are generalized plugin format and by generalized i mean that it has very many different uses it's not just for effects it's not just for software instruments and what we'll do today's later on as a demo for some other what air units and i just want to give an overview generally if what audio units are the types of audio units that you can have and they're kind of categories and their functionality so how does it work an audio unit uses the component manager and we use the component manager because it has some nice features about it has discovery mechanisms find next component you can just make this call and you can specify the type of audio unit that you're looking for and we have different types once you find that the audio unit that you want and you can either be doing this programmatically or you can present menus based on what you've discovered to the user then you open that component and once you've opened two component you get back what's called a component instance so you can think of a component in object-oriented terms as a class specifies an API specifies what that a member of that class can do and the audio unit itself is the component instant is like an object an instance of the class and so audio unit as a typist is type desk to a component instance so you've opened your audio unit so what do you need to do to it well it can be as simple and complex as the type of audio unit you're dealing with what you want to do with it and the first steps really for an audio unit is to look at the properties that it has and properties really represent in in this kind of sense of the property mechanism it's the state of the audio unit and so you it is a general mechanism it's extensible we define property types you can get information about the property how big is the property can I read the property or write the property or both I can use the property making mechanism to find out the state that the unit comes up in and I can change the state of it and so forth and it's really the way that you manage that fundamental state of an audio unit and then once you've sort of set up the state of the unit then we have a second phase which is initialization we split these up because there's often a lot of things you might want to discover about what an audio unit can do particularly if you're in like a hosting environment before you initialize does it make it able to render and able to do its job and so audio unit initializes this concept of of allocation in order to actually operate and once an order unity is initialized and it's considered to be in a state to be usable and that is the the one call that you do really to use it is audio unit render and there's a you'll so I'm not going to go into the specific details of the API so that that there's a lot of arguments that you can pass to this and flags and so forth some move fairly well documented in the headers and you can ask questions on the list now we're ready to go we're going to call a unit render that where we're going to get inputs are the form you can get input for an audio unit from two different places we wanted to have with audio units an idea that we can connect them up into process and grass all we can use them independently either just one off or maybe two or three and then I want to provide data to it but I don't want to have to be an audio units to provide data to another audio unit so you can also have a callback function or you can have the connections for there's two ways that you can provide data to an audio unit and at this point we're talking about audio data it's an audio unit and so when you call audio unit render on an audio unit it's going to go if it wants input and some of their units may not want input and we'll have a look at those in a minute it'll call it its input proc or its connection for a simple data when that returns it's now got input data to process that processes that data and then you get that back in the Buffaloes that you provide audio unit render and you're done well are we done no not quite because one of the things you want to do when you're processing audios you want to be able to tweak it you want to be able to set delay times differently you may want to be able to set frequencies if you're talking about volumes going to change volumes and so forth so all of these are considered as real time operations on the audio unit things that you can do to the audio unit while you're in the process of rendering it and we abstract that into what we call parameters an audio unit using the property mechanism publishes a list of the parameters that it allows the user to manipulate it publishes things like what's the range is a parameter what kind of values does it have and the DB or Hertz or maybe just a generic 021 sort of parameter there's a whole bunch of different types of parameters that an audio unit may publish and it really discs and we've seen some third-party units that have a couple of hundred parameters a lot of our units may be fairly simple they made just two or three parameters it really depends on what the unit is doing and what the developer of the unit wants you to be able to manipulate so we have effects units and that's really kind of the meat of a lot of where we think the third parties will be developing units in Jaguar we ship various filters there's high-low pass band pass high shelf and low shelf filters we ship a reverb unit and the reverb unit has quality settings that you can adjust the quality really determines how much CPU that the unit's going to take at runtime and it actually affects the quality of the rendering and so things like games and stuff they may be less concerned about a very high quality and more concerned about the load that the unit's going to take and we have a digital delay you know we have a peak limiter and Panther was added a multiband compressor unit it's a full band compressor and it's so pretty nice actually and you know we're not going to publish a couple of hundred audio units as accompanying ourselves this is for us the big part of audio units is what developers are going to do with is this is where it can get very interesting and very bizarre sometimes so to sort of summarize that we create one we've got the state management we've got the resource allocation we've got rendering and we've got the control of the rendering well is that all we need no typically if you if you used logic or if you've used digital perform or any of these for the hosting environment and type applications you want to present some kind of view to the audio unit so the user can interact with them we've published a generic view which will just query the parameters that the unit tells us and we will assemble like a bunch of sliders and that's you know it's not bad but it's probably not as as good as what you can do if you really understand your DSP and so a developer of the naughty yoona can publish a custom view and some of the views if you've seen these are pretty creative and pretty interesting and jagua we only had the ability to publish carbon views in panto adding support for cocoa so you can publish a class that implements a cocoa protocol and it can be discovered from Asking naudia unit for its cocoa view and the cut a carbon app can put a cocoa you I up in a separate window and the cocoa lap can put a carbon you I up in a separate window so there's not a lot of extra work on the hosting side to do to deal with both Aziz and we think that probably a lot of developers would be very interested in the cocoa you are so that's there now and the other side of this is communication and control mechanisms in Jaguar we have a parameter listener API it's in audio toolbox audio unit utilities dot H and it is an API called au parameter listener laws gets hold off when I'm wandering around but I just can't stand still so I'm going to keep wandering excusing so the parameter listener when when we were looking at this to design this there's two ways that you can do this you can have a UI that or an asset is going to you know 30 times a second or whatever it's going to pause for the values of the parameters see if anything's changed and it seemed to us a lesson elegant way of dealing with this so we decided to do a notification service and the notification services aimed at allowing anybody who wants to know about a parameter changing on an audio unit to be able to listen to that parameter to see that that parameter is changing them to react appropriately and then when they want to set a value of the parameter if they just use the standard audio unit set parameter call that's not going to invoke the notification you need to use to a parameter set for which is in this in this header file and that basically then we'll tell the notifier the notification mechanism that hey if you've got anyone who's listening to this you better tell them about it we've decided to extend this a little bit in Panther and to include the ability to notify state changes from the a audio unit property and so you can do both parameter chain and property changes using this notification mechanism now one of the important things about audio units with all of this kind of stuff is that we're never going to know everything that we need to know about every possible audio unit that every developers ever going to write and so there are mechanisms in place to be able to share private data with private property IDs and all this kind of thing between your audio units and your views or this kind of stuff and so this mechanism can be used to to communicate that there's a need to go and get some state that may be in the audio unit if you're the view and it can be done in the thread safe manner so you can call this sort of API from your render sphere as well as from a UI thread the other weakness that I think we had in Jaguar with the carbon you is that we had this idea if you're doing automation that you need to know like start and end state as you're doing automation and we sort of put that into the view and that really wasn't the right place for that and it kind of restricted the sum of the use of it where the audio unit may know that it's a start and end state not to view and so we added to the Panther services for this this idea of the beginning end of a parameter ramp state so you could imagine if you've got a control surface when the user touches the control some of these control surfaces a sense it is to actual touch so you could touch the control and then that would be a signal to say hey I'm bout to start ramping this parameter and the UI could reflect that the users types that control with the changing button in the UI and by putting this into the controller this means that we can also support this with Coco you eyes as well as carbon you eyes without having to add additional logic to the Coco you I so I think this is a very nice addition to this and ever said before this is real-time thread safe and will continue to work very hard to ensure that that remains true okay you all asleep yet was nice I'm actually still with me okay so we've got a fax and this is really I think we're a lot of the work is done for third parties but also another area that a lot of third parties do work in these instruments and software synthesizers and the idea with this of course was is that you're not going to be processing audio although some sense will let you have audio input into their synthesis algorithms but also the main part here is that Bobby will be MIDI data coming either from a host apple from an external keyboard and so we have an extension to the basic audio unit API to a musical instruments and get my clicker to work and we call this a music device that's in the music device dot H file when this just adds semantics for starting and stopping notes and for the controller's notes we've got a couple of api's there that really just talk committee so you've got you know music device send MIDI and you just take the the two or three by midi for channel and then you've got another API for dealing with the sis X and the other sorts of extended midi messages we also added an additional API for the music devices because we didn't want to just talk midi as a software architecture we don't really have to abide by the same limitations as hardware does with with the MIDI stuff so we have an extended API where you have the ability to have a software send have more than just 16 channels we call them groups in in our parlance so you can have an arbitrary number of groups you can actually group notes into different instruments playing on the same group so i could have three notes that are playing on different sounds and they're all playing on the same group and i can just send control messages to that group the extended protocol works in a similar kind of way to MIDI if you kind of break nidhi down in a suit way and you can express all of the control semantics of meeting in the extended protocol that it gives you some flexibility you can specify pitch using a fractional floating point numbers 60.3 60.5 will give you the half time between C and C sharp and away you go so this is a something that we're seeing a lot of two parties using the music to buy stuff there's some very interesting third-party cents out there already one of the things that we were working on this is a to address some concerns were raised by ways who are a very large developer of audio processing applica plugins and they wanted to have a way to do offline rendering offline rendering is typically done when you want to actually process the file of data and you want to look at the whole contents of the file not just in the real time all all of the audio unit development that we've done up until this this particular unit is really aimed at working in real time and so it has constraints about having to work in real time you need to report like latency you need to report how long it takes you to process sounds in terms of the time delay between input and output we've got an offline unit you need to be able to look at the data all of the data that you're going to be asked to process before you process so there's to render phases with an offline unit there's an analyze phase and then there's a render fo so do you think of reversing the sample order in a file you need to actually like start at the end and work work your way back if you think of an offline unit that may normalize you need to look at the whole audio data before you start to process it so you can do the normalization and there's not really any additional ati changes for this there's a couple of different flags there's a couple of properties that are specific for an offline unit there's a new audio unit type a you o el audio unit offline it's not in your Panther headers because we're still revising discussing this with waves and some other companies and will be that this will be published in Panther and we're getting pretty close and we can always if people in sedate and contact me and I'll send them the spec in the code that we've got at the moment and we will ship a reverse annoyer unit that does reversal in the SDK at some point as an example and they'll be code there as well to have you host these offline units okay so when I said generalized audio unit we're still sort of like in the the general field at the moment about normal types of audio units now let's look at some abnormal types of audio units with and one of those types is mixer units so in in jago we had two mixes a stereo mixer that takes mono or stereo inputs and gives you a next single stereo output and in jago as well we had a 3d mixer the 3d mixer will take multiple inputs it will have a single output and will be in either 24 or 5 channel output and the four channel is like a quad setup which is pretty much what we've got in the room today and 50 where we don't actually do the 3d stuff into the lse channel we just do the 55 channels and what you can do with this 3d mixer is quite a lot you can pan and localize within a 3d space you have a number of on-off options and I'll get Chris Rogers to come up and we're just going to give you a fairly quick demo of the CD mix [Applause] well thanks still setting me up there last year I gave a more complete demonstration of the 3d mixer but some things have changed since last year because developers have made some requests of the mixer so we put some new features into the 3d mixer and we can have a look right here what I have right here is a simple little app simple little user interface on to the 3d mixer and it has a number of different controls for choosing the type of rendering for the source in this demo there are going to be three sources or it can be up to three sources and you can choose equal power panning a simple spherical head model an HR TF model and these first three are for rendering the stereo output and then the last two sounds field and vector based those can be used for stereo quad or 50 then over here we have some checkboxes that let certain fixture features of the mixer be enabled or disabled for individual sources down here we have master volume control and here this is kind of an obscure slider that controls distance attenuation that is when sources when sound sources get farther away they get quieter but how much quieter do they get say if they're ten meters away or 100 meters away how much quieter did they get this can control what that curve is that the follow up and that's a feature that developers have been asking for so down here in this part of the display we have have some meters which the 3d mr. now supports lives metering you can meter the input levels and output levels both RMS power and peak levels metering is something that we put into a couple of our other units and later on today we'll see the new new audio unit matrix mixer also supports this which is kind of interesting so maybe I should just bring a source in and see ok so maybe I should simplify this a little bit and you have turned all my stuff is off okay here we go so I'm using vector-based hanging right here and I turned these these sources off so well not listening to those we're just listening to this blue one here but if I put this dot right here then we're essentially just in the center to center channel hold on we have the helicopter coming I'm sorry there it is let me turn him off okay I think now maybe our meters will I'll show this a little better ok so I'm about straight ahead here I could be coming out of the center speaker and the channel ordering down here is left right surround left right center so ok now we have left and rear rear left we have probably surround left around right ok now let me change this sound I'll use this end this sound right here okay okay turn ok it looked like we're running a little bit short on time so I'll try to wrap it up here basically the new features that are the most important are the ability to turn offer on individual features here and that affects the performance that you're going to get this distance filter is a low-pass filter that makes sounds Sun more muted is for getting further away and some developers had some comments about that that they didn't necessarily want their sounds to get more muted so there's a way that that can be turned on or off and any of these any of these characteristics can be turned on or off separately and if art performance goes we've made some optimizations to the mixer and to give you an idea of the kind of performance that you can get for in for an individual source using equal power panning of stereo I think I'm pretty modest machine like an 800 megahertz g4 you can get a single equal power source at I think it's point one eight percent of the cpu and h RTF which would be our high quality stereo panning mode that's at about point five five percent of the cpu / source so little tiny bit more than half a percent now I have a we'll just move on okay go back to the slides thank you Chris the other demo that we work in the show but we're running a bit short is a very speed unit and that's also new in panto and you're able to make have your fam come into the various speed and it can go faster or slower and go you know Connors oh what's the chipmunk effect so show you that later on if you want to be in a lab tomorrow from one o'clock till six o'clock the QuickTime lab and we can give you a demo there if you like ok so in Panther as well we have a matrix mixer matrix mixer is a very powerful beast and we'll be going to some detail about that in the next session all mixers have metering capabilities in Panther so it can be you'll see some of that in the next demo next session the other type of audio unit that we have is a converting unit will be talking about the audio converter in the next session as well and this brings some of the functionality of the audio converter into the audio unit world essentially all of the conversion operations to deal with PCM audio so sample rates bit depths channel mapping and so forth and all of this is configured with properties and actually there's very little configuration of it for you to do because just describing to the audio unit what you input and your Apple format is is enough to tell the converter what work that you do and the converter unit functionality is that is included is a part of the output unit the output units interface to a device there's no additional latency there's no runtime cost to you for dealing with the output units and you can manage how much work the output unit does for you by seeing the difference between the format of the device which is on the on the output unit and the format that you provided in Jaguar we only did output an hour output units to input so that's why they're rabbit is output units now in Panther and the output units will do output as they do in Jaguar on bus 1 and element 1 or bus bus 0 mean on album element Marnell bus one you can do input and so what does this look like it looks like this so I'm going to use the sliding I they hate us using these things that I love it so here's your output unit on the output side here is your device output this is on bus 0 and then this is either a call back or a connection that you make to the audio unit and this is if you're using this unit this is what you're doing today in Panther you can also see if there is input available there's a property to query that and then the devices input is actually on the input side of the output unit confused yet I took me a while and then the application actually calls audio unit render for bus 1 and that's where it gets the input from the device and it can do the conversion for you as well so if your device is 20 channels and you only 14 channels and it will remove those 20 channels and just give you the four channels and you can tell the output unit which four channels that you want from the device including rate conversion all this sort of thing and if your device just has input or just a and you need to know when to get input then you can get a call back and it will tell you hey the inputs ready go and get it and so that's a new in pampa okay so Audio Units we talked about connection we talked about wanting to be able to connect all these up and we have a structure in the audio toolbox api's called au grass and a aggress manages the connections between these units it'll manage the state is it has a very abstract representation of the graph and then the graph itself has a couple of different states you open the graph and that'll basically open the audio units that you've described as being a part of your graph then you can initialize them and then you can actually start the graph running and you can update the state of the graph while is changing you can make or break connections you can have a bunch of audio units sitting off to the side that maybe one channel one chain into a mixer or something and then you can just connect that chain in players down disconnected and the graph will will do all of this it will manage the different threads of the thread that you're doing the rendering in the threads are you doing the state management from and it'll just make this a lot simpler to do then you can write your own code but if you going to do this it's a good idea to look at another API in the toolbox is the music sequence API music sequences of a collection of events and are there's really two different types of events and we have two different types of tracks they're still just basically a track but a music sequence has some concept of cancer music sequences turn talk about their timing in terms of beeps and so the tempo describes how many beats per second if you want to deal in seconds you can just make a sequence with the tempo advantage of 60 beats per minute and then you can deal in seconds and then you've got any number of events tracks and the event shacks can take any number of different types of events you can have MIDI events you can have user events where you can put your own data in there and you can have parameter event so you can talk directly to particular parameters on different audio units and of course the connection is that you have a sequence and you connect that up to a graph and we'll show you that in a minute you can create a sequence from a MIDI file or you can create a sequence programmatically you can once created a sequence you can save the sequence off to a MIDI file will only save the MIDI type of dance that are in that sequence to the MIDI file obviously we won't do the other ones and you can iterate over the events you can loop mute solo tracks when you're playing them and you can copy-paste insert edit them and these can be edited while the sequence is actually being played in real time and you can target the tracks that are in a sequence to do two different destinations you can target them to a specific audio unit that's in a graph that you've attached to the sequence or you can target a sequences track to a MIDI endpoint so you could be sending primer sequence that you're playing directly out to a MIDI device and if you want to play it then you've got to have some object to play it and that's the music player music player has very simple start and stop semantics it has pre rolling so you can particularly with software's is very important so you can actually pre-roll chase your events to whatever the time that you set one of the things that we added in Panther is the ability to scale the playback rate so that you can if you were wanting to do some work to synchronize this to some external time source or you just want to play the events back faster or slower you can do this with this additional API so if we go back to what we started with connecting it all together this was the kind of the mess of the api's and then if we sort of put some blocks around it this is how we kind of see it all fitting in I'm going to go back to my laserwriter here laser pointer so here's the audio units here's the howl output to buy his the MIDI device the audio units are sort of connected with these orange bars you don't really see this connection here that's really a part of the output unit and the state that it manages in this case here which is very similar to the 3d mix if you if you remove this filter and you think of the demo the christed then really what he's doing is sort of this part of of this whole kit and kaboodle so we're using the audio file API to read the file data from the disk the converter to convert that into a format that we can use for the 3d mixer and then the that just goes into the as an input to the 3d mixer and in the demo that Chris was running we had three inputs into that mixer one thing we didn't chose is this side of it so you can have a music sequences collection of tracks you can address the track events to different nodes in the grassland you can make the association between the sequence and the graph and then the play actually owns a sequence there's player plays of sequence which starts the graph or if it's just talking to many then it starts a MIDI scheduler and and the way you go and of course you don't excuse me you don't need to use all of this you can just use this vid or you can just talk to the lower levels of the system where you can just use audio units without using their graphs and so it's a very modular architecture with lots of components there lots of complexity to get lost in we are working on documentation it has me to know and we are going to do some more work on some sample code and we'll see in the next session are Doug's one of the things Doug's been working on which will be in the SDK and so that's pretty much all lid we provide those sort of services I'm not going to read that you can read it yourself and yeah don't panic you don't have to know all of these things you really need to understand well what is it that your app needs to do and then what is the appropriate API that you need to do to do that and you can just ignore the stuff that you don't need to do ok so there the Four frameworks Claudio for drivers call media committee the Audio Units really that framework just publishes the API for the extendable audio units and codecs and then the toolbox for formats files when talking a lot about the format stuff in the next session wrap up roadmap there's some of the sessions we have the audio and quicktime session tomorrow which is a going to be very interesting session i think if you go to feedback forum for our asses after then and nick and craig and myself and others will be there who to contact and for more information we've got the choreo or the audio technologies for Mac os10 website that ipod app electrons flash audio you