WWDC2003 Session 100

Transcript

Kind: captions Language: en good afternoon and welcome to the i/o kit session I'm Craig Keithley I'm the i/o technology evangelist and Apple's Worldwide Developer Relations group when we started about five minutes ago or 10 minutes ago I could have probably identified everybody in this room it's hard to go up against Xcode one of the things that's toughest is properly architecting and writing IO kit kernel extensions to get optimum performance you can improve on this you can prove on your techniques by doing multi-threading as a common well almost want to say misconception that you can't do threading in kernel extensions you can and to go into that will bring up Godfrey good eye my name is Godfrey van der Linden I'm a iokit architect and like the vast majority of people who I expect it to be here I would probably prefer to be an X code right now myself another thing is the the handouts suggest that I shall be talking about memory even though memory is interesting most of this presentation will be about threading the new piece of hardware has got some interesting memory issues and I will be available in the porting lab after this and on Wednesday if anybody wants to talk to me about how to set up memory Maps on the new hardware but there's no formal session on memory on in this presentation okay so as an introduction Mac os10 has something like a hundred threads running in the system at any one time even when it's idle I mean I was running top on my system today and it had a hundred 40 threads going s what makes kernel programming so much fun when you've got a hundred 40 threads operating inside that environment at one point in time it can lead to some interesting mind-bending problems that just is why I enjoy kernel programming a lot so in this session I'm going to be discussing threading generally how sort of high-priority threads work inside the system and also I'll be talking about iokit threading and then finally I shall discuss the teardown synchronous teardown of device drivers when you get a hot unplug it's sort of it's not really threading but if you don't know how to do it properly it's very easy to get nasty explosions so what I'm hoping you'll learn is a better understanding of how thread schedule on Mac os10 how Iook it does the synchronization I get synchronization model is very unusual I've never seen it in any other operating system and probably because it's my invention I think it's very very cool and also Mac OS 10 is hot unplug we took to attempt so that before we got it right and I don't think that's really been presented before so I think it's really it works very well so this is its presentation okay so the the first part of the presentation will be on threading specifically how threads work inside the operating system I won't really be talking about threads and the kernels so much as how threads interact how the scheduler interacts with it how the dispatcher works Mac OS 10 doesn't differentiate between kernel threads and user threads yes we have different priority bands but we don't really differentiate the way they operate so over the next few slides I'll be discussing the thread priority bans how the dispatcher works what the scheduler does what it means to take high priority because it's different to what most programmers think and also priority inversions which is a very standard problem that we're having now so these are the priority bans as you can see there's quite a few of them and there's a few there that aren't really threads at all the primary interrupts and idle threads aren't really threads but you can consider them thread context at least to the extent of when a primary interrupt is running no thread is running and also when the idle thread is running by definition no thread is running the bands that we would like you to be in is the regular user area but most high ends hardware and let's face it that's what I orchid is all about tends to have tighter requirements and they'd like to go for higher priorities now we're experiencing a lot of I call it priority arms races people are going for higher and higher priorities and it's really degrading the overall system performance so I'm hoping in this presentation to convince you to get out of the real time band if at all possible and down into the top of the user band because I think that's probably the the best place for most of us to be okay so Mac OS 10 is a dispatcher based system there is a subtle difference between what a dispatcher is and what a scheduler is we do have some sort of scheduling that we're basically a dispatcher based system what that means is that dispatcher takes the current thread that's executing clocks it then selects the next thread and runs it so what is blocking mean well generally a thread and our operating system as I said earlier we I had 140 threads running but in fact they were all blocked waiting for some user events waiting for some IO to complete waiting for the lid to close or open or for the battery to run dead for all I know and that's what we mean by blocked in our operating system those threads are put on to wait queues and we really just ignore them so yes we have 140 threads running but they're really all asleep which is the best way to have a thread in my opinion the next thread thing that happens is a pre engine so what is the preemption it's essentially you've used your quantum and the system says okay I'm going to put you on the end of your priority bans run Q we'll discuss that in a second and then I'll select the next guy to run every time the dispatcher is invoked it selects the highest priority thread available and runs ER so that's really has some very potentially nasty side effects if you have an infinite loop at high priority and that high priority thread runs to completion then nothing in lower priority will run so let's say you overload the real-time band remember earlier the real-time band is the highest priority band in the system if that overloads then you're not going to get any time at all for the IO band and the IO band is probably where you're trying to store your data onto a disk or take it off a disk so you've just stopped the system from doing what you need to have done and that's probably not what you're after so what does the scheduler do well it's job is sort of an oversight committee we do have a scheduler it will get better in the future the scheduler we have right now is essentially for time share when your thread has run for long enough we will change your priority down a little that doesn't necessarily mean you're going to start running slower at least not straightaway if there's no other thread that's competing with you that's runnable then you're going to continue going and it won't make any difference if your priority is down however if you are competing with another thread the scheduler is job is to try to make sure that the system balances its loads appropriately and the other thing is that that aforementioned spinning real-time thread and we had this problem early on with the system spinning real-time threads will not give any time to the system including to the keyboard so that you can stop it so one of the jobs of the scheduler is to say hey this real-time thread is taking far too much time in which case it will taint it over too timeshare and then say oh by the way I've been running from eight seconds so it gets depressed very quickly which is a good thing because it means that you can use kill -9 and get rid of the thing it came quite late really the original development of Mac OS 10 when we got the real-time phrase that was quite a common problem because you know everybody's written infinite loops and a real-time thread infinite loop meant taking the big hammer out and hitting reboot and that's really really painful when we didn't have journaling file systems okay so what does this usually do essentially the communication mechanism between the scheduler and the dispatcher is the Run queues now earlier I mentioned that the dispatcher finds the highest priority Fred that's runnable in the system and then runs it well that's the Run queues logically you can think of it as run run queue / priority in the system and then if one thread is runnable it's in one location in the run up queues and the scheduler just manipulates the locations in the run queue also the scheduler collects statistics so that you can do things like top and latency and a number of other tools so that you can find out what what the system is really doing for and on your behalf so in a time share thread once the thread has run for so an example of what the scheduler does is time share thread as I mentioned if your thread has run for sufficient Quantum's we will drop your priority dropping your priority isn't as I said really a bad thing it's only sometimes bad if you're using so much CPU power and some you need that much CPU power and another thread comes up and say another task the user launches another task and then your thread will fall out well you know the user did launch that other task perhaps he really does want that task to run so let the timeshare do its job except for when you were certain that the user really really cannot afford to let any CPU go to the other guys in which case you would use different things and as I said the infinite misbehaving real-time threads is another example of what the scheduler does so what does high-priority really mean there's nothing really that can make slow code go fast if your code is slow high priority will not make your code go faster you will get slightly more cpu time but it it really is measured in percent maybe one or two percent more CPU time higher priority won't give you high faster code and the only way to get you faster code I'm afraid is to run your code through performance analysis and clean it up it's very easy to write bad algorithms unfortunately what high priority does give you is it gives you a reasonable chance of running with a very low latency so your thread is blocked midi event comes in for example and Mac OS 10 will probably get you a thread run in around less than a millisecond on average our max jitter I haven't run this for a while but the last time I saw it the max jitter for a real-time band in no competition was running at about 600 microseconds or something unfortunately in the real world there is always a little bit of other competition at that high band so I think we're running our jitters at around 3 milliseconds again I'm not sure exactly what those numbers are of course when you have high priority then you can end up very very easily using so much CPU time that you're not allowing the low-level paths of the system like disk i/o anytime at all it's a bit of a shame we had recently a developer raised a problem which was they were using so much higher priority time that the firewire thread the firewire work loop which is an i/o thread wasn't getting sufficient time to even acknowledge packets on the bus and when that happens you start getting weird little disk errors and the system itself hasn't really got time to clean up because you're using all of the time at high priority we call that a priority inversion and priority inversions are really hard to get to get rid of and this is really the biggest problem with arms races if you in a priority inversion that's going to cause some problems and it may take a very fundamental redesign so the way you've set up your workloads so how do you decide your thread priority really it comes down to exactly what your latency requirements are it's not what performance you're after it's what your latency is user interface events for instance a keyboard for a midi sequencer or something like that really does need very low latency because a human has said I will move my finger and if the human system the the detection system is aware can't hear us the sensors don't hear it within a certain amount of time then the keyboard feels wrong and that time is very short I mean for computer time it seems enormous it's about five milliseconds but five milliseconds isn't really very very long on a modern operating system especially because our standard quantum is 10 milliseconds so the higher priority that you want the if you really do need that extremely low latencies that's when you go for high priority but you really want to be certain that you need a very low latency if you're reacting to data off the internet well frankly it's who cares what the performance is you're dealing with 30-second timeout so anyhow now I'm not suggesting that you go time share I don't think tie chair is appropriate if you're doing some sort of stream based information processing however you probably don't need to be real time because the whole internet itself is arbitrary and finally there's sort of low latency it's stuff when you're waiting for local resource or something off the disk a local firewire or something like that it's sort of low latency without being ultra-low latency so that's how you would use your bands I would suggest for extremely load you would use the time constraint stuff the real time banned for who cares I would probably use hi use herb and possibly below the carbon async threads but you can play with it a bit and then for low disc stuff I would suggest that you go to the top of the user bands and you disable time share altogether these are all things that you can look up on the ADC websites to find out how to do it there are priority inversions I want the highest priority except for when I don't party inversions can really happen almost anywhere in the system but the most common ones we're seeing is again the real-time bans Mac OS tens real-time band is very very good as extraordinarily powerful but unfortunately to get a really good low level maximum jitter we've had to give you enough power to hang the system effectively and that means that your code now has to be far more complicated because you have to work out how to back outs your high priority thread to give the rest of the system some time now traditionally on most operating systems Mac OS 9 and Windows for instance iOS really high priority there is nothing you can do to get them out of the way and you would have to take whatever jitter is around with Mac OS 10 we have deliberately chosen to make the real time thread the highest priority threads in the system even higher priority than iota which gives you extraordinarily good jetta characteristics but it comes at the cost of complexity so there's a couple of priority inversion strategies the best priority best strategy of all is get out of that high priority band if you're experiencing priority inversions drop your priority if you can if the Deuter is appropriate examiner there are some wonderful tools in the system my favorite is latency latency will show you a histogram if the performances in the system if you can evaluate and have really hard numbers for what performance you need latency will let you know what priority band will work well on your target system so if you can lower your priority that's the best thing ever if not you're going to have to complicate your algorithm you'll need to split into produce a consumer model where you have small amounts of work to be done at very very high priority and larger amounts of work done at low priority so for instance if you're streaming off a desk you would have a low priority thread in your system and you know I don't usually recommend having multiple threads but this is the time to use it you'd have a lower priority thread in the system that's feeding a high priority thread but at the cost of introducing some latencies the high priority thread would just take whatever data it needs when it's available and that way you get a producer consumer it's pretty good it's complex it works very well indeed the worst choice it's really bad because it doesn't give you 100% of the CPU is to deliberately say I'm going to let the system have some time so approximately every 10 milliseconds or there abouts one buffer every two buffers or however it is that your workload is divided go to sleep for a millisecond cause you sleep for a milliseconds and then you will guarantee the system or at least some other threads sometime I don't like that the problem with this solution is that if you're not competing with anything else then that extra 10% or 8% after the system is used it is gone and you can't use it and you're only doing that to save yourself the complexity of a good producer/consumer queue or lowering your priority in the first place you see if you lower your priority you can use 100% of the CPU there's a really nice anecdote iTunes started early on with timeshare threads for it's ripping and the thread actually drops in priority because it uses 100% of the CPU one of the cool things I'm not sure if you've done it I've recently done it with the AAC encoding I was ripping my entire record collection over to 128-bit AAC and the system was very very performant while it was going and I never had any idle time on the system at all it was just 0% idle that's because the ripping was a very low priority so look out for lower than regular priority is actually a good thing if you're using a hundred percent of the CPU anyhow that's just the introduction to threading there is a lot more to be said I could talk for hours but unfortunately I don't have it we'll have to move on the next set is work looping I a kid this is essentially how I occurred does its synchronization I shall be discussing the work loop and the event sources in this part if you're a traditional iokit driver this is the mechanism we're recommending and it's really quite hard to avoid now unfortunately work loop itself is an unfortunate name the the way it was originally designed we did have a thread that all iOS went and we could guarantee single threaded access to hardware because we only had one thread that talked to the hardware but the difficulty is that the i/o systems were taking context switches which was slowing down all i/o so we came up with this idea of the gate and the gate allows us to schedule IO on hardware directly without having taken a context switch and you know what's a gate well the gates a lock it's a recursive lock it's not really very complicated at all and it's sort of obvious but it took us a while to come up with it and it made a big difference in our performance so what a work loop really is on our system now is it's a container for the gate which is a recursive lock it's a list of event sources that need to synchronize with respect to that lock and by the way it has a thread yes okay it has a thread in fact the threads optional one day in the future I'm going to get rid of the thread and only do it if you have interrupts event sources so a work loops gate the single threading is provided by the work loops gate being closed across all event source action routines I shall define what that term means in a little while so traditionally eunuchs traditional unix solution for MP is to have one big like one goober lock that protects the whole operating system so whenever you need to do anything you would take the Euro burlock and then you would be safe until the uber lock gets dropped and there would only be one lock and naturally you get contention and only one thread could one on the system at a time the other end is muk muk has hundreds and hundreds of micro locks and extraordinarily complicated locking hierarchies so that you can make sure that you get locks in the right order and it's got lots and lots and lots of tiny little locks which is great but they're very heavy it also is extraordinarily complicated locking hierarchies are nasty and they have to be done in one direction which means for i/o systems completion routines are painful so we needed to come up with something different what we came up with is the workload we schedule we have one work loop one gate as it were per major interrupt delivery as part of the system so a pc i scuzzy card for instance has a work loop a USB controller has a work loop a firewire controller has a work loop so on a typical running system we have maybe 13 work loops this is a compromise between the hundreds of micro locks that mark uses and the to uber locks that BST users it turns out it's very very powerful because this allows us to deliver completion routines so all of our drivers stack on top of this one lock so by far the majority of our key drivers as I say don't create their own work loop they use their providers work loop now if you've used iokit burning lengths of time you would have seen the client provider model and the client provider stacking and you will see that this statement is recursive if I call my provider and the provider also doesn't implement get work loop it calls its providers and eventually you get down to the bottom of the system and you say hey here's the work loop use this so high-level drivers always synchronize against the bottom of the system as I mentioned earlier only PCI devices and motherboard device drivers 10 to create work loops in most cases your hardware will not need a work loop and is probably better if you don't create one in fact if you do create a work loop that builds on top of another work loop you can be in for a whole world of hurt talk to I'm sure we'll have a raid developer around here if you want to see somebody who really experiences pain discuss device teardown with a raid developer so you can use the systems there is because the statement is recursive there has to be a way of terminating the recursion there is a system work loop that you can grab hold of just by walking down the stack eventually you hit the roots of the provider tree and bingo there's a work loop it's not a bad work loop to use and we really do encourage you to use it because we'd like to limit the number of threads in the system this is a good thing for system performance however if you're using it's a shared resource so don't be too greedy with it if you need if you expect a lot of interrupts to be used or you have very tight timing requirements it's probably better not to use the system work loop but to create your own so an event source an event source has an action routine which I'm now going to define but essentially it's an action routine as synchronous with respect to the work loop all event sources have an action routine and an owner and usually registered on a workload in fact an event source is really only meaningful when it's registered on a work loop but of course people can temporarily register it and then remove it and register and remove it because there's a fairly lightweight operation registering on an event source an action routine it's just a call out function when you create an event source you're saying to the system I expect this event to occur at some time in the future and when it does call this function and that's what an action routine is all action routines in the system are synchronous with respect to all registered event sources on a particular work loop I mean if you're familiar with Java you may have seen Java's synchronous routine concept where you can have a number of routines in a class and you say this is a synchronous routine or these routines are synchronous with respect to each other only run one of them that's how I think of eventsource actions all of the event sources up and down the entire stack are synchronous with each other now that sounds as though it's a recipe for contention but it hasn't proved to be so far but there are some tricks there that you need to be aware of in general don't go to sleep while you're in an action routine very bad things happen again we recently found a driver which was going to sleep in an action routine for eight milliseconds and that introduced eight milliseconds worth of latency we do have ways of pointing fingers in the system so you won't get away with it for any length of time okay and when you register an event source with the work loop and you generally just do is service : get work loop and that's the mechanism that gives you the entry into the recursive statement or saying that's how you find the workload one of the things actually will cover that later okay so the first event source the most hardware pci hardware developers I was about to say real hardware developers which is the side of my background unfortunately think of is okay how do we get interrupts because it's what it's one of the fundamental things that vary from OS to OS our filter interrupt event source is the mechanism we recommend for firewire app for PCI Hardware the event source is used to deliver hubber interrupts to a driver it takes the interrupts causes the work loop to schedule this is the this is the only thing that causes the workload to schedule in fact so the primary at primary interrupt time it's very quick it just comes along and increments a number and it says hey work loop you've got some work to do kick and then it goes back to sleep again which automatically gets back into the dispatcher that I mentioned earlier the dispatcher says hey look I'm looking for the highest priority thread in the system and it's a work loop the work loop start scheduling so the latencies are very very short and the filters generally don't have to do any work at all but we do recommend that you must always implement a filter because you don't know if your hardware is going to be in a shared chassis or not and when you're sharing interrupts event sources it's a very good idea for you if your hardware supports it to say hey this wasn't me just return false from the filter now the action routine is synchronous with respect to the mic loop you're going to see this statement a lot but the filter is totally asynchronous it's a primary interrupt you have to do special things to stop it from coming out which is why I would recommend single producers single consumer queueing or something of that nature with the filter routine you need to synchronize with the filter routine you've you've got to be very careful okay so now the other major event source is the timer event source there's lots of reasons to use the timer pole mode drivers which we don't recommend but people are doing it so that's one of the reasons for using it but the most common one is Hardware timeouts oh dear nothing is responded in 30 seconds I have to do something I owe kit timers the timer event source is built on top of the current flash thread cool api's they're very wonderful api so highly I just love them they're very very lightweight and they're a great solution there is a problem though if you remember back to my earlier diagram thread cool threads of very high priority they're higher priority than work loops which means if your timeout and your interrupt occur at exactly the same time the timeout will schedule first so best thing check to see if your hardware is done in the timeout code and if it is fine you've beaten the interrupt before it got delivered if not a timeout is occurred okay here I have to make an embarrassing admission this is my bug it's been my bug for a long time now and I will fix it soon there is no synchronous way of canceling a timeout really it's just painful it's embarrassing there's not I'm turning red up here the safest way to delete a timer is to let the timer expire and then on another thread delete it don't rearm the tie sorry I have to give you the warning because it is the big caveat with these things but it's a really a problem and I'm hoping to fix it but I can't go back with some time and fix it and Jaguar and cheetah serve I'm afraid if your drivers have to run back in time in Puma and Jaguar systems then you are going to have to let the timer expire and guess what the timer's action routine is synchronous with respect to the i/o work Lou same as usual ok the command gate command gates rather interesting a lot of people think it's a lock it isn't really it's just a sort of container a pointer to the lock that is in the work loop remember I said the i/o work loop should be called the work gate or the command gate gives you access to that work loop so for all command gates on a particular work loop only one there is still only one gate command gates allowing you to run code synchronously with respect to the workload but without a thread switch it just takes the gate allows you to run some code and then you will drop the gate fairly quickly now I admit that the run action run command API is clunky especially if you use to writing locks and just saying hey take the lock drop the lock take the lock drop the lock you know it turns out run action has really come to our rescue several times first of all debugging recursive locks where you mismatch the lock unlock pair is really painful so with run action you can't get it wrong because it's a subroutine it just says take the lock call the subroutine return the lock on the exit path there is no avoiding it so you can't get it wrong the other thing that it gives you is that it gives you when when you use show all stacks and it's a really wonderful command for tracking down dead locks and other problems that are running in the system show all stacks will show up run actions they will be there on the system and you we have caught so many dead locks because of show wall stacks and run action is there on the system whereas if you just take a lock you have to memorize everybody else's dry even once you don't write and say oh look this this routine 15 levels down in the stack it takes a lock and I know that because well I can read minds run action you don't have to read mais there it is it's in the back-trace you know okay this is the really cool part about command gates it's command sleep command wake up another thing that it sort of came a bit late it's when a client thread is calling into your driver it often says hey I want some data and your Hardware hasn't got any data available yet for streaming for whatever reason like the device you're talking to a slow so what you can do is you can block the client thread by calling command sleep and it will block until some event occurs now this is in fact the mechanism I was talking about that the dispatcher uses this is how you block a thread until some event occurs now there's lots of other ways of doing it but this is the one that's built into the way the command gate does its job data acquisition drivers are a typical case for this we don't really have any hardware direct call-outs one of the most common requests we got is well we can't write our application because the interrupt routine don't call out into user lands well no we're not going to call out into use land we can't allow that thread to disappear into some code that we don't trust the command sleep command wake up gives you something that is very very close to that if you have a sufficiently high priority thread blocked in command sleep then when you take your interrupts to routine and your hardware turns up and says I have some data available the scheduled using command wake up to wake up the thread it's just so fast it's amazing so you can use command sleep to emulate interrupt call-outs out to user Lance have the user provide a thread it's your application you provide the thread block it in your kernel extension using commands sleep very lightweight wake it up using command wake up so that's it for our work loops what we're about to do is how we use this stacking model of

• synchronously tear down device drivers

so oops sorry off-by-one sorry I was good okay so I remember I was mentioning the locking PST does it's working with currently does it's locking with funnels mostly it doesn't affect my acute developers however kernel extension developers generally must be aware of the funnels there are two funnels in the system we do share if we go dual CPU you can issue an i/o on the network funnel or one processor and on the system funnel on the other processor and it's al compromise on the traditional BSD overlock funnels are good but they're really not locks and this is not the right floor on to discuss funnels writing funnel code that can switch between the system funnel and the networking funnel is difficult counseling impossible because NFS works but it's bloody close to impossible funnels can cause lot delays on work loops though so you do have to be aware of it if you've got a piece of hardware that's delivering into BS dB at the TT of the serial ports the disk drive system or the networking system you must be aware that those completion routines will probably try to take a funnel those funnels are going to cause some sort of latencies because there's only two of them okay now we can do a synchronous device teardown so although my device is gone this can cause nightmare tearing down a stack is just so hard and this animation I'm hoping will demonstrate what's going on as you can see here I'm just trying to emulate the stacking that we have in our system so far on the left is where your bus is let's Korell a USB bus and on the right you have the client thread running so the first step is we got to tear down the buses detected the device is gone and this is how we implement it and first didn't work real well we disappeared the device but at the same time we're on an MP system a client thread has just come down and it's issued an IO request it's a bit of a problem because they're going to meet eventually and when they do you get a panic and very bad things happen when that happens no blue screen of death whatever panics are really hard to debug and this particular one is nasty because everything looks perfectly alright but your hardware is crashed and it's not really obvious so how do we deal with this well we do it synchronously I guess that's obvious our solution is to use the work loop stacking this is why drivers really can't opt out of the work loop system not if they want to do dynamic unloading and most of our developers like the idea that they can unload their drivers so it means you have to be at least partially aware of were clips to do unloading what we do is when we get an unload we will tell the nub that has disappeared to terminate and the terminate does a fewer things like it goes recursively up the stack making marking everybody is inactive it does that through request terminate but basically it calls a function called do terminate and do terminate is a recursive function as you can see as I've implemented here in pseudocode it essentially just does a headfirst recursion with will terminate calls and tail recursion on did terminate you can rely on will terminate messages turning up in your driver before any of the clients know to forget will terminate and you can rely on did terminate after all of your clients have got their notifications so your responsibility and will terminate is to if you it sort of depends on where you are if you're an intermediate driver Eames you have a series of commands that you know are outstanding and they're in your own queues you haven't handed off to the next driver down then it's your responsibility to return those io requests with errors immediately if you have client threads blocked in your driver on command sleep command wake now the command sleeps you should return those immediately with an arrow as well wake them up and notify them that they're going to wake up with an offline error the error we use is K IO return offline and by the way if you're higher in the stacks and you start seeing offline errors coming by you know what's happening now somebody's got to will terminate and you can expect it will terminate fairly soon by the time you get to the top of the drivers FAQ it should be expected that all outstanding iOS and block threads ideally have been returned so that makes the top of stack drivers job much easier notice we haven't torn anything down yet all of our pointers are valid one other thing with will terminate is you should be returning errors if possible given the API immediately if any other IO commands come down while you're doing this you should be returning errors after you see will terminate okay if the driver is on the top of the stack you're expected to implement did terminate no top of the stack varies you are top of the stack because there is nobody on top of you which means when you're tearing down eventually you're going to be top of the stack again now indeed terminate you must stop all future calls down to your provider you must wait asynchronously and that's a bit subtle for all provider calls to return so if you have threads that have gone through you then you should be aware of those threads and you should not call closed on your provider until all client pools have gone through now unfortunately you have to do that asynchronously you have to return from the did terminate so your primary responsibility though is to close your provider as soon as you reasonably can as soon as you know that you can synchronously guarantee that no client threads will get through you and no client threads are already gone through have already gone through you then you can call closed on your provider but not before if you cannot make that determination that you would have to wait for some threads to return then you must return from terminate immediately anyhow it's a bit subtle and what then happens is when the client thread does return take the command gate and then call close it's really tricky to implement well in general you don't have to worry about it you can make certain assumptions if you're an intermediate driver the only drivers that really have to be aware of this is top of stack drivers and we write those usually Apple writes those we've got the user client for USB and firewire we have the media BSD client and I would like to say the serial bsd client which we own but it's broken I owned that one as well and that's about it really in conclusion I guess Freddy comes down to please lower your priority we don't want an arms race and the system will work a whole lot better if you use the lower priority the other thing that was interesting is work loops work loops are way cool they integrate well with the system you can't get deadlocks if you're on a workload unless you're a rate driver and if you are a raid driver heaven help you that's what Darwin is for I guess and finally synchronous teardown please implement it properly you'll terminate did terminate and by the way synchronous teardown applies even to PCI devices I mean you could be pc card but also if whenever you do a text unload you're essentially going through a device teardown so further things that might be interesting we have an open source presentation that we'll be discussing how X and new works among other things that's coming up tomorrow we have kernel programming interfaces on Wednesday and we have writing threaded applications on Mac OS 10 writing threaded applications isn't a direct hit on what we're trying to do is very very high-level but it should be interesting and also there's a series of Hardware talks coming up tomorrow the Bluetooth USB firewire and some feedback forums who to contact is Craig Keithley and I think I'll hand over to him [Applause]