WWDC2001 Session 204

Transcript

Kind: captions Language: en over the last couple of years with working with the developers and working with Apple's engineering team we've learned a lot about what mass storage device drivers need to do and what we were doing wrong in Mac OS 9 so hopefully you'll find with what we're doing intend to be a much more powerful solution for you to create device drivers without necessarily needing to write to the entire driver you can leverage off of major portions of ours and to sort of describe that overall architecture I'd like to bring up Craig Marciniak Craig good evening I'm Craig Marciniak senior engineer at Apple I've been working on mass storage in 10 for about a year now and tonight we're gonna delve in and get into all the details of that architecture I've partitioned this talk into really two pieces third-party hardware vendors that want to bring their hardware to the platform and utility writers and people that need to talk to hardware from application space the second group that want to talk to applications from user space will find the first half semi interesting because it exposes how things work in the kernel which will help you to understand and debug talking to those drivers from user space so we're going to really expose and get into the details of the architecture I'll do this mostly by showing how the driver stack is a stanch eiated how to subclass support for given devices how to talk to those devices from user space and how most importantly how to get more information and if there's time at the end we'll do some QA how many people here have a comfortable understanding of the basic concepts and i/o kit and that's about what I expected so drivers are kernel extensions we referred to them as qex they're written in a restricted subset of C++ that is most notably missing are TTI and templates multiple inheritance and exceptions thank you I hope it makes a big distinction between family versus driver's families are an encapsulation of all the commonality and drivers in the vernacular of i/o kit are really instances of device specific things that you would subclass over and I'm going to keep refining that it may seem a little repetitive but it's really an important concept and I've been doing drivers in nine space for a decade and when I hear the word driver I tend to think of this monolithic giant driver that encompasses a lot more than what drivers mean an i/o kit we refer to them of leave classes of the family another concept that's really important is the registry and registry is a dynamic database it's a tree that contains a whole bunch of properties that get filled up over time and this these properties are used for matching so that devices and objects can be Estancia dand built up there's some other sessions this week that go into detail and depth and all the they'll be enumerated at the end of this presentation there's some appropriate URLs sprinkled through this and summarized at the end there's a new book that just got released called inside Mac OS 10 the kernel environment that has a very good overview of il kit concepts and finally there's always Darwin I've spent a lot of time reading the header especially of i/o service that eh and looking at the implementations and I all service that CPP which really can help a lot so I'm going to put the whole thing into context here we have devices that we find and the first layer is the physical interconnect layer this layer encompasses a whole bunch of a collection of objects of bus specific things these are disguisey drivers the sketchy bus drivers the controller drivers excuse me that firewire controller drivers the USB controller drivers and all the objects that are to support that layer the transport driver layer is where requests get packaged up and turned into commands to get on to the appropriate bus we're going to go into more detail in that in a minute the device services layer is a generic layer that it contains partition errs and a generic driver and then finally we have they can't the the yellow line there represents the user kernel boundary and above that we have clients ultimately your applications so we're going to zoom in and get in the details of the transport driver layer historically we were looking at nine and we had a sense of deja vu because we kept pretty much writing the same driver over and over again and we had no reusability and invariably there were subtleties and implementation from one interpretation of how to do this and that and another and over time all kinds of metadata bled into those drivers partition information special command special control and status calls and we're gonna get it be able to finally get away from that what we did intend is we had a chance to really reuse that commonality we wanted to reuse that commonality ioq it gave us an object framework to actually achieve that so we were discussing how are we going to do this and it turns out that there was a formal document that served as a really good guideline to do this it's called Sam Sam is a scuzzy archetype architectural model and I want to stop and make a distinction here when I say Sam and I mean scuzzy in the sense of because the architectural model I'm not referring to scuzzy Hardware I'm not talking about scuzzy hard drive and not talking about its Guzzi scanners I'm not talking about anything what I mean that I'll say scuzzy parallel so Sam was this wonderful document that really was a blueprint for us to do exactly what we wanted to achieve by implementing and specifying a partitioning of the command sets and the internet protocols internet interconnect protocols you can go to the URL on the bottom to get more information in the formal Sam document when we started on Sam a year ago I believe we were using draft 5 and it has been it is not ratified at this point but it is definitely farther than when we started another thing is half the challenge of understanding this is getting the subtleties of all the language we used a lot of the wording and terminology out of the Sam document in our class and implementation so that's where all this comes from so the transport driver layer is split into two natural layers the scuzzy protocol layer which is bus specific and the device disguisey application layer which is a device specific to what I mean by that is we ship bus specific drivers for USB the tapi firewire bus specific stuff at the protocol layer and we expand this other layer into the scuzzy application layer we have this peripheral device type nub the peripheral device type nubs job in life is to really send it inquiry command down to the device and discover what kind of peripheral device type it is is it you know it's going to be one of the peripheral device types in kernel we have above that a logical unit driver we have a peripheral device type 0 0 which is block storage peripheral device type 5 which is multimedia 7 which is vendhya the video optical and E which is reduced block commands so what happens is the peripheral device type nub populates the registry with the peripheral device type type so that a logical unit driver can be accentuated appropriately so there's one other thing above that and the service device layer and this is just some linkage objects that link the logical unit driver to the appropriate generic system driver above it in a typical since what will happen is let's say we have plugged a device into the USB bus that we woulda Stan she ate a IO USB mass-storage class protocol driver which would in turn a Stan she ate the peripheral device knob which would in turn query and populate the device the registry with the appropriate device type and if it was a block storage device in the case of a hard drive a fir filled device type 0 0 logical unit driver would be accentuated so up into this point I've only been talking about the functional stacking and layering of how all these pieces in it relate now you have to work with me here a little bit this slide may be kind of confusing at first a little bit of convention context here these puzzle pieces aren't components they're not extensions they're objects this is just an object hierarchy you usually see them as ovals I don't know if you can tell but the darker purple is means that it's a concrete class and excuse me an abstract class and the lighter purple are concrete classes now at the scuzzy protocol layer the base class is iOS Guzzi protocol services we subclass of family for a given bus off that so if we wanted to support scuzzy we would do an i/o scuzzy parallel transport subclass of the i/o scuzzy protocol services the next layer in the stack is the scuzzy application layer object hierarchy here are basically the pieces that represent the logical unit drivers so in the case of a block storage driver the peripheral device type 0 0 is sub class top of the scuzzy block commands device and the base class in this layer is the i/o scuzzy primary commands device so I'm gonna bring this all kind of together right now here's an example of a firewire drive at the physical interconnect layer the appropriate objects are as Dan she didn't built up what I'm showing here is this driver stack to the left and the object hierarchy works this way so what will happen is once the appropriate objects have been built up a peripheral scuzzy Perez II protocol driver will be accentuated the base family will be i/o firewire serial bus protocol transport which in turn instantiates that peripheral device nub which in turn would put a peripheral device type of zero into the registry which would a Stan she ate a peripheral scuzzy device type 0 0 object and we see how the object hierarchy moves out to the right which internist and she eighths all the way up until a piece of media now this is the magic sly here to put this all together I keep referencing to this sub classing and exactly well really what does that mean how does it fit into context when we subclass something in this case we're going to subclass the peripheral device type 0 0 driver with my device subclass all we're doing is inserting a new leaf class and shifting the object hierarchy to the right so if I want to subclass and add some support to the scuzzy protocol layer I would simply subclass the i/o firewire serial bus protocol transport that would when we accentuated the stack that is the name that would actually show up in the registry and we would shift that hierarchy one to the right so matching is really important this in the matching heuristics are a little bit different in implementation even on what layer you're on the highest probe score wins and the at the in the previous slide above the peripheral device type knob and where the logical unit driver gets a Stan she ated we have this kind of matching the peripheral device type chooses what logical unit driver if any is going to get is San Shi ated then if we have a vendor match the probe score goes up if we have a product ID the score goes even higher and then finally if we have a revision ID we've actually honed in to a specific instance of a device that we want to add certain support for sometimes you want the refer to the previous matching that's all passive matching just using the properties that are in the registry tree sometimes you need to have active matching in which we override a probe method overriding probe sometimes you might have a property that has some bit squirrel the way up and you have to mask them out and pull them out to determine what kind of device you are you'll know your requirements for more than we will be careful if you use probe not to reference or count on the state of member variables because other potential candidate drivers could be loaded and could have changed the state of the device behind you and to see an example of how this works you can look in amplifier where masters drivers there's a good example of overriding probe that's quite extensive now for me it the best thing ice come to WWC I see the slides and it all seems pretty and makes some sense but it's kind of abstract and I learned the most by standing over a competent engineer shoulders and watching him do it and that's what we're going to do right now we're actually going to go in I'm going to bring up Chris sarcone and we're going to subclass a logical unit driver thanks Craig if we could bring up demo one please all right so I'm gonna go ahead and fire up project builder we're gonna go ahead and create a subclass of a peripheral device type driver so we're gonna create a new project we're gonna scroll down here and choose i/o kit driver and we're gonna call this my logical unit driver okay project builder goes ahead and creates us a new project and inside here it'll place a you know a dummy header file and a C++ file what we want to do is we want to go over to the targets pane right now and take a look at the bundle settings Craig just went into some details about how we do matching and how a driver gets instantiated we do this by creating an i/o kit personality inside of our text so what I'm gonna go ahead ahead and do here is create a new child and we're gonna call this child my logical unit driver if I could type and the i/o kit personality is a dictionary inside of it we will have more properties which we will add so I'm gonna go ahead and change that to a dictionary and I'm gonna create a whole bunch of properties here from experience I know I need to create six properties here depending on what type of driver you write you will need to add more properties or have less properties it all depends on what kind of driver you're writing the first item that should be inside of your personality is the CF bundle identifier and since we are going to load a code fragment from this bundle that we've created we're gonna paste the same C a bundle identifier in you can of course have a personality which loads code from another module if you do do that then you need to make sure that you have the correct CF bundle identifier for that module the next property that we need to add is iö class we need to instantiate a particular class when we load our kernel extension so we're gonna put in my logical unit driver because that'll be our class name the next thing that I oak it needs for matching is a provider class this tells i/o kit to consider your driver for every instance of this class the class that we want to attach to is the iOS Guzzi peripheral device node now this is where I oak its generic matching services stop and this is where the iOS cozy architecture model family family specific matching begins Craig mentioned that we score on peripheral device type first so we need to specify a peripheral device type which is of course a number now if you want to subclass a block bought command device you would want to have a 0 here I don't have one of those devices here on this machine but I do have a cd-rom drive and that shows up as a peripheral device type 5 so we're going to go ahead and subclass the cd-rom driver for this machine the next two properties that we need to have are a vendor identification and a product identification now to find these values we need to look in the i/o kit registry I'm gonna go ahead and fire up terminal and type I or reg - c io Scaasi peripheral device num this goes ahead and prints out the contents of the i/o scuzzy peripheral device nub entry what I am interested in here are the vendor identification right here which says this is a pioneer drive so I'm going to go ahead and copy paste that directly into my vendor identification here and then it also tells me the product identification and I'm gonna paste that into my driver here now if you wanted a driver that matched on all Pioneer drives you could of course remove this product identification property and you know take your chances that your driver will match of course if somebody had a pioneer dvd-rw DVR 103 driver that would get considered for matching before yours would so for vendor and product identification you know it's very important that you have those keys in there to get your driver considered for matching now if you only want to match on a specific firmware revision for the device you could also add that in and that is the product revision level which you would see right here we're not going to add that key today so that is it with our personality here a couple more properties that we need to add to our text are the OS bundle libraries this key allows you to specify families that you require to have your module loaded into the kernel I know from experience that I need to have two of such model modules one is the comm Apple IO kit iOS cozy architecture model and the other one is Iowa's because even multimedia commands device and in the string category here you just put the revision level of that kernel extension that you require both of these can be ones or a zero for now and the last piece that we need is to make sure that we get loaded at boot time so to add that key we have OS bundle required and that is local - root if you need more specifics on what each of these properties does and which ones are required I ask you to consult the inside Mac os10 kernel extension or it's an inside kernel or a kernel extension one of those books has the correct information and of course you can reference the website and now we need to actually write some code for this driver we can go ahead and add code to subclass a particular function we can go ahead and add code to write an entire logical unit driver if we want what I'm going to do is actually open up some prefab files that I did already and I'm going to copy paste that into here as you can see we're working with a peripheral device type zero five driver which is a cd-rom driver and I'm most familiar and most comfortable with the read table of contents command so I have decided to subclass the read talk command and inside the C++ file all I've done is add a status log the status log here is and will print out an IO log message to the system lock so when this driver gets loaded any time we do a read table of contents command it will get spit out to the console that that message right here and we can go ahead here and build this and it succeeded in building our kernel extensions so let's go ahead and put it on this machine and load it there we go and I'll go ahead and copy this over so I'm gonna copy the directory into the extensions folder we're gonna go ahead and reboot this machine and you will see that it is really that easy to subclass and add vendor specific functionality to your device or for your device rather there's already a driver there was already a driver loaded for the cd-rom driver here of course if we had a firewire drive or a USB Drive which we could dynamically load a driver for we could have loaded the driver into the kernel and as long as it was the curtain the kernel extension the i/o kit personality matched for that drive we could definitely load for that drive so as soon as this starts up here I'll go ahead and pop up terminal again you'll see that in the ir registry our driver will be loaded for this computer so let me go into a couple reasons as to why you might want to subclass you might want to subclass a logical unit driver to add some sort of functionality to your device like say you have password protection or say you're developing a product that has SDM eye support in it and you want to send those commands to your drive that would be a very good reason to just subclass the the generic support that we bring to the table with the generic IO functionality and then just add your vendor specific functionality to it and you'll see here my logical unit driver got loaded right above the peripheral device numb so with that I'll turn this back over to Crick you can finish this presentation [Applause] thanks a lot Chris okay I hope that made sense I so close to it it's sometimes hard to know if we're actually telling you what you need or making too many assumptions so up and at this point we've talked really about estancia ting the stack and all the pieces that get put together for that we've got into the hierarchy the object hierarchy of those functional pieces there are other objects in the Sam implementation that we need to talk about the scuzzy task is a very important object a scuzzy task is an encapsulation of a CDB data buffers error information just about everything you need to take and actually process the CDB and follow through it's full lifespan a really good place to understand what we mean by this is basically what we've done is we've made an object out of the scuzzy command model in the Sam document which is chapter 5 in fact the names come right out of the guidelines in that chapter so another thing we provide is this command builders these are utility classes that build up command set specific commands for you obviously we have block commands for peripheral device that 0 0 and the multimedia MMC commands and reduce block commands in the future if a new set of commands come out you would want to probably build a utility class for that so how are scuzzy tasks processed we've talked about just Stan she ating and building these stacks up so what happens and I all request is going to come in from the system for something and it's going to get into the transport driver layer and it's going to get packaged up into a scuzzy task that scuzzy task is that I'm going to get handed off to the scuzzy protocol driver layer and that's going to get packaged up into the appropriate for the target device so in the case of firewire the scuzzy task will get turned into an orb that or will be thrown out the bus process it all happens asynchronously and the completion routine will fire back up the stack so device compliance we really don't want you to have to reinvent the wheel here by sub classing and adding your device specific stuff you get power management you get Driver life cycle stuff hot plugging support all the termination all the gnarly undocumented stuff that you really don't want to do what we mean by device compliance is that your device will process commands as they're documented in the command sets and the official scuzzy documentation I always override the whole stack all you need to do is out score the probe at the protocol layer and do anything you want we will not prohibit that so what's supported in 10 as it ships today is we have a top view support firewire support and USB support ironically we don't have scuzzy parallel support at this time it's a work in progress and we're investigating it and working on it and ata just doesn't apply because it doesn't use us because you can fly a command set in theory what we're discussing that it would be possible to actually make a shim layer and wrap up ata requests and use our architecture but it added an extra layer that we didn't want to take the performance hit on so we instead have a monolithic ata driver that ships today so I'm going to grab some water accessing devices from applications there's a lot of compelling reasons you want to do this by resources we mean if you don't have to wire memory down in the kernel we don't want to that's really precious stuff there's a much better tool environment up in user space you have a source-level debugger you have much better tools you don't have to use gdb from the command line and system stability is another thing if you blow up in user space there's a less of a statistic probability will take them a whole machine down if you blow up in the kernel it's much higher risk examples would be utilities that need to set some kind of intrinsic device feature with a vendor specific cdb in any out of kernel driver what I mean by that is we ship a block storage a multimedia magnificat magnetic optical and a RBC logical unit driver that are in kernel and if you have a tape drive which is a peripheral device type one for sequential access it really doesn't need to be in the kernel as a general rule of thumb if your device doesn't require booting and it doesn't require a file system its driver does not need to be in the kernel user clients are nothing more than a mechanism to negotiate the user kernel boundary there may be other places in this big stack that you want to connect you might want to have a user client connect to the physical interconnect layer in the case of firewire if you look in their SDK they have a user client that allows you to click connect to the SBP two layer you might want to have a utility that does firmware that connects at a lower device or a lower point in the stack I'm also this is a good place to point this out on our website in the developer documentation there's a document called accessing Hardware from user space that just recently was updated and went from like 50 to 100 pages and that's something you want to really get your hands on especially because there's a summary of the i/o kit that's a lot more extensive than what I've got into here I found it to be a good reference for getting my head around some of the basic concepts of iokit the scuzzy tasks user client is what we've implemented at our layer for user client access we have two access points we have the peripheral device type nub access point back if you think back to the diagram I showed this is the point where you would attach a non in kernel logical unit driver and we also have the generic service layers for multimedia and this is for authoring and it all aided it I'll get into more details in the next slides this the user client uses a commlite CF plug-in architecture again the document I just referenced the accessing hardware from user space goes in to some of the rules in that and it's there are some tools that will provide an SDK to show you how to do all that we have three interfaces into our scuzzy client the first one is the amp MMC device interface this is only for the authoring needs this basically exists so you can get device and media information for the purposes of authoring we have a scuzzy device interface which lets you create release and do all the maintenance things that you need create the callback handlers and connect up the CF run loops etc and finally there's the scuzzy tasks interface once you've actually connected up this is where you send and you this is the interface you use this in raw scuzzy tasks that encapsulate the CTB's to drive your device so we have two access models we have an exclusive access model and a non exclusive access model once your havest anshi ated the user client that user client is the logical unit driver so I'll have a diagram in a few slides that will really bring this home but there are no restrictions the client has absolute control we don't filter disallow any kind of commands and this last point if you happen to be using the generic services for multimedia they are authoring api's and you need to transition to exclusive access what has to happen is you have to unmount any mounted media anymore any amount of partitions and obtain a reservation for that media through disk arbitration and in our SDK that will have an example code to show how to do this the non exclusive access model is for the generic services multimedia access point only this is like I said before only to get device and media specific information for the purposes of offering it is API driven you cannot send raw CD B's and if an existing client a user client has got exclusive access it will not allow non exclusive access so here's an example if we have a tape backup drive that gets discovered by the physical interconnect layer in this example it's firewire again it would be a stanch eiated and built up and the there'd be a scuzzy protocol driver accentuated and it would build up in the i/o peripheral device now what issue an inquiry find out that it's a peripheral device type 1 populate the registry with it and just stop then what will happen is you will launch an application up in user space and the scuzzy test user client that we provide straddles the user kernel boundary and once the application is launched it will look for the appropriate peripheral device information through a command called I always get this wrong io service get at that point the client application is the logical unit driver and it can drive that device the other interesting one is in the case of authoring where we have a DVD device that's been a stanch eat it up and we have an internal driver a peripheral device driver type v that's going to drive this device and we might have been doing io2 it all day long and eventually an application will get launched and your user client will be this is the second type I was talking about you can send API you can use the API is to get device information and media information in a non exclusive fashion at some point you will have unmounted the media and you will seized control and become exclusive access you would again call the i/o create CF plugin for service at which point the client application has become the logical unit driver and we will quiesce the internal peripheral device type driver v so now the user client logical unit driver has absolute control over the device and can do whatever it needs to do and in summary it should be relatively straightforward and easy for you to instead of having the ship gigantic monolithic drivers for all of your stuff just bundle up and ship out strategic small checks and make everybody happy resources so I've covered most of this and the slides already that there's nothing here that I haven't and later this week the I up update was yesterday the PCI drivers are is tomorrow morning firewire and depth and both USB in depth there's a good place to go if you need to get more information on bus specific stuff if you have a bridge that you your company is providing that has unique requirements it might be interesting or helpful for you to go there and I'll be there if anyone has any protocol transport specific information they'd like to ask me about and finally we have a list a mailing list that you can get on this is a place where we'll be making announcements of where you can get SDKs and help on everything we have and if you have any questions at all please feel free to get ahold of Craig chiefly you