WWDC2001 Session 204

Transcript

Kind: captions
Language: en
over the last couple of years with
working with the developers and working
with Apple's engineering team we've
learned a lot about what mass storage
device drivers need to do and what we
were doing wrong in Mac OS 9 so
hopefully you'll find with what we're
doing intend to be a much more powerful
solution for you to create device
drivers without necessarily needing to
write to the entire driver you can
leverage off of major portions of ours
and to sort of describe that overall
architecture I'd like to bring up Craig
Marciniak Craig good evening I'm Craig
Marciniak senior engineer at Apple I've
been working on mass storage in 10 for
about a year now and tonight we're gonna
delve in and get into all the details of
that architecture
I've partitioned this talk into really
two pieces third-party hardware vendors
that want to bring their hardware to the
platform and utility writers and people
that need to talk to hardware from
application space the second group that
want to talk to applications from user
space will find the first half semi
interesting because it exposes how
things work in the kernel which will
help you to understand and debug talking
to those drivers from user space so
we're going to really expose and get
into the details of the architecture
I'll do this mostly by showing how the
driver stack is a stanch eiated how to
subclass support for given devices how
to talk to those devices from user space
and how most importantly how to get more
information and if there's time at the
end we'll do some QA how many people
here have a comfortable understanding of
the basic concepts and i/o kit and
that's about what I expected
so drivers are kernel extensions we
referred to them as qex
they're written in a restricted subset
of C++ that is most notably missing are
TTI and
templates multiple inheritance and
exceptions thank you I hope it makes a
big distinction between family versus
driver's families are an encapsulation
of all the commonality and drivers in
the vernacular of i/o kit are really
instances of device specific things that
you would subclass over and I'm going to
keep refining that it may seem a little
repetitive but it's really an important
concept and I've been doing drivers in
nine space for a decade and when I hear
the word driver I tend to think of this
monolithic giant driver that encompasses
a lot more than what drivers mean an i/o
kit we refer to them of leave classes of
the family another concept that's really
important is the registry and registry
is a dynamic database it's a tree that
contains a whole bunch of properties
that get filled up over time and this
these properties are used for matching
so that devices and objects can be
Estancia dand built up there's some
other sessions this week that go into
detail and depth and all the they'll be
enumerated at the end of this
presentation there's some appropriate
URLs sprinkled through this and
summarized at the end there's a new book
that just got released called
inside Mac OS 10 the kernel environment
that has a very good overview of il kit
concepts and finally there's always
Darwin I've spent a lot of time reading
the header especially of i/o service
that eh and looking at the
implementations and I all service that
CPP which really can help a lot so I'm
going to put the whole thing into
context here we have devices that we
find and the first layer is the physical
interconnect layer this layer
encompasses a whole bunch of a
collection of objects of bus specific
things these are disguisey drivers the
sketchy bus drivers the controller
drivers excuse me that firewire
controller drivers the USB controller
drivers and all the objects that are
to support that layer the transport
driver layer is where requests get
packaged up and turned into commands to
get on to the appropriate bus we're
going to go into more detail in that in
a minute the device services layer is a
generic layer that it contains partition
errs and a generic driver and then
finally we have they can't the the
yellow line there represents the user
kernel boundary and above that we have
clients ultimately your applications so
we're going to zoom in and get in the
details of the transport driver layer
historically we were looking at nine and
we had a sense of deja vu because we
kept pretty much writing the same driver
over and over again and we had no
reusability and invariably there were
subtleties and implementation from one
interpretation of how to do this and
that and another and over time all kinds
of metadata bled into those drivers
partition information special command
special control and status calls and
we're gonna get it be able to finally
get away from that what we did intend is
we had a chance to really reuse that
commonality we wanted to reuse that
commonality ioq it gave us an object
framework to actually achieve that so we
were discussing how are we going to do
this and it turns out that there was a
formal document that served as a really
good guideline to do this it's called
Sam Sam is a scuzzy archetype
architectural model and I want to stop
and make a distinction here when I say
Sam and I mean scuzzy in the sense of
because the architectural model I'm not
referring to scuzzy Hardware I'm not
talking about scuzzy hard drive and not
talking about its Guzzi scanners I'm not
talking about anything what I mean that
I'll say scuzzy parallel so Sam was this
wonderful document that really was a
blueprint for us to do exactly what we
wanted to achieve by implementing and
specifying a partitioning of the command
sets and the internet protocols
internet interconnect protocols you can
go to the URL on the bottom to get more
information in the formal Sam document
when we started on Sam a year ago I
believe we were using draft 5 and it has
been it is not ratified at this point
but it is definitely farther than when
we started
another thing is half the challenge of
understanding this is getting the
subtleties of all the language we used a
lot of the wording and terminology out
of the Sam document in our class and
implementation so that's where all this
comes from so the transport driver layer
is split into two natural layers the
scuzzy protocol layer which is bus
specific and the device disguisey
application layer which is a device
specific to what I mean by that is we
ship bus specific drivers for USB the
tapi firewire bus specific stuff at the
protocol layer and we expand this other
layer into the scuzzy application layer
we have this peripheral device type nub
the peripheral device type nubs job in
life is to really send it inquiry
command down to the device and discover
what kind of peripheral device type it
is is it you know it's going to be one
of the peripheral device types in kernel
we have above that a logical unit driver
we have a peripheral device type 0 0
which is block storage peripheral device
type 5 which is multimedia 7 which is
vendhya the video optical and E which is
reduced block commands so what happens
is the peripheral device type nub
populates the registry with the
peripheral device type type so that a
logical unit driver can be accentuated
appropriately so there's one other thing
above that and the service device layer
and this is just some linkage objects
that link the logical unit driver to the
appropriate generic system driver above
it in a typical
since what will happen is let's say we
have plugged a device into the USB bus
that we woulda Stan she ate a IO USB
mass-storage class protocol driver which
would in turn a Stan she ate the
peripheral device knob which would in
turn query and populate the device the
registry with the appropriate device
type and if it was a block storage
device in the case of a hard drive a fir
filled device type 0 0 logical unit
driver would be accentuated so up into
this point I've only been talking about
the functional stacking and layering of
how all these pieces in it relate now
you have to work with me here a little
bit this slide may be kind of confusing
at first a little bit of convention
context here these puzzle pieces aren't
components they're not extensions
they're objects this is just an object
hierarchy you usually see them as ovals
I don't know if you can tell but the
darker purple is means that it's a
concrete class and excuse me an abstract
class and the lighter purple are
concrete classes now at the scuzzy
protocol layer the base class is iOS
Guzzi protocol services we subclass of
family for a given bus off that so if we
wanted to support scuzzy we would do an
i/o scuzzy parallel transport subclass
of the i/o scuzzy protocol services the
next layer in the stack is the scuzzy
application layer object hierarchy here
are basically the pieces that represent
the logical unit drivers so in the case
of a block storage driver the peripheral
device type 0 0 is sub class top of the
scuzzy block commands device and the
base class in this layer is the i/o
scuzzy primary commands device
so I'm gonna bring this all kind of
together right now here's an example of
a firewire drive at the physical
interconnect layer the appropriate
objects are as Dan she didn't built up
what I'm showing here is this driver
stack to the left and the object
hierarchy works this way so what will
happen is once the appropriate objects
have been built up a peripheral scuzzy
Perez II protocol driver will be
accentuated the base family will be i/o
firewire serial bus protocol transport
which in turn instantiates that
peripheral device nub which in turn
would put a peripheral device type of
zero into the registry which would a
Stan she ate a peripheral scuzzy device
type 0 0 object and we see how the
object hierarchy moves out to the right
which internist and she eighths all the
way up until a piece of media now this
is the magic sly here to put this all
together I keep referencing to this sub
classing and exactly well really what
does that mean how does it fit into
context when we subclass something in
this case we're going to subclass the
peripheral device type 0 0 driver with
my device subclass all we're doing is
inserting a new leaf class and shifting
the object hierarchy to the right so if
I want to subclass and add some support
to the scuzzy protocol layer I would
simply subclass the i/o firewire serial
bus protocol transport that would when
we accentuated the stack that is the
name that would actually show up in the
registry and we would shift that
hierarchy one to the right so matching
is really important this in the matching
heuristics are a little bit different in
implementation even on what layer you're
on the highest probe score wins
and the at the in the previous slide
above the peripheral device type knob
and where the logical unit driver gets a
Stan she ated we have this kind of
matching
the peripheral device type chooses what
logical unit driver if any is going to
get is San Shi ated then if we have a
vendor match the probe score goes up if
we have a product ID the score goes even
higher and then finally if we have a
revision ID we've actually honed in to a
specific instance of a device that we
want to add certain support for
sometimes you want the refer to the
previous matching that's all passive
matching just using the properties that
are in the registry tree sometimes you
need to have active matching in which we
override a probe method overriding probe
sometimes you might have a property that
has some bit squirrel the way up and you
have to mask them out and pull them out
to determine what kind of device you are
you'll know your requirements for more
than we will be careful if you use probe
not to reference or count on the state
of member variables because other
potential candidate drivers could be
loaded and could have changed the state
of the device behind you and to see an
example of how this works you can look
in amplifier where masters drivers
there's a good example of overriding
probe that's quite extensive now for me
it the best thing ice come to WWC I see
the slides and it all seems pretty and
makes some sense but it's kind of
abstract and I learned the most by
standing over a competent engineer
shoulders and watching him do it and
that's what we're going to do right now
we're actually going to go in I'm going
to bring up Chris sarcone and we're
going to subclass a logical unit driver
thanks Craig if we could bring up demo
one please all right so I'm gonna go
ahead and fire up project builder we're
gonna go ahead and create a subclass of
a peripheral device type driver so we're
gonna create a new project we're gonna
scroll down here and choose i/o kit
driver and we're gonna call this my
logical unit driver okay
project builder goes ahead and creates
us a new project and inside here it'll
place a you know a dummy header file and
a C++ file what we want to do is we want
to go over to the targets pane right now
and take a look at the bundle settings
Craig just went into some details about
how we do matching and how a driver gets
instantiated we do this by creating an
i/o kit personality inside of our text
so what I'm gonna go ahead ahead and do
here is create a new child and we're
gonna call this child my logical unit
driver if I could type and the i/o kit
personality is a dictionary inside of it
we will have more properties which we
will add so I'm gonna go ahead and
change that to a dictionary and I'm
gonna create a whole bunch of properties
here from experience I know I need to
create six properties here depending on
what type of driver you write you will
need to add more properties or have less
properties it all depends on what kind
of driver you're writing the first item
that should be inside of your
personality is the CF bundle identifier
and since we are going to load a code
fragment from this bundle that we've
created we're gonna paste the same C a
bundle identifier in you can of course
have a personality which loads code from
another module if you do do that then
you need to make sure that you have the
correct CF bundle identifier for that
module the next property that we need to
add is
iö class we need to instantiate a
particular class when we load our kernel
extension so we're gonna put in my
logical unit driver because that'll be
our class name the next thing that I oak
it needs for matching is a provider
class this tells i/o kit to consider
your driver for every instance of this
class the class that we want to attach
to is the iOS Guzzi peripheral device
node now this is where I oak its generic
matching services stop and this is where
the iOS cozy architecture model family
family specific matching begins Craig
mentioned that we score on peripheral
device type first so we need to specify
a peripheral device type which is of
course a number now if you want to
subclass a block bought command device
you would want to have a 0 here I don't
have one of those devices here on this
machine but I do have a cd-rom drive and
that shows up as a peripheral device
type 5 so we're going to go ahead and
subclass the cd-rom driver for this
machine the next two properties that we
need to have are a vendor identification
and a product identification
now to find these values we need to look
in the i/o kit registry I'm gonna go
ahead and fire up terminal and type I or
reg - c io Scaasi peripheral device num
this goes ahead and prints out the
contents of the i/o scuzzy peripheral
device nub entry what I am interested in
here are the vendor identification right
here which says this is a pioneer drive
so I'm going to go ahead and copy paste
that directly into my vendor
identification here and then it also
tells me the product identification and
I'm gonna paste that into my driver here
now if you wanted a driver that matched
on all Pioneer drives you could of
course remove this product
identification property and you know
take your chances that your driver will
match of course if somebody had a
pioneer dvd-rw DVR 103 driver that would
get considered for matching before yours
would so for vendor and product
identification you know it's very
important that you have those keys in
there to get your driver considered for
matching now if you only want to match
on a specific firmware revision for the
device you could also add that in and
that is the product revision level which
you would see right here we're not going
to add that key today so that is it with
our personality here a couple more
properties that we need to add to our
text are the OS bundle libraries this
key allows you to specify families that
you require to have your module loaded
into the kernel I know from experience
that I need to have two of such model
modules one is the comm Apple IO kit iOS
cozy architecture model
and the other one is Iowa's because even
multimedia commands device and in the
string category here you just put the
revision level of that kernel extension
that you require both of these can be
ones or a zero for now and the last
piece that we need is to make sure that
we get loaded at boot time so to add
that key we have OS bundle required and
that is local - root if you need more
specifics on what each of these
properties does and which ones are
required I ask you to consult the inside
Mac os10 kernel extension or it's an
inside kernel or a kernel extension one
of those books has the correct
information and of course you can
reference the website and now we need to
actually write some code for this driver
we can go ahead and add code to subclass
a particular function we can go ahead
and add code to write an entire logical
unit driver if we want what I'm going to
do is actually open up some prefab files
that I did already and I'm going to copy
paste that into
here as you can see we're working with a
peripheral device type zero five driver
which is a cd-rom driver and I'm most
familiar and most comfortable with the
read table of contents command so I have
decided to subclass the read talk
command and inside the C++ file all I've
done is add a status log the status log
here is and will print out an IO log
message to the system lock so when this
driver gets loaded any time we do a read
table of contents command it will get
spit out to the console that that
message right here and we can go ahead
here and build this
and it succeeded in building our kernel
extensions so let's go ahead and put it
on this machine and load it
there we go and I'll go ahead and copy
this over so I'm gonna copy the
directory into the extensions folder
we're gonna go ahead and reboot this
machine and you will see that it is
really that easy to subclass and add
vendor specific functionality to your
device or for your device rather
there's already a driver there was
already a driver loaded for the cd-rom
driver here of course if we had a
firewire drive or a USB Drive which we
could dynamically load a driver for we
could have loaded the driver into the
kernel and as long as it was the curtain
the kernel extension the i/o kit
personality matched for that drive we
could definitely load for that drive so
as soon as this starts up here I'll go
ahead and pop up terminal again you'll
see that in the ir registry our driver
will be loaded for this computer so let
me go into a couple reasons as to why
you might want to subclass you might
want to subclass a logical unit driver
to add some sort of functionality to
your device like say you have password
protection or say you're developing a
product that has SDM eye support in it
and you want to send those commands to
your drive that would be a very good
reason to just subclass the the generic
support that we bring to the table with
the generic IO functionality and then
just add your vendor specific
functionality to it and you'll see here
my logical unit driver got loaded right
above the peripheral device numb so with
that I'll turn this back over to Crick
you can finish this presentation
[Applause]
thanks a lot Chris okay I hope that made
sense I so close to it it's sometimes
hard to know if we're actually telling
you what you need or making too many
assumptions so up and at this point
we've talked really about estancia ting
the stack and all the pieces that get
put together for that we've got into the
hierarchy the object hierarchy of those
functional pieces there are other
objects in the Sam implementation that
we need to talk about the scuzzy task is
a very important object a scuzzy task is
an encapsulation of a CDB data buffers
error information just about everything
you need to take and actually process
the CDB and follow through it's full
lifespan a really good place to
understand what we mean by this is
basically what we've done is we've made
an object out of the scuzzy command
model in the Sam document which is
chapter 5 in fact the names come right
out of the guidelines in that chapter so
another thing we provide is this command
builders these are utility classes that
build up command set specific commands
for you obviously we have block commands
for peripheral device that 0 0 and the
multimedia MMC commands and reduce block
commands in the future if a new set of
commands come out you would want to
probably build a utility class for that
so how are scuzzy tasks processed we've
talked about just Stan she ating and
building these stacks up so what happens
and I all request is going to come in
from the system for something and it's
going to get into the transport driver
layer and it's going to get packaged up
into a scuzzy task that scuzzy task is
that I'm going to get handed off to the
scuzzy protocol driver layer and that's
going to get packaged up into the
appropriate for the target device so in
the case of firewire the scuzzy task
will get turned into an orb that or will
be thrown out
the bus process it all happens
asynchronously and the completion
routine will fire back up the stack so
device compliance we really don't want
you to have to reinvent the wheel here
by sub classing and adding your device
specific stuff you get power management
you get Driver life cycle stuff hot
plugging support all the termination all
the gnarly undocumented stuff that you
really don't want to do what we mean by
device compliance is that your device
will process commands as they're
documented in the command sets and the
official scuzzy documentation I always
override the whole stack all you need to
do is out score the probe at the
protocol layer and do anything you want
we will not prohibit that so what's
supported in 10 as it ships today is we
have a top view support firewire support
and USB support ironically we don't have
scuzzy parallel support at this time
it's a work in progress and we're
investigating it and working on it and
ata just doesn't apply because it
doesn't use us because you can fly a
command set in theory what we're
discussing that it would be possible to
actually make a shim layer and wrap up
ata requests and use our architecture
but it added an extra layer that we
didn't want to take the performance hit
on so we instead have a monolithic ata
driver that ships today so I'm going to
grab some water
accessing devices from applications
there's a lot of compelling reasons you
want to do this by resources we mean if
you don't have to wire memory down in
the kernel we don't want to that's
really precious stuff there's a much
better tool environment up in user space
you have a source-level debugger you
have much better tools you don't have to
use gdb from the command line and system
stability is another thing if you blow
up in user space there's a less of a
statistic probability will take them a
whole machine down if you blow up in the
kernel it's much higher risk examples
would be utilities that need to set some
kind of intrinsic device feature with a
vendor specific cdb in any out of kernel
driver what I mean by that is we ship a
block storage a multimedia magnificat
magnetic optical and a RBC logical unit
driver that are in kernel and if you
have a tape drive which is a peripheral
device type one for sequential access it
really doesn't need to be in the kernel
as a general rule of thumb if your
device doesn't require booting and it
doesn't require a file system its driver
does not need to be in the kernel user
clients are nothing more than a
mechanism to negotiate the user kernel
boundary there may be other places in
this big stack that you want to connect
you might want to have a user client
connect to the physical interconnect
layer in the case of firewire if you
look in their SDK they have a user
client that allows you to click connect
to the SBP two layer you might want to
have a utility that does firmware that
connects at a lower device or a lower
point in the stack I'm also this is a
good place to point this out on our
website in the developer documentation
there's a document called accessing
Hardware from user space that just
recently was updated and went from like
50 to 100 pages and that's something you
want to really get your hands on
especially because there's a summary of
the i/o kit that's a lot more extensive
than what I've got into here I found it
to be a good reference for
getting my head around some of the basic
concepts of iokit the scuzzy tasks user
client is what we've implemented at our
layer for user client access we have two
access points we have the peripheral
device type nub access point back if you
think back to the diagram I showed this
is the point where you would attach a
non in kernel logical unit driver and we
also have the generic service layers for
multimedia and this is for authoring and
it all aided it I'll get into more
details in the next slides this the user
client uses a commlite CF plug-in
architecture again the document I just
referenced the accessing hardware from
user space goes in to some of the rules
in that and it's there are some tools
that will provide an SDK to show you how
to do all that we have three interfaces
into our scuzzy client the first one is
the amp MMC device interface this is
only for the authoring needs this
basically exists so you can get device
and media information for the purposes
of authoring we have a scuzzy device
interface which lets you create release
and do all the maintenance things that
you need create the callback handlers
and connect up the CF run loops etc and
finally there's the scuzzy tasks
interface once you've actually connected
up this is where you send and you this
is the interface you use this in raw
scuzzy tasks that encapsulate the CTB's
to drive your device so we have two
access models we have an exclusive
access model and a non exclusive access
model once your havest anshi ated the
user client that user client is the
logical unit driver so I'll have a
diagram in a few slides that will really
bring this home but there are no
restrictions the client has absolute
control
we don't filter disallow any kind of
commands and this last point if you
happen to be using the generic services
for multimedia they are authoring api's
and you need to transition to exclusive
access what has to happen is you have to
unmount any mounted media anymore any
amount of partitions and obtain a
reservation for that media through disk
arbitration and in our SDK that will
have an example code to show how to do
this the non exclusive access model is
for the generic services multimedia
access point only this is like I said
before only to get device and media
specific information for the purposes of
offering it is API driven you cannot
send raw CD B's and if an existing
client a user client has got exclusive
access it will not allow non exclusive
access so here's an example if we have a
tape backup drive that gets discovered
by the physical interconnect layer in
this example it's firewire again it
would be a stanch eiated and built up
and the there'd be a scuzzy protocol
driver accentuated and it would build up
in the i/o peripheral device now what
issue an inquiry find out that it's a
peripheral device type 1 populate the
registry with it and just stop then what
will happen is you will launch an
application up in user space and the
scuzzy test user client that we provide
straddles the user kernel boundary and
once the application is launched it will
look for the appropriate peripheral
device information through a command
called I always get this wrong io
service get at that point the client
application is the logical unit driver
and it can drive that device the other
interesting one is in the case of
authoring where we have a DVD device
that's been a stanch eat it up and we
have an internal driver a peripheral
device driver type v that's going to
drive this device and we might have been
doing
io2 it all day long and eventually an
application will get launched and your
user client will be this is the second
type I was talking about you can send
API you can use the API is to get device
information and media information in a
non exclusive fashion at some point you
will have unmounted the media and you
will seized control and become exclusive
access you would again call the i/o
create CF plugin for service at which
point the client application has become
the logical unit driver and we will
quiesce the internal peripheral device
type driver v so now the user client
logical unit driver has absolute control
over the device and can do whatever it
needs to do and in summary it should be
relatively straightforward and easy for
you to instead of having the ship
gigantic monolithic drivers for all of
your stuff just bundle up and ship out
strategic small checks and make
everybody happy
resources so I've covered most of this
and the slides already that there's
nothing here that I haven't and later
this week the I up update was yesterday
the PCI drivers are is tomorrow morning
firewire and depth and both USB in depth
there's a good place to go if you need
to get more information on bus specific
stuff if you have a bridge that you your
company is providing that has unique
requirements it might be interesting or
helpful for you to go there and I'll be
there if anyone has any protocol
transport specific information they'd
like to ask me about and finally we have
a list a mailing list that you can get
on this is a place where we'll be making
announcements of where you can get SDKs
and help on everything we have and if
you have any questions at all please
feel free to get ahold of Craig
chiefly
you