WWDC2004 Session 103

Transcript

Kind: captions
Language: en
good afternoon welcome to the afternoon
session this is actually my second WWDC
presentation so relatively new to Apple
in that respect and one of the growing
group of UNIX professionals that we're
finding in the core OS group at Apple
which is very good to see a very strong
UNIX background in in the in the group
so today we're going to be talking about
kernel programming writing kernel
extensions for Mac Mac OS 10 in
particular what we've done in Tiger and
how that's going to be affecting you
so the weave that makes them mate made
some major changes to the kernel in
tiger for a number of good reasons and
to be perfectly honest most kicks are
going to be unaffected by these changes
but there are a small number of that
will be affected and what we'll be doing
talking today about is is how these
changes affect those case there will be
source changes needed to these need to
be made to these kicks and we'll cover
what they are and what you need to do so
what you'll learn today is why these
changes to the kernel are necessary
which kicks are affected what changes
are needed tech needs techniques you'll
need to apply so that your kicks will
work correctly how kernel interfaces are
managed the lifecycle of a kernel
interface and generally why what we've
done is a good thing so just so that you
know what we're talking about today this
is what I would consider to be basically
the core OS of the core of the the
operating system the UNIX part of of OS
10 and what we're talking about today is
pretty much the BSD file system
networking area and the changes that we
made in this space
so let me discuss a bit about the
problem that we've been facing so here
we are back at the kernel and the core
system and we have your text so you load
your text into the kernel this
particular one is a networking case but
it's true of others and there are parts
of the kicks that have got functional
references and these references are fine
we can tell what they are there these
are unresolved symbols in your text and
we can understand those but some of them
are really unsustainable from our
perspective they're reaching into parts
of the system that we really consider to
be an implementation detail as opposed
to what we would consider be a
supportable interface in addition there
are a whole set of what I would call
detectable data references these are
things where you've you've reference
data certain data structures and the
symbols for those are there
therefore unresolved in in your text and
we can see that you're touching those
but some of these are also what we would
consider to be unsustainable they're
pointing in to data structures that we
really consider to be private or you
know an internal part of the
implementation as opposed to a anything
that you should be using as a
programming interface but worse still is
that there's a whole set of of what I of
indirect data references this is where
you've gone into a data structure and
inside that data structure there's a
pointer to something you've followed
that pointer you picked another pointer
out of there and you've potentially gone
all over the kernel and we have no clues
to what it is that you're referencing
and clearly some of those are very
unsupportable as well
but the problem is that we consider
binary compatibility to be paramount and
yet we have no visibility and what was
actually being depended on so what has
happened is that kernel development was
grinding to a halt where there was lots
of big features that we wanted to
implement that we just couldn't do
because we would almost certainly break
somebody part of the problems that we
didn't actually know who it was that we
were going to break hi there so we had
to make changes very very cautiously but
the bigger problems was that new
hardware support was extremely hard and
in fact one particular example of this
was when we introduced the the g5 we had
to obsolete the interface for called P
map extract which was really not a
programming interface that we would
wanted to export we actually had a
perfectly good one we're using higher
memory descriptors but not everybody
used it so new hardware support was was
becoming more more difficult and we
certainly saw that this was going to
become increasingly more difficult as
time went by we were limited in the
number of new features we could
implement because of the disruption that
that would have potentially to data or
to implementations where we'd have to
change interfaces inside the kernel it
even got to the point where performance
optimizations couldn't be done because
we would have to move things around and
therefore potentially break things some
bugs were not being fixed because we
couldn't add fields to data structures
in general the code paths in the kernel
were being warped and despite all this
we'd still find that developers would
occasionally break
so we decided that we had to bite the
bullet and tiger as the release to make
progress we needed to desperately or a
desperately needed function a
functionality what we wanted to do also
was to extend the IO kid experience
where he had stable interfaces and
extend that experience out to the rest
of the kernel
so that qex could be given much stronger
guarantees of binary compatibility from
release to release as I mentioned before
IO kicks iokit kicks are unaffected and
that's by far and away the majority of
kernel extensions but we will be
breaking binary compatibility for for a
very few for kicks and these cakes are
really the ones that are dependent on
the Condor Apple look kernel got BSD
sets of symbols and we'll talk about
about that more later
so as I said we had to add who we needed
to add new functionality and so we have
a new heart for Tigers Colonel SMP
support is clearly a big thing if you
noticed that all our g5s are now SMP are
all dual processors the previous funnel
architecture was just not serving them
well so we had to radically change that
64-bit application support is another
new feature in tiger if you have a
64-bit processor 64-bit applications
seems to make an awful lot of sense
fastest term access control list this is
a feature that people in the enterprise
space have been pushing for for quite a
while and now we've had the opportunity
to perform certain optimizations one of
the most of biggest examples of this
that I can that came up recently was
that there's a certain amount of data
attached to every V node for read and
write operations
now not all V nodes have read to and
written true for example all directory
be nodes and even if they are they do
have i/o associated with them it's
normally a one type or the other and
that this this lump of data that's
attached to the Vino was actually bigger
for write than it was for read but it
was present on all V nodes and read
operations are far more frequent in the
system and so we've been able to
actually change the data structure
remove those those extra pieces of extra
fields and only attach them when the V
node is in that appropriate operation
and doing this actually saved us a
megabyte of wired memory space so this
is non-trivial stuff that we're dealing
with and finally as I said there were
bugs that we could fix the fact that we
can now add a field to a data structure
without potentially breaking something
has actually allowed us to go in and fix
some bugs that we've had hanging around
for a little while
so improved SNP support at last people
have been asking this offer for a long
time especially in the server space
where they've noticed that the funnels
our servers tend to operate in in one
mode so they'll be doing sort of a lot
of networking operations or a lot of
file system operations and so therefore
they can't take advantage of the split
funnel quite so so much and so they've
been hitting this contention on the
funnel so finally we're getting to break
it out and remove it from our code
insert finer grain locking I use the
word finer on purpose here we're not
going down and locking every last little
thing we're trying to put the
granularity at the right level for
efficient operation we've also taken the
opportunity to restructure a bunch of
code as I mentioned earlier on code was
sort of kind of getting warped around
because we had to tread carefully not to
break binary compatibility in the in the
first place and now we've been able to
streamline the algorithms much more
successfully which is actually led to
some performance improvements and we can
reorganize the data to associate the
data with the locks that are protecting
them so in Panther this is really what
the system looked like we had as you
went in through the users user space
system call entry point we took the big
kernel log and as you pass down through
generic system call handling impacts the
file system had a faster than descriptor
handling code down through say structure
file structures and into the into the
file system you were holding this one
lock at the time and the kernel was
fundamentally single threaded and you'd
go right the way down to the i/o layer
and once you got into i/o kit then that
lock was dropped in your back into a
much more parallel environment again
if you were doing networking calls then
you would still take the same lock as
you came into the kernel and then you'd
go down through those layers again into
the networking side of things where we
drop the kernel funnel and take the
networking funnel and then the entirety
of networking stack would then be single
threaded from networking's perspective
down to the device layer again so there
was very very coarse level of
synchronization within the kernel this
has all sorts of interesting problems
apart from the obvious parallelism
problems but it's it's amazing how lazy
you can get when you believe that nobody
else can can mess around with you while
you're executing and you can start
accessing random data structures without
any real discipline so what we now have
is much finer grain locking in the
kernel with locking at most levels of
the system being independent from each
other
the locks are either at the subsystem
level or at the object level so here's
subsystem level locks where basically
there's a single lock to be able to
allocate em buffs for an example and
then there are other locks which are
held actually on the individual objects
so we're actually intelligently looking
at the granularity of locking that needs
to be held at all where's the system so
that we don't introduce arbitrarily
fine-grain locking which would increase
performance overheads or have to coerce
a grain level which means you wouldn't
get the parallelism that you would like
so we believe that we actually have a
reasonable level of locking granularity
throughout the system and and our
general philosophy is that we start
coarse and then move finer grain as
we've determined that there are
performance issues
another big feature 64-bit support in
Panther that the support was somewhat
limited we gave you a dashi that should
say a 64-bit physical address space I
apologize about that which allowed you
to have large amounts of physical memory
in the machine and the kernel could then
handle that we also gave you access to
64-bit instructions and 64-bit registers
for performance reasons however the
usage for those tended to be in
assembler leaf functions so it wasn't
universally applicable but it did allow
certain very important mathematical
operations to be optimized to take full
advantage of the processor so this
address that a couple of the dimensions
of 64-bit support in terms of large
physical configurations and some
performance benefits that you can get
from using a 64-bit architecture in
Panther we've decided we want to be able
to leverage the full power of the g5 and
so we're supporting 64-bit non GUI UNIX
applications so that basically means
we're providing a 64-bit Lib system and
that's it the reason we're doing this is
because we're aimed we're aiming at
applications which tend to have a
naturally large data set so these are
compute intensive applications in
general and these tend to come from the
UNIX world what we've had to do in the
kernel is add support for multiple api's
so we have to be aware the fact that
system calls can come through either
from a 32-bit application or a 64-bit
application which means that they have
different address spaces and therefore
different size pointers in that and
these structures within the kernel or
the structures that they pass to the
kernel I should say could be different
formats and so we had to be able to deal
with that a lot of that's been at the
system call entry time but there's other
places within the kernel that they do
that we've had to make changes to
support a large address space
breaking the 4 gigabyte Bamar barrier
but of course if you have a non GUI
application you probably want to have a
GUI application to represent the results
of your app and so we've had to do 32 to
64 bit API interoperability so we have
done various different shared memory IPC
kinds of communication parts between the
apps with different api's file system
access control lists fundamentally the
different ranges that we've had to do is
to centralize the credential checking
for authorization to access to to
operations basically eliminating the UID
GID comparisons that you've seen
sprinkled throughout the system we have
new kernel interfaces for these checks
that are I hesitate to say capability
based but because it's not a capability
system but effectively given an object
and your credentials this is the
operation that you'd like to apply to
this object do I have permission to do
so or do I have the authorization to do
so and we've also enabled native file
system support for for ackles through
changing the v node interfaces so given
the changes to the kernel that we'd made
what are the changes that are that you
will have to do for kernel programming
for your case
so we discussed this last year we've
added a whole set of formal kernel
programming interfaces in Tiger IO kit
already had its own set of formal
interfaces so IO kid cakes are basically
unaffected by this the KX types that
we're supporting in the bsd side of
things are file systems and network
extensions and filters there's probably
a few more details but that's
fundamentally the the set of kicks that
we're supporting through these KPIs one
of the big things is that there's no
more open and indiscriminate access to
all of kernel code and data being
supported by these formal interfaces now
we understand that there are some kicks
for various reasons might want to get
access to the whole of the kernel we
would advise that not to be in product
code simply because these kicks if they
do get at kernel access or access to the
whole of kernel symbols will be tied to
exactly one kernel version so whenever
we update the kernel those cakes will
not load however if you use the kpi's
you're guaranteed release to release
binary compatibility and of course
source compatibility
so what are the kpi's do they basically
encapsulate all the kernel objects that
have one to be accessed by kernel
extensions we've made all the data
structures a Paik so that you don't have
to go and access the the data structures
directly which means that these data
structures are now invisible to you and
we can change their organization we can
add fields and do whatever however you
can still get access to the information
contained with in these data structure
structures by using accesses perhaps
more importantly is that the locking
that we've implemented implemented for
synchronization for these various
objects is completely hidden behind the
ape or the KPIs all the objects are
reference counted so when you access an
object in the kernel you have to
formally look it up basically do the
operations that you wish to and then
release it not doing this will cause
leakage in reference accounting that
will create that that will cause
problems and bugs in the system later on
so basically what I'm saying is that the
semantic model for a lot of the objects
within the data structures in the kernel
has changed because of this reference
counting and because of locking so don't
be fooled into thinking that you can
just take you know the Vino pointer that
you used to have and do all the
operations that used to have just sort
of recast the the pointers that you were
getting back from the lookups because
this won't work we've also gone through
and changed names of various interfaces
data structures within the kernel and
we've done this intentionally it's not
just to annoy people it's basically to
give you a strong signal that the
semantics of that object has changed and
you have to relook at that code and
change your code to reflect the
semantics of those objects
as I was mentioning before you can
actually use all of the kernel and if
you don't use those KPIs and you decide
to go off and get a hold of hold of a
data structure that is encapsulated
within the KPI you can do that however
you must understand and correctly honor
all of kernel locking and the changes
that we make to that over time so this
is really not a great idea however using
the KPIs the kernel implementation of
those objects it does not affect you and
so therefore whatever we do will still
remain compatible for you
so the other thing that is going to
affect you is that we've removed the
funnel significantly from our code that
the finalists in the seed is certainly
still there there's large parts of the
system that are still protected by it
but that's being removed as we progress
so you should be aware the fact that
what you're seeing is is development
work in mid-cycle
but the goal is to remove the funnel
from app from Apple code before we ship
tiger what this means is that the kernel
is no longer single threaded and as it
passes through kernel extension code
there could be multiple threads passing
through this now so your kicks need to
protect their own data if you have data
inside it you have to be aware the fact
that there could be multiple threads
trying to access this and so therefore
if you need any atomicity you're going
to have to do this and to do that we
have a rich set of synchronization
primitives spin locks mutexes
reader/writer locks and some atomic
operations for you to use by and large
we think that the mutex is actually
going to be the one that you're going to
be using most we also recognize that you
know it might be difficult for some
kicks to become multi-threaded its
multi-threading is sort of supporting
SNP and locking is actually not a
trivial exercise as we found out so if
there may be times where you actually
want to be able to load and still be
protected by the funnel and so we do
have single threaded funnel executions
supported for example in as you pass
through the vfv node layer and going
into the file system we will take a a
funnel operation that will allow your
code to be single threaded
so because we've added all these lovely
synchronization primitives for you to
use it would be rather unfair not to to
help you debug them because locking is a
tricky business so we've added a whole
bunch of things to help you debug your
code in the face of deadlocks and so
forth so we have a bunch of things for
lock debugging for mutexes as I said we
expect that mostly you'll be using mutex
is in in your in your code so for for
mutex locks you can get the current
owner for the mutex and if you've got a
read/write a lock if it's held exclusive
you'll get the owner for that the PC of
the last time that the lock was was held
so you know where in your code it was
that lock was taken and who it was that
took it as in the thread also there's an
option where you can actually get the
law that the back trace of that last
lock for 8 frames this will indicate the
path that you got down there because
frequently you know you've got you've
taken the lock in some sort of generic
function you have no idea who it was
that's called it and it was called in
you know one of 40 different places so
we give you the back trace option to
find out exactly the path that you took
to get to that log we also have some
runtime checks to help you get to help
prevent you from getting into trouble we
assert that you actually own the lock
when you unlock it we don't allow one
thread over here to go off and lock
something and then effectively hand the
object off and be unlocked by somebody
else that's generally considered to be
very bad style and leads to a lot of
almost undue Bugaboo situations and so
we just assert that you have to own the
lock to unlock it if you've taken the
lock exclusive we detect that you've
just taken it again that's almost
certainly not what you really meant to
do because that implies that there's two
code paths that want to use this lock in
an exclusive mode at the same
time even if it's the same thread and so
we detect that we make sure that in the
interfaces that require synchronization
our locks that you've actually passed
the right lock type in and you're not
getting confused and giving it simple
like we're a mutex was needed and this
one is is again a stylistic thing in
some respects that we if you take a lock
and you need to go to sleep we assert
that you actually don't hold any
resources that that prevent you from
going to sleep in particular that means
another lock basically it's if you're
going to sleep for long times in the
system then you shouldn't be holding
kernel resources if you you know if
you're waiting for i/o or something of
that nature if it's a non bounded lock
sleep and you shouldn't be holding
kernel resources because that will lead
to deadlock we also have some log
statistics being kept
I my lights go off here is it working
okay sorry about it so long statistics
sorry van so how long the lock has been
held for average and maximum lengths of
time clear lead over the whole locks for
very large periods of time because that
will increase the likelihood of
contention we also count how many times
a lot was contended and when we when you
take mutexes to be sort of like hang
around and look if there's the lock
holder is still active on another CPU so
[Music]
if the lock is held by somebody on
another CPU with them we don't
immediately go to sleep you sort of wait
for a bit just to see if they're gonna
be holding the lock for a short period
of time and then you'll immediately jump
in as soon as they release it so we
differentiate between contention and and
actually sleeping on the log so there's
a bunch of stuff that that will help you
be able to find your way through the SMP
morass so for 64 bits in the user in the
kernel we thought about how best to
address this issue the fact that you've
got two different kinds of address
spaces out there so we decided that that
we would represent user address by
addresses in the kernel as 64 bits
regardless of whether it's a 64 bit
process or a 32 bit process so the
kernel data type for a user address is
64 bits whenever you actually access the
user address space the access is needed
that that you use to access it have been
changed their signature has changed
because of this so if you use things
like copy in and copy out or UI or move
then those interfaces have also changed
and also the ironn memory descriptor has
been extended for 64 bits this really
only affects you if you're creating one
as opposed to just handing one around
and so fundamentally it normally only
affects the user client providers
so for filesystem ackles as i mentioned
earlier on we have a new authorization
model with and have eliminated all the
UID GID comparisons in the kernel and
have used this new centralized
credential manager
sorry centralized authorization service
called ko in addition we have new
credential management interfaces
you can't cons up your own credentials
and hand them off to the system because
these things are now all reference
counted so you have to create them and
manipulate them through KPIs
we have as I mentioned this sort of
authorization model which is a
capability checking interface am i
capable of performing this operation on
this object and this is actually
extendable we have pluggable
authorization policies that you can
insert into what are called scopes which
gives you the ability to contribute to
the decision-making as to whether
somebody can perform this operation on
this set of objects this was covered in
the the file system ACL talked yesterday
and I hope people cause it but failing
that if you would like to catch that on
your DVDs that'll be explained in a lot
more detail so we also have added native
support for faster stoom a CLS and if
you your file system does support Ackles
then there are new v8 VN ops to be able
to plug into to be able to export those
four file systems that don't support
native ackles if they do support
extended attributes we have new VN ops 4
to support extended attributes natively
and if you have those then we actually
store the ACLs in the extended
attributes if you your file system
doesn't support either then we store
extended attributes in you know Apple
double files the dot underbar files and
then ACLs are then stored in the extent
of attributes in that way so even if you
even if you don't have any of these
features will still expose them to the
user by smoke and mirrors basic
but this gives you the opportunity of
taking existing file systems from other
UNIX systems that already have these
advanced features and now being able to
support them natively within OS 10
so at this point I'd like to welcome
Dean Reese who's the manager of the IO
ket team to talk about how access to
kernel interfaces is managed within the
system good afternoon
okay so first I'd like to talk a little
bit about some extras that you're
getting with your I guess it's available
on the website the Apple Developer
connection website content for this page
for this for this session we've got
several interesting new tidbits for you
to help with your text development for
Tiger in no particular order we've got a
plugin for firewire debugging basically
this allows you to do the two machine
gdb debugging through a firewire
connection
functionally it's similar to the
Ethernet debugging that you can already
do it's got a few advantages one being
that it's available much earlier during
boot and sleep-wake so if you're trying
to debug a sleep wake problem you have
earlier access also we've got a new
command called hexam for gdb if you're
connected to another machine and you
have kernel extensions loaded and you
like to generate symbols for those
rather than having to generate them
manually you can run XM and point it at
a folder on your local machine that has
all the has all the binaries
yeah we uh we definitely appreciate
having this internally as well so
basically it will generate symbols for
all loaded kernel extensions in a single
command we've also got a Perl script a
little utility calk extract now this is
something it's a bit of an experiment
but as Simon said earlier we have we
have a situation where we have a lot of
qex developers out here doing really
cool development but we don't have a
good understanding of what all you
depend on what specific API is you're
using and why so this is sort of the
beginnings of a feedback mechanism and
what kext rec does is you run it against
your kernel extension and it basically
extracts a whole lot of information
basically it's looking at the pea list
and it's looking at the undefined
symbols in your binary and it formats
that into a little text file that you
can send to Apple by way of my way of
radar web or bug web and we're gonna
keep these in a database that will allow
us to search anytime we want to make a
change or we're just curious who's using
an API we'll be able to find any kernel
extension using it we'll be able to find
trace it back to the developer who
submitted it this is purely voluntary
you know we're gonna be using this again
it's a bit of an experiment it's not a
certification program there's no
guarantees made but this will be a good
way for you to let us know what it is
you're using it'll be a good way for us
to find out and possibly contact you
about upcoming changes so that I think
this could be very useful so we've also
got a new kernel available on the
website this is probably the single
largest piece of the download but the
kernel that actually shipped on the
tiger seed still has a few KPI bugs in
it that we wanted to get fixed so that
you could play around with it but it was
a little late in the the tiger seed
development cycle for us to be putting
kernel changes in for something that
wasn't critical path for the release so
if you're going to be doing KPI
developer qex development you're going
to be using the new KPIs you should
install this kernel you should only
install this kernel on top of the WWDC
seed and there's a readme in there that
has
and strict instructions for how to
install it we've also got some
documentation and examples in the image
the examples are works in progress that
two file systems that we have there
WebDAV FS and ms-dos FS these are the
source code for the actual binary that
shipped on the tiger seed so you're
looking at the same thing that we
actually sent in to get built and as I
said there works in progress these are
these are not in their final form but
they do work using the KPIs and it'll
give you something to look at sort of a
starting point we've also got a couple a
couple other examples that are not
shipping code they're just specific
tight examples of a one of two KPIs
about one for filters and one for socket
one for a socket filter and one for an
IP filter the documentation that's on
the site is a collection of header doc
and some RTF files sort of porting
guides to help you one go through and
actually examine the KPIs and their
intent and their parameters and so on
and the porting guide is a little bit
more of a tutorial basically trying to
walk you through the process and it was
actually written as a experience that we
went through doing porting our own file
systems and and such to the KPIs
obviously is this whole KPI process
moves forward will make more information
available but this should be enough to
get you started ok to talk about kernel
extensions a little bit we're we're
making use of something called interface
texts now fundamentally they're there
they look and feel like any other kernel
extension but there are a few special
things about them we actually introduced
this concept in Panther and you might
remember me talking about them last year
but we didn't really use them until now
the best way to think of them is there a
library mechanism for the KPIs the KPIs
as you know it are built into the kernel
so they don't really need a text but
we're using the text mechanism as a way
of managing and versioning the
interfaces so they really only contain
linkage they don't contain any
implementation whatsoever the I'll
actually show you some diagrams in
that show you how the linkage works but
fundamentally it's just a big symbol
table wrapped in a text bundle so what
will happen is in the past your kernel
extensions linked against the kernel and
for tiger and going forward they're
gonna link against these interface texts
instead and that'll have the advantage
of course allowing us to have multiple
versions of the same API set or same KPI
set coexisting on the system and of
course the text mechanism already has
versioning built into it so that will
allow you to express the version that
you need and will automatically link up
to the right thing so it really gives us
a mechanism to control when we introduce
a new KPI set how it evolves over its
lifecycle and then when it comes time in
the distant future to obsolete it it
gives us a way to do that I like to
think of it as sort of a conveyor belt
of interfaces that you know we can set
to be several years long but we'll
introduce an interface and keep it
around as long as it makes sense and is
supportable alright so in our previous
operating systems anytime you linked
against any of these 5ive this OS bundle
libraries or CF bundle identifiers you
were getting the whole kernel now what
we had done is we subdivided the kernel
into four basic areas because we knew
that we knew that there were these
logical divisions within the codebase
itself and the way people were using
them but we didn't really create a
linkage difference because hey you were
linking against the kernel so the to
talk about what each of them are a
little bit I think they're it's fairly
obvious if you look at the header
structure which one of these an API or
KPI is actually being supported by
obviously anything that is part of the
i/o kit header subtree is part of i/o
kit itself and so comm Apple kernel i/o
kit is the appropriate one there BSD and
mock similarly have their own namespaces
and their own header spaces Lib Kern is
something we created as a common library
for all of the things that the three
other main areas of the kernel needed to
share some of the
the primitives Lipsy type stuff memory
some memory moving copying stuff like
that atomic operations so some of the
things that you might think of as bsd
ish like you know b copy is not part of
bsd that's actually part of lib current
and we'll be providing more guidance on
exactly which one is we're coming up in
information very soon okay so anyway for
all versions of the kernel prior to 8.0
when you link against any of these
you're getting the whole kernel so this
is this is a diagram i actually have
several animations coming up around this
to kind of show you how this design is
evolving but fundamentally we had the
kernel with these four partitions inside
of it logically being treated as a
single thing and the kernel extension
that you see here in orange is basically
a pure i/o kit kext in that it only
depends on Io kit and Lib Kern the Lib
current is allowed because it's common
to all now the problem comes in because
you're linking against the whole kernel
if you access a symbol that's considered
part of BSD let's say copy n or maybe
you're using the dev efest to create a
device node to communicate with the
utility those things are not part of Lib
Kern or i/o kit and as a result you have
an undeclared dependency there we can't
detect the we can't detect it by looking
at your P list because you haven't
declared it but you're getting it
because you're linking against the whole
kernel so what we've done for Tiger is
to basically take this entire collection
of interfaces symbols and group it
together into one interface kext we call
it the compatibility text okay and all
existing kernel extensions that link
against existing versions of the kernel
will link against this and tiger but
what we've done is taking the
implementation out of it and moved it
off into a separate place so the kernel
continues to be the implementation
as it always has been but when you link
against any of those four areas you're
getting this compatibility text and of
course all of its symbols are being
resolved by the kernel now the new KPI
is that the things Simon has talked
about and other sessions at WWDC will be
talking about they're introducing new
interfaces in the kernel those in the
the final version of tiger will not be
in the compatibility text they will be
in new interface libraries interface qex
so again we have the same four we've
changed the name to represent the fact
that these are sustainable kernel
programming interfaces so comm Apple
kernel KPI
you're not linking against the whole
kernel you're linking against a single
interface okay so in this case if you
wanted to have a driver load against all
of those you could but you would have to
list bsd as an explicit reference
otherwise your text would not load
because of undefined symbols now another
thing I need to point out is we really
kind of have old world a new world here
and the two coexist on the system fine
but there's no crossover allowed in
other words you can't write a new style
text that then goes and tries to pick up
one or two old symbols that have been
deprecated you either need to run
against the old symbols or you need to
run against the new sets the other
aspect of this is as Simon had mentioned
earlier we have a lot of symbols that
were exported that we believe people may
be using we don't know for sure but
their implementation energy through
things that we don't believe kernel
extensions should be getting at and in a
lot of cases they cannot continue to
exist changes to the architecture of the
kernel semantic changes make it
impossible and so what's happened is
with Tiger a certain number of
interfaces the symbols that have been
there in the past are no longer there in
the compatibility text that will break
some drivers right there the the failure
mode that you'll see if you encounter
this is an undefined symbol when you try
to load your kernel extension and the
only way to remedy this is to fix your
code so that it doesn't need that
interface or to port over to the new
kpis which would obviously we would
prefer
and as the system evolves we have made a
commitment to continue supporting the
KPIs but you can expect as we vary
further and further from the the older
kernel we're going to have more and more
breakage of interfaces that can't there
were never sustainable in the first
place so you're gonna see those symbols
starting to slowly erode but again we're
not doing this because we want to force
people onto new interfaces we're only
taking them out because we have to for
performance enhancements for new
features and so on so we we obviously
want to provide you a way to move your
product forward and that would be the
KPIs okay so to sum up as I had said
will only remove the symbols that we
just read that we need to the whole
point of a compatibility interface text
is to be compatible right so if you have
a kext that you've shipped previously we
would like we'd like to see it be able
to load against that the reality is if
your text has dependency on BSD it
probably won't if you're doing anything
interesting with BSD you're probably
hitting on a symbol that's that's no
longer there and we'll need to move
forward to the KPIs if you're writing an
i/o kit text there's a very good chance
that you won't break because you're
probably not using the BSD innards in
that case but again until we know
exactly what symbols you're using it's
very hard for us to tell okay so to help
the users get through this transition
what we've realized is we don't really
have a good mechanism to talk to users
about specific kernel extensions giving
them a path name is probably not a very
good user experience so what we've done
is we've defined two new properties that
we'd ask you to add into your kernel
extensions that are OS bundle product
name and basically it should be anything
that you would use to identify your
product if somebody calls your support
line and says hey I've got your product
what do you need to know to identify it
we also realized that there there are
cases where you have a whole suite of
qex working together and from the
customers perspective it's a single
product well if we have problems with
with two or three of those we really
don't want to put
three panels up to the user and say you
know this one in this one or this one
what we're gonna do is look at the
examine these properties and if this
property is identical in multiple qex
we'll treat it as a single product and
we will only talk to the user is it with
it being a singular thing you know
obviously we don't want to we're not
going to try to we're not gonna scare
the user we're not going to put panels
up unless the product cannot be loaded
for whatever reason but we'd like to be
able to talk to them in a way that it's
a little cleaner we also realize that
you Matt you might have a fixed kernel
extension available and we'd like to
make it easy for the user to get at it
so if you can add the OS bundle product
URL which isn't again as an optional
property then if we wind up presenting a
UI to the user we might consider
including this as a shortcut so that the
user can go right to a downloads page or
a product support page to be honest we
don't know yet exactly how we're going
to use these properties that we know
that these are there to be presented to
the user in dialogues will probably have
Apple system profile or display them but
we wanted to get this message out for
WWDC and we don't have the the use
nailed down yet but put it in now next
time you ship the kernel extension make
sure you've got these in there
okay so there is as Simon pointed out
the possibility that you do need to get
at some of the kernel internals and you
might have a legitimate reason for
needing to do this we also have the
legitimate need to say that you're
getting at something that we can't
support you know we can't provide
release to release compatibility so
we've we've provided sort of the
compromise in that we allow you to
continue to link against the whole
kernel to link directly against the
implementation when you do that however
you have to request the kernel by an
exacta version number and whenever we
release a new kernel whether it be a new
major release and you software update it
could even be a security update the
kernel version will change if we if
there's a new kernel in that update and
as a result your kernel extension will
not load so you know by and large we
really see this as being useful for
internal development when you're trying
to port something over you're maybe
partly on the new KPIs and partly not so
you want to get at some of the old
symbols you can do that by linking
against the kernel until you're ready to
ship also maybe in an education
environment where you're not shipping a
product and also it possibly you know
the one commercial product where we
could see it being useful as a
subscription product or maybe you're
selling some sort of a security product
that you expect to update with each
major or with each OS update so you know
we have that available as an option but
we really recommend not using it for any
commercial product you should not link
against the whole kernel unless they're
just really is no other recourse and I
would also like to point out that we
have the KPI feedback form we'll talk
about that in a minute and if you do
find that you have to link against the
kernel you know maybe you should drop us
a note and say why what is it that's
missing that is required you to do that
and you know we may be able to do
something about that
all right on an unrelated note I'd like
to talk a little bit about IO kit and
64-bit as the i/o kit manager I wanted
to make this information available we
don't have a full Iowa kit session
because the core of i/o kit is remained
largely the same with tiger most of our
energies have been focused elsewhere
with the MP work and other things
but the kernel virtual address space
remains 32-bit in tiger
so for i/o kit kernel extensions
pointers are still 32 bits now drivers
can get to greater than 4 gigabytes of
memory just like they did in Panther
updates that supported the g5 and you do
that by calling io memory descriptor and
it has various successors of course you
need to call prepare and what that does
is that creates a mapping between the
physical memory which may be a large
address and a what we call a bus a
virtual address basically it's a 32-bit
address that can be passed around to
your device you can also for Pio mode
you can get directly to read bytes and
write bytes and that that actually reads
physical memory directly so that's how
you get it the physical 64-bit address
space now for applications talking to
i/o kit they can go through any of the
POSIX API s that that have been made
available as part of the 64-bit app
support so you know standard open closed
readwrite IO control so that'll work
fine for networking interfaces for the
storage interfaces and serial as well
all of those use of POSIX what it will
not support though is direct driver
access through user clients and the
reason here is that the i/o kit user
client model and Lib Kern and everything
else are about varying data directly
between an app and i/o kit driver is all
based on C F and C F is not being made
available for 64-bit applications at
this time so stay tuned we might you
know we might be able to do this in the
future but for now we're focusing more
on the 64-bit apps that are working on
large data sets and generally they're
not also scanning the bus for USB
devices so alright well with that I'd
like to hand the podium back over to
Simon to continue with the presentation
thank you
good job thank you very much Dean so I'd
like to briefly touch on now is the
lifecycle of a kernel interface so what
are the objectives behind having KPIs
well we want a set of well-defined
interfaces and for that for Apple what
that means is that we can focus on
making sure that we guarantee binary
compatibility from release to release so
for developers what that means is that
you now have a stable environment from
release to release that you can program
your solutions to we want to have a well
understood path for change between the
Apple and developers this allows Apple
to plan for kernel evolution so that we
know that we can change things we can we
can make plans to add new features and
we know how we can face this in and it
also sends a signal to developers that
they know when to my start to migrate
from one interface to another and we'd
like this to be predictable we are not
going to change KPIs arbitrarily or
gratuitously
just because we happen to think that the
third argument should go somewhere else
or whatever it is the only reason to
change a kernel programming interface is
because something is forcing our hand
because there's new hardware that is
being shipped or you know the technology
changes in Hardware all the time there
may be new who knows processors memory
architectures whatever that force us to
revisit some interfaces and it's so
these these KPIs will be stable from our
perspective
so we wanted a lifecycle that really
didn't give you any surprises and so
we've based the model on the API
lifecycle for applications service
so we basically have four modes of
within the lifecycle start off in being
supported which means that the interface
is fully source and binary compatible
from release to release this is going to
be the state when Tiger ships of
probably every single interface that we
have in the KPI set if we find that that
you know when we're looking forward in
our our development you know we tend to
be in engineering we are looking a
couple of years out and we see that
there's going to be problems in a given
interface for because of hardware
reasons or whatever then we will mark an
interface is being deprecated what this
means is that when you build your text
the compiler will start to generate
warnings on its use just letting you
know that that you're using an interface
that we've think is is we're going to be
removing the inter the interface is
still supported in terms of it will
still work but we're just basically
giving you a positive heads-up saying
you know this interface is at risk and
what we're reserving the right to do is
to remove the interface in the following
major release after that we're looking
at obsoleting the interface at which
point you will no longer be able to
build your text using that interface
will effectively remove the definitions
from the header files and there'll be
nothing for you to compile against but
the binary can the binary compatibility
is still assured you will still be able
to take your text and use it on a system
and then finally when we've got to the
point where we've had to implement the
given feature and we've had to change
the kernel that the kick the interface
will become unsupported which means that
the implementation behind that that
interface is no longer in the kernel and
therefore the case can no longer low
so what do we need you to do well
initially we'd like you or just yeah in
immediately we'd like you to adopt the
OS bundle product name and the OS bundle
product URL to identify your kext if we
talk to the user the correlation between
a kext name as we see it which is a path
name or or something of that nature and
the actual user perceived functionality
that it provides is really not evident
and so this is a way in which the user
can get a sensible interaction between
the system and your product actually so
that they can be identified more
reasonably we'd like you to download
this session extra material it's on
connector apple comm under set session
103 which is this session and look at
all the stuff that we've put in there
for you
all the additional documentation but
more specifically in the short term we
would like you to run a extract on your
text and to send the the report that k-x
tracked generates for you to apple via
radar web this will allow us to be able
to see before we even get to Tiger what
problems we may be creating and whether
we can actually alleviate some some some
problems for people so I do very
strongly urge you to do that I think it
will benefit both of us another very
important thing to do is once you've
downloaded the the extra material is to
review the interfaces there is full
headed dock for all the KPIs and you
should bear in mind when you look at
these is that these seed interfaces are
evolving you know we're porting our own
kernel extensions that as we speak we've
done some we haven't done all so as we
continue porting we'll we'll be spotting
some limitations or some things that are
missing or or awkwardness in some of the
interfaces and so we will be changing
things hopefully not radically but you
know just fine-tuning as we go so please
review those interfaces have a look at
the examples that we have in those
extras both the the file system ones and
the networking ones depending on what
your text is doing and see what changes
we've had to make I apologize for some
of the others we have fixed bugs in
there so there'll be some extraneous
changes not strictly related to KPIs but
it should be pretty obvious what changes
are are there for purpose for the
purpose of using the well-defined
interfaces versus not and finally well
not finally then I'd like like to
encourage you to actually try to port
your text using the kpi's it's very
important that that we understand your
experience with these interfaces because
we want to be able to ensure that it
will be able to fully support your
functionality and finally we would like
you to give feedback
there's the KPI - feedback at Apple comm
mail alias which we are all sitting and
watching and we would like to know your
experience is both good and bad in other
words if you found that moving your text
over was simple then we'd really like to
understand which sets of kicks were not
having problems and if you find that
there are some things missing or there's
some awkwardness we would be very
interested in that too because we'll do
our best to make this experience as as
easy as we can for you
so more information the contact at Apple
is Jason yo he's the Mac OS technology
manager there's email please let him
know of questions that you may have
already feedback importantly for us in
engineering the KPI feedback mail list I
say I encourage you to to let us know
your experiences with using the KPIs
there's some additional documentation
you can find on the DVD I was referring
to the network kernel extensions API
references as some 64-bit transition
guides a kernel programming document and
there's some additional documentation
for kernel extensions this is on the ADC
home website going down through darwin
some descriptions of lib can and there's
another document on on network kernel
extensions
you