WWDC2004 Session 103

Transcript

Kind: captions Language: en good afternoon welcome to the afternoon session this is actually my second WWDC presentation so relatively new to Apple in that respect and one of the growing group of UNIX professionals that we're finding in the core OS group at Apple which is very good to see a very strong UNIX background in in the in the group so today we're going to be talking about kernel programming writing kernel extensions for Mac Mac OS 10 in particular what we've done in Tiger and how that's going to be affecting you so the weave that makes them mate made some major changes to the kernel in tiger for a number of good reasons and to be perfectly honest most kicks are going to be unaffected by these changes but there are a small number of that will be affected and what we'll be doing talking today about is is how these changes affect those case there will be source changes needed to these need to be made to these kicks and we'll cover what they are and what you need to do so what you'll learn today is why these changes to the kernel are necessary which kicks are affected what changes are needed tech needs techniques you'll need to apply so that your kicks will work correctly how kernel interfaces are managed the lifecycle of a kernel interface and generally why what we've done is a good thing so just so that you know what we're talking about today this is what I would consider to be basically the core OS of the core of the the operating system the UNIX part of of OS 10 and what we're talking about today is pretty much the BSD file system networking area and the changes that we made in this space so let me discuss a bit about the problem that we've been facing so here we are back at the kernel and the core system and we have your text so you load your text into the kernel this particular one is a networking case but it's true of others and there are parts of the kicks that have got functional references and these references are fine we can tell what they are there these are unresolved symbols in your text and we can understand those but some of them are really unsustainable from our perspective they're reaching into parts of the system that we really consider to be an implementation detail as opposed to what we would consider be a supportable interface in addition there are a whole set of what I would call detectable data references these are things where you've you've reference data certain data structures and the symbols for those are there therefore unresolved in in your text and we can see that you're touching those but some of these are also what we would consider to be unsustainable they're pointing in to data structures that we really consider to be private or you know an internal part of the implementation as opposed to a anything that you should be using as a programming interface but worse still is that there's a whole set of of what I of indirect data references this is where you've gone into a data structure and inside that data structure there's a pointer to something you've followed that pointer you picked another pointer out of there and you've potentially gone all over the kernel and we have no clues to what it is that you're referencing and clearly some of those are very unsupportable as well but the problem is that we consider binary compatibility to be paramount and yet we have no visibility and what was actually being depended on so what has happened is that kernel development was grinding to a halt where there was lots of big features that we wanted to implement that we just couldn't do because we would almost certainly break somebody part of the problems that we didn't actually know who it was that we were going to break hi there so we had to make changes very very cautiously but the bigger problems was that new hardware support was extremely hard and in fact one particular example of this was when we introduced the the g5 we had to obsolete the interface for called P map extract which was really not a programming interface that we would wanted to export we actually had a perfectly good one we're using higher memory descriptors but not everybody used it so new hardware support was was becoming more more difficult and we certainly saw that this was going to become increasingly more difficult as time went by we were limited in the number of new features we could implement because of the disruption that that would have potentially to data or to implementations where we'd have to change interfaces inside the kernel it even got to the point where performance optimizations couldn't be done because we would have to move things around and therefore potentially break things some bugs were not being fixed because we couldn't add fields to data structures in general the code paths in the kernel were being warped and despite all this we'd still find that developers would occasionally break so we decided that we had to bite the bullet and tiger as the release to make progress we needed to desperately or a desperately needed function a functionality what we wanted to do also was to extend the IO kid experience where he had stable interfaces and extend that experience out to the rest of the kernel so that qex could be given much stronger guarantees of binary compatibility from release to release as I mentioned before IO kicks iokit kicks are unaffected and that's by far and away the majority of kernel extensions but we will be breaking binary compatibility for for a very few for kicks and these cakes are really the ones that are dependent on the Condor Apple look kernel got BSD sets of symbols and we'll talk about about that more later so as I said we had to add who we needed to add new functionality and so we have a new heart for Tigers Colonel SMP support is clearly a big thing if you noticed that all our g5s are now SMP are all dual processors the previous funnel architecture was just not serving them well so we had to radically change that 64-bit application support is another new feature in tiger if you have a 64-bit processor 64-bit applications seems to make an awful lot of sense fastest term access control list this is a feature that people in the enterprise space have been pushing for for quite a while and now we've had the opportunity to perform certain optimizations one of the most of biggest examples of this that I can that came up recently was that there's a certain amount of data attached to every V node for read and write operations now not all V nodes have read to and written true for example all directory be nodes and even if they are they do have i/o associated with them it's normally a one type or the other and that this this lump of data that's attached to the Vino was actually bigger for write than it was for read but it was present on all V nodes and read operations are far more frequent in the system and so we've been able to actually change the data structure remove those those extra pieces of extra fields and only attach them when the V node is in that appropriate operation and doing this actually saved us a megabyte of wired memory space so this is non-trivial stuff that we're dealing with and finally as I said there were bugs that we could fix the fact that we can now add a field to a data structure without potentially breaking something has actually allowed us to go in and fix some bugs that we've had hanging around for a little while so improved SNP support at last people have been asking this offer for a long time especially in the server space where they've noticed that the funnels our servers tend to operate in in one mode so they'll be doing sort of a lot of networking operations or a lot of file system operations and so therefore they can't take advantage of the split funnel quite so so much and so they've been hitting this contention on the funnel so finally we're getting to break it out and remove it from our code insert finer grain locking I use the word finer on purpose here we're not going down and locking every last little thing we're trying to put the granularity at the right level for efficient operation we've also taken the opportunity to restructure a bunch of code as I mentioned earlier on code was sort of kind of getting warped around because we had to tread carefully not to break binary compatibility in the in the first place and now we've been able to streamline the algorithms much more successfully which is actually led to some performance improvements and we can reorganize the data to associate the data with the locks that are protecting them so in Panther this is really what the system looked like we had as you went in through the users user space system call entry point we took the big kernel log and as you pass down through generic system call handling impacts the file system had a faster than descriptor handling code down through say structure file structures and into the into the file system you were holding this one lock at the time and the kernel was fundamentally single threaded and you'd go right the way down to the i/o layer and once you got into i/o kit then that lock was dropped in your back into a much more parallel environment again if you were doing networking calls then you would still take the same lock as you came into the kernel and then you'd go down through those layers again into the networking side of things where we drop the kernel funnel and take the networking funnel and then the entirety of networking stack would then be single threaded from networking's perspective down to the device layer again so there was very very coarse level of synchronization within the kernel this has all sorts of interesting problems apart from the obvious parallelism problems but it's it's amazing how lazy you can get when you believe that nobody else can can mess around with you while you're executing and you can start accessing random data structures without any real discipline so what we now have is much finer grain locking in the kernel with locking at most levels of the system being independent from each other the locks are either at the subsystem level or at the object level so here's subsystem level locks where basically there's a single lock to be able to allocate em buffs for an example and then there are other locks which are held actually on the individual objects so we're actually intelligently looking at the granularity of locking that needs to be held at all where's the system so that we don't introduce arbitrarily fine-grain locking which would increase performance overheads or have to coerce a grain level which means you wouldn't get the parallelism that you would like so we believe that we actually have a reasonable level of locking granularity throughout the system and and our general philosophy is that we start coarse and then move finer grain as we've determined that there are performance issues another big feature 64-bit support in Panther that the support was somewhat limited we gave you a dashi that should say a 64-bit physical address space I apologize about that which allowed you to have large amounts of physical memory in the machine and the kernel could then handle that we also gave you access to 64-bit instructions and 64-bit registers for performance reasons however the usage for those tended to be in assembler leaf functions so it wasn't universally applicable but it did allow certain very important mathematical operations to be optimized to take full advantage of the processor so this address that a couple of the dimensions of 64-bit support in terms of large physical configurations and some performance benefits that you can get from using a 64-bit architecture in Panther we've decided we want to be able to leverage the full power of the g5 and so we're supporting 64-bit non GUI UNIX applications so that basically means we're providing a 64-bit Lib system and that's it the reason we're doing this is because we're aimed we're aiming at applications which tend to have a naturally large data set so these are compute intensive applications in general and these tend to come from the UNIX world what we've had to do in the kernel is add support for multiple api's so we have to be aware the fact that system calls can come through either from a 32-bit application or a 64-bit application which means that they have different address spaces and therefore different size pointers in that and these structures within the kernel or the structures that they pass to the kernel I should say could be different formats and so we had to be able to deal with that a lot of that's been at the system call entry time but there's other places within the kernel that they do that we've had to make changes to support a large address space breaking the 4 gigabyte Bamar barrier but of course if you have a non GUI application you probably want to have a GUI application to represent the results of your app and so we've had to do 32 to 64 bit API interoperability so we have done various different shared memory IPC kinds of communication parts between the apps with different api's file system access control lists fundamentally the different ranges that we've had to do is to centralize the credential checking for authorization to access to to operations basically eliminating the UID GID comparisons that you've seen sprinkled throughout the system we have new kernel interfaces for these checks that are I hesitate to say capability based but because it's not a capability system but effectively given an object and your credentials this is the operation that you'd like to apply to this object do I have permission to do so or do I have the authorization to do so and we've also enabled native file system support for for ackles through changing the v node interfaces so given the changes to the kernel that we'd made what are the changes that are that you will have to do for kernel programming for your case so we discussed this last year we've added a whole set of formal kernel programming interfaces in Tiger IO kit already had its own set of formal interfaces so IO kid cakes are basically unaffected by this the KX types that we're supporting in the bsd side of things are file systems and network extensions and filters there's probably a few more details but that's fundamentally the the set of kicks that we're supporting through these KPIs one of the big things is that there's no more open and indiscriminate access to all of kernel code and data being supported by these formal interfaces now we understand that there are some kicks for various reasons might want to get access to the whole of the kernel we would advise that not to be in product code simply because these kicks if they do get at kernel access or access to the whole of kernel symbols will be tied to exactly one kernel version so whenever we update the kernel those cakes will not load however if you use the kpi's you're guaranteed release to release binary compatibility and of course source compatibility so what are the kpi's do they basically encapsulate all the kernel objects that have one to be accessed by kernel extensions we've made all the data structures a Paik so that you don't have to go and access the the data structures directly which means that these data structures are now invisible to you and we can change their organization we can add fields and do whatever however you can still get access to the information contained with in these data structure structures by using accesses perhaps more importantly is that the locking that we've implemented implemented for synchronization for these various objects is completely hidden behind the ape or the KPIs all the objects are reference counted so when you access an object in the kernel you have to formally look it up basically do the operations that you wish to and then release it not doing this will cause leakage in reference accounting that will create that that will cause problems and bugs in the system later on so basically what I'm saying is that the semantic model for a lot of the objects within the data structures in the kernel has changed because of this reference counting and because of locking so don't be fooled into thinking that you can just take you know the Vino pointer that you used to have and do all the operations that used to have just sort of recast the the pointers that you were getting back from the lookups because this won't work we've also gone through and changed names of various interfaces data structures within the kernel and we've done this intentionally it's not just to annoy people it's basically to give you a strong signal that the semantics of that object has changed and you have to relook at that code and change your code to reflect the semantics of those objects as I was mentioning before you can actually use all of the kernel and if you don't use those KPIs and you decide to go off and get a hold of hold of a data structure that is encapsulated within the KPI you can do that however you must understand and correctly honor all of kernel locking and the changes that we make to that over time so this is really not a great idea however using the KPIs the kernel implementation of those objects it does not affect you and so therefore whatever we do will still remain compatible for you so the other thing that is going to affect you is that we've removed the funnel significantly from our code that the finalists in the seed is certainly still there there's large parts of the system that are still protected by it but that's being removed as we progress so you should be aware the fact that what you're seeing is is development work in mid-cycle but the goal is to remove the funnel from app from Apple code before we ship tiger what this means is that the kernel is no longer single threaded and as it passes through kernel extension code there could be multiple threads passing through this now so your kicks need to protect their own data if you have data inside it you have to be aware the fact that there could be multiple threads trying to access this and so therefore if you need any atomicity you're going to have to do this and to do that we have a rich set of synchronization primitives spin locks mutexes reader/writer locks and some atomic operations for you to use by and large we think that the mutex is actually going to be the one that you're going to be using most we also recognize that you know it might be difficult for some kicks to become multi-threaded its multi-threading is sort of supporting SNP and locking is actually not a trivial exercise as we found out so if there may be times where you actually want to be able to load and still be protected by the funnel and so we do have single threaded funnel executions supported for example in as you pass through the vfv node layer and going into the file system we will take a a funnel operation that will allow your code to be single threaded so because we've added all these lovely synchronization primitives for you to use it would be rather unfair not to to help you debug them because locking is a tricky business so we've added a whole bunch of things to help you debug your code in the face of deadlocks and so forth so we have a bunch of things for lock debugging for mutexes as I said we expect that mostly you'll be using mutex is in in your in your code so for for mutex locks you can get the current owner for the mutex and if you've got a read/write a lock if it's held exclusive you'll get the owner for that the PC of the last time that the lock was was held so you know where in your code it was that lock was taken and who it was that took it as in the thread also there's an option where you can actually get the law that the back trace of that last lock for 8 frames this will indicate the path that you got down there because frequently you know you've got you've taken the lock in some sort of generic function you have no idea who it was that's called it and it was called in you know one of 40 different places so we give you the back trace option to find out exactly the path that you took to get to that log we also have some runtime checks to help you get to help prevent you from getting into trouble we assert that you actually own the lock when you unlock it we don't allow one thread over here to go off and lock something and then effectively hand the object off and be unlocked by somebody else that's generally considered to be very bad style and leads to a lot of almost undue Bugaboo situations and so we just assert that you have to own the lock to unlock it if you've taken the lock exclusive we detect that you've just taken it again that's almost certainly not what you really meant to do because that implies that there's two code paths that want to use this lock in an exclusive mode at the same time even if it's the same thread and so we detect that we make sure that in the interfaces that require synchronization our locks that you've actually passed the right lock type in and you're not getting confused and giving it simple like we're a mutex was needed and this one is is again a stylistic thing in some respects that we if you take a lock and you need to go to sleep we assert that you actually don't hold any resources that that prevent you from going to sleep in particular that means another lock basically it's if you're going to sleep for long times in the system then you shouldn't be holding kernel resources if you you know if you're waiting for i/o or something of that nature if it's a non bounded lock sleep and you shouldn't be holding kernel resources because that will lead to deadlock we also have some log statistics being kept I my lights go off here is it working okay sorry about it so long statistics sorry van so how long the lock has been held for average and maximum lengths of time clear lead over the whole locks for very large periods of time because that will increase the likelihood of contention we also count how many times a lot was contended and when we when you take mutexes to be sort of like hang around and look if there's the lock holder is still active on another CPU so [Music] if the lock is held by somebody on another CPU with them we don't immediately go to sleep you sort of wait for a bit just to see if they're gonna be holding the lock for a short period of time and then you'll immediately jump in as soon as they release it so we differentiate between contention and and actually sleeping on the log so there's a bunch of stuff that that will help you be able to find your way through the SMP morass so for 64 bits in the user in the kernel we thought about how best to address this issue the fact that you've got two different kinds of address spaces out there so we decided that that we would represent user address by addresses in the kernel as 64 bits regardless of whether it's a 64 bit process or a 32 bit process so the kernel data type for a user address is 64 bits whenever you actually access the user address space the access is needed that that you use to access it have been changed their signature has changed because of this so if you use things like copy in and copy out or UI or move then those interfaces have also changed and also the ironn memory descriptor has been extended for 64 bits this really only affects you if you're creating one as opposed to just handing one around and so fundamentally it normally only affects the user client providers so for filesystem ackles as i mentioned earlier on we have a new authorization model with and have eliminated all the UID GID comparisons in the kernel and have used this new centralized credential manager sorry centralized authorization service called ko in addition we have new credential management interfaces you can't cons up your own credentials and hand them off to the system because these things are now all reference counted so you have to create them and manipulate them through KPIs we have as I mentioned this sort of authorization model which is a capability checking interface am i capable of performing this operation on this object and this is actually extendable we have pluggable authorization policies that you can insert into what are called scopes which gives you the ability to contribute to the decision-making as to whether somebody can perform this operation on this set of objects this was covered in the the file system ACL talked yesterday and I hope people cause it but failing that if you would like to catch that on your DVDs that'll be explained in a lot more detail so we also have added native support for faster stoom a CLS and if you your file system does support Ackles then there are new v8 VN ops to be able to plug into to be able to export those four file systems that don't support native ackles if they do support extended attributes we have new VN ops 4 to support extended attributes natively and if you have those then we actually store the ACLs in the extended attributes if you your file system doesn't support either then we store extended attributes in you know Apple double files the dot underbar files and then ACLs are then stored in the extent of attributes in that way so even if you even if you don't have any of these features will still expose them to the user by smoke and mirrors basic but this gives you the opportunity of taking existing file systems from other UNIX systems that already have these advanced features and now being able to support them natively within OS 10 so at this point I'd like to welcome Dean Reese who's the manager of the IO ket team to talk about how access to kernel interfaces is managed within the system good afternoon okay so first I'd like to talk a little bit about some extras that you're getting with your I guess it's available on the website the Apple Developer connection website content for this page for this for this session we've got several interesting new tidbits for you to help with your text development for Tiger in no particular order we've got a plugin for firewire debugging basically this allows you to do the two machine gdb debugging through a firewire connection functionally it's similar to the Ethernet debugging that you can already do it's got a few advantages one being that it's available much earlier during boot and sleep-wake so if you're trying to debug a sleep wake problem you have earlier access also we've got a new command called hexam for gdb if you're connected to another machine and you have kernel extensions loaded and you like to generate symbols for those rather than having to generate them manually you can run XM and point it at a folder on your local machine that has all the has all the binaries yeah we uh we definitely appreciate having this internally as well so basically it will generate symbols for all loaded kernel extensions in a single command we've also got a Perl script a little utility calk extract now this is something it's a bit of an experiment but as Simon said earlier we have we have a situation where we have a lot of qex developers out here doing really cool development but we don't have a good understanding of what all you depend on what specific API is you're using and why so this is sort of the beginnings of a feedback mechanism and what kext rec does is you run it against your kernel extension and it basically extracts a whole lot of information basically it's looking at the pea list and it's looking at the undefined symbols in your binary and it formats that into a little text file that you can send to Apple by way of my way of radar web or bug web and we're gonna keep these in a database that will allow us to search anytime we want to make a change or we're just curious who's using an API we'll be able to find any kernel extension using it we'll be able to find trace it back to the developer who submitted it this is purely voluntary you know we're gonna be using this again it's a bit of an experiment it's not a certification program there's no guarantees made but this will be a good way for you to let us know what it is you're using it'll be a good way for us to find out and possibly contact you about upcoming changes so that I think this could be very useful so we've also got a new kernel available on the website this is probably the single largest piece of the download but the kernel that actually shipped on the tiger seed still has a few KPI bugs in it that we wanted to get fixed so that you could play around with it but it was a little late in the the tiger seed development cycle for us to be putting kernel changes in for something that wasn't critical path for the release so if you're going to be doing KPI developer qex development you're going to be using the new KPIs you should install this kernel you should only install this kernel on top of the WWDC seed and there's a readme in there that has and strict instructions for how to install it we've also got some documentation and examples in the image the examples are works in progress that two file systems that we have there WebDAV FS and ms-dos FS these are the source code for the actual binary that shipped on the tiger seed so you're looking at the same thing that we actually sent in to get built and as I said there works in progress these are these are not in their final form but they do work using the KPIs and it'll give you something to look at sort of a starting point we've also got a couple a couple other examples that are not shipping code they're just specific tight examples of a one of two KPIs about one for filters and one for socket one for a socket filter and one for an IP filter the documentation that's on the site is a collection of header doc and some RTF files sort of porting guides to help you one go through and actually examine the KPIs and their intent and their parameters and so on and the porting guide is a little bit more of a tutorial basically trying to walk you through the process and it was actually written as a experience that we went through doing porting our own file systems and and such to the KPIs obviously is this whole KPI process moves forward will make more information available but this should be enough to get you started ok to talk about kernel extensions a little bit we're we're making use of something called interface texts now fundamentally they're there they look and feel like any other kernel extension but there are a few special things about them we actually introduced this concept in Panther and you might remember me talking about them last year but we didn't really use them until now the best way to think of them is there a library mechanism for the KPIs the KPIs as you know it are built into the kernel so they don't really need a text but we're using the text mechanism as a way of managing and versioning the interfaces so they really only contain linkage they don't contain any implementation whatsoever the I'll actually show you some diagrams in that show you how the linkage works but fundamentally it's just a big symbol table wrapped in a text bundle so what will happen is in the past your kernel extensions linked against the kernel and for tiger and going forward they're gonna link against these interface texts instead and that'll have the advantage of course allowing us to have multiple versions of the same API set or same KPI set coexisting on the system and of course the text mechanism already has versioning built into it so that will allow you to express the version that you need and will automatically link up to the right thing so it really gives us a mechanism to control when we introduce a new KPI set how it evolves over its lifecycle and then when it comes time in the distant future to obsolete it it gives us a way to do that I like to think of it as sort of a conveyor belt of interfaces that you know we can set to be several years long but we'll introduce an interface and keep it around as long as it makes sense and is supportable alright so in our previous operating systems anytime you linked against any of these 5ive this OS bundle libraries or CF bundle identifiers you were getting the whole kernel now what we had done is we subdivided the kernel into four basic areas because we knew that we knew that there were these logical divisions within the codebase itself and the way people were using them but we didn't really create a linkage difference because hey you were linking against the kernel so the to talk about what each of them are a little bit I think they're it's fairly obvious if you look at the header structure which one of these an API or KPI is actually being supported by obviously anything that is part of the i/o kit header subtree is part of i/o kit itself and so comm Apple kernel i/o kit is the appropriate one there BSD and mock similarly have their own namespaces and their own header spaces Lib Kern is something we created as a common library for all of the things that the three other main areas of the kernel needed to share some of the the primitives Lipsy type stuff memory some memory moving copying stuff like that atomic operations so some of the things that you might think of as bsd ish like you know b copy is not part of bsd that's actually part of lib current and we'll be providing more guidance on exactly which one is we're coming up in information very soon okay so anyway for all versions of the kernel prior to 8.0 when you link against any of these you're getting the whole kernel so this is this is a diagram i actually have several animations coming up around this to kind of show you how this design is evolving but fundamentally we had the kernel with these four partitions inside of it logically being treated as a single thing and the kernel extension that you see here in orange is basically a pure i/o kit kext in that it only depends on Io kit and Lib Kern the Lib current is allowed because it's common to all now the problem comes in because you're linking against the whole kernel if you access a symbol that's considered part of BSD let's say copy n or maybe you're using the dev efest to create a device node to communicate with the utility those things are not part of Lib Kern or i/o kit and as a result you have an undeclared dependency there we can't detect the we can't detect it by looking at your P list because you haven't declared it but you're getting it because you're linking against the whole kernel so what we've done for Tiger is to basically take this entire collection of interfaces symbols and group it together into one interface kext we call it the compatibility text okay and all existing kernel extensions that link against existing versions of the kernel will link against this and tiger but what we've done is taking the implementation out of it and moved it off into a separate place so the kernel continues to be the implementation as it always has been but when you link against any of those four areas you're getting this compatibility text and of course all of its symbols are being resolved by the kernel now the new KPI is that the things Simon has talked about and other sessions at WWDC will be talking about they're introducing new interfaces in the kernel those in the the final version of tiger will not be in the compatibility text they will be in new interface libraries interface qex so again we have the same four we've changed the name to represent the fact that these are sustainable kernel programming interfaces so comm Apple kernel KPI you're not linking against the whole kernel you're linking against a single interface okay so in this case if you wanted to have a driver load against all of those you could but you would have to list bsd as an explicit reference otherwise your text would not load because of undefined symbols now another thing I need to point out is we really kind of have old world a new world here and the two coexist on the system fine but there's no crossover allowed in other words you can't write a new style text that then goes and tries to pick up one or two old symbols that have been deprecated you either need to run against the old symbols or you need to run against the new sets the other aspect of this is as Simon had mentioned earlier we have a lot of symbols that were exported that we believe people may be using we don't know for sure but their implementation energy through things that we don't believe kernel extensions should be getting at and in a lot of cases they cannot continue to exist changes to the architecture of the kernel semantic changes make it impossible and so what's happened is with Tiger a certain number of interfaces the symbols that have been there in the past are no longer there in the compatibility text that will break some drivers right there the the failure mode that you'll see if you encounter this is an undefined symbol when you try to load your kernel extension and the only way to remedy this is to fix your code so that it doesn't need that interface or to port over to the new kpis which would obviously we would prefer and as the system evolves we have made a commitment to continue supporting the KPIs but you can expect as we vary further and further from the the older kernel we're going to have more and more breakage of interfaces that can't there were never sustainable in the first place so you're gonna see those symbols starting to slowly erode but again we're not doing this because we want to force people onto new interfaces we're only taking them out because we have to for performance enhancements for new features and so on so we we obviously want to provide you a way to move your product forward and that would be the KPIs okay so to sum up as I had said will only remove the symbols that we just read that we need to the whole point of a compatibility interface text is to be compatible right so if you have a kext that you've shipped previously we would like we'd like to see it be able to load against that the reality is if your text has dependency on BSD it probably won't if you're doing anything interesting with BSD you're probably hitting on a symbol that's that's no longer there and we'll need to move forward to the KPIs if you're writing an i/o kit text there's a very good chance that you won't break because you're probably not using the BSD innards in that case but again until we know exactly what symbols you're using it's very hard for us to tell okay so to help the users get through this transition what we've realized is we don't really have a good mechanism to talk to users about specific kernel extensions giving them a path name is probably not a very good user experience so what we've done is we've defined two new properties that we'd ask you to add into your kernel extensions that are OS bundle product name and basically it should be anything that you would use to identify your product if somebody calls your support line and says hey I've got your product what do you need to know to identify it we also realized that there there are cases where you have a whole suite of qex working together and from the customers perspective it's a single product well if we have problems with with two or three of those we really don't want to put three panels up to the user and say you know this one in this one or this one what we're gonna do is look at the examine these properties and if this property is identical in multiple qex we'll treat it as a single product and we will only talk to the user is it with it being a singular thing you know obviously we don't want to we're not going to try to we're not gonna scare the user we're not going to put panels up unless the product cannot be loaded for whatever reason but we'd like to be able to talk to them in a way that it's a little cleaner we also realize that you Matt you might have a fixed kernel extension available and we'd like to make it easy for the user to get at it so if you can add the OS bundle product URL which isn't again as an optional property then if we wind up presenting a UI to the user we might consider including this as a shortcut so that the user can go right to a downloads page or a product support page to be honest we don't know yet exactly how we're going to use these properties that we know that these are there to be presented to the user in dialogues will probably have Apple system profile or display them but we wanted to get this message out for WWDC and we don't have the the use nailed down yet but put it in now next time you ship the kernel extension make sure you've got these in there okay so there is as Simon pointed out the possibility that you do need to get at some of the kernel internals and you might have a legitimate reason for needing to do this we also have the legitimate need to say that you're getting at something that we can't support you know we can't provide release to release compatibility so we've we've provided sort of the compromise in that we allow you to continue to link against the whole kernel to link directly against the implementation when you do that however you have to request the kernel by an exacta version number and whenever we release a new kernel whether it be a new major release and you software update it could even be a security update the kernel version will change if we if there's a new kernel in that update and as a result your kernel extension will not load so you know by and large we really see this as being useful for internal development when you're trying to port something over you're maybe partly on the new KPIs and partly not so you want to get at some of the old symbols you can do that by linking against the kernel until you're ready to ship also maybe in an education environment where you're not shipping a product and also it possibly you know the one commercial product where we could see it being useful as a subscription product or maybe you're selling some sort of a security product that you expect to update with each major or with each OS update so you know we have that available as an option but we really recommend not using it for any commercial product you should not link against the whole kernel unless they're just really is no other recourse and I would also like to point out that we have the KPI feedback form we'll talk about that in a minute and if you do find that you have to link against the kernel you know maybe you should drop us a note and say why what is it that's missing that is required you to do that and you know we may be able to do something about that all right on an unrelated note I'd like to talk a little bit about IO kit and 64-bit as the i/o kit manager I wanted to make this information available we don't have a full Iowa kit session because the core of i/o kit is remained largely the same with tiger most of our energies have been focused elsewhere with the MP work and other things but the kernel virtual address space remains 32-bit in tiger so for i/o kit kernel extensions pointers are still 32 bits now drivers can get to greater than 4 gigabytes of memory just like they did in Panther updates that supported the g5 and you do that by calling io memory descriptor and it has various successors of course you need to call prepare and what that does is that creates a mapping between the physical memory which may be a large address and a what we call a bus a virtual address basically it's a 32-bit address that can be passed around to your device you can also for Pio mode you can get directly to read bytes and write bytes and that that actually reads physical memory directly so that's how you get it the physical 64-bit address space now for applications talking to i/o kit they can go through any of the POSIX API s that that have been made available as part of the 64-bit app support so you know standard open closed readwrite IO control so that'll work fine for networking interfaces for the storage interfaces and serial as well all of those use of POSIX what it will not support though is direct driver access through user clients and the reason here is that the i/o kit user client model and Lib Kern and everything else are about varying data directly between an app and i/o kit driver is all based on C F and C F is not being made available for 64-bit applications at this time so stay tuned we might you know we might be able to do this in the future but for now we're focusing more on the 64-bit apps that are working on large data sets and generally they're not also scanning the bus for USB devices so alright well with that I'd like to hand the podium back over to Simon to continue with the presentation thank you good job thank you very much Dean so I'd like to briefly touch on now is the lifecycle of a kernel interface so what are the objectives behind having KPIs well we want a set of well-defined interfaces and for that for Apple what that means is that we can focus on making sure that we guarantee binary compatibility from release to release so for developers what that means is that you now have a stable environment from release to release that you can program your solutions to we want to have a well understood path for change between the Apple and developers this allows Apple to plan for kernel evolution so that we know that we can change things we can we can make plans to add new features and we know how we can face this in and it also sends a signal to developers that they know when to my start to migrate from one interface to another and we'd like this to be predictable we are not going to change KPIs arbitrarily or gratuitously just because we happen to think that the third argument should go somewhere else or whatever it is the only reason to change a kernel programming interface is because something is forcing our hand because there's new hardware that is being shipped or you know the technology changes in Hardware all the time there may be new who knows processors memory architectures whatever that force us to revisit some interfaces and it's so these these KPIs will be stable from our perspective so we wanted a lifecycle that really didn't give you any surprises and so we've based the model on the API lifecycle for applications service so we basically have four modes of within the lifecycle start off in being supported which means that the interface is fully source and binary compatible from release to release this is going to be the state when Tiger ships of probably every single interface that we have in the KPI set if we find that that you know when we're looking forward in our our development you know we tend to be in engineering we are looking a couple of years out and we see that there's going to be problems in a given interface for because of hardware reasons or whatever then we will mark an interface is being deprecated what this means is that when you build your text the compiler will start to generate warnings on its use just letting you know that that you're using an interface that we've think is is we're going to be removing the inter the interface is still supported in terms of it will still work but we're just basically giving you a positive heads-up saying you know this interface is at risk and what we're reserving the right to do is to remove the interface in the following major release after that we're looking at obsoleting the interface at which point you will no longer be able to build your text using that interface will effectively remove the definitions from the header files and there'll be nothing for you to compile against but the binary can the binary compatibility is still assured you will still be able to take your text and use it on a system and then finally when we've got to the point where we've had to implement the given feature and we've had to change the kernel that the kick the interface will become unsupported which means that the implementation behind that that interface is no longer in the kernel and therefore the case can no longer low so what do we need you to do well initially we'd like you or just yeah in immediately we'd like you to adopt the OS bundle product name and the OS bundle product URL to identify your kext if we talk to the user the correlation between a kext name as we see it which is a path name or or something of that nature and the actual user perceived functionality that it provides is really not evident and so this is a way in which the user can get a sensible interaction between the system and your product actually so that they can be identified more reasonably we'd like you to download this session extra material it's on connector apple comm under set session 103 which is this session and look at all the stuff that we've put in there for you all the additional documentation but more specifically in the short term we would like you to run a extract on your text and to send the the report that k-x tracked generates for you to apple via radar web this will allow us to be able to see before we even get to Tiger what problems we may be creating and whether we can actually alleviate some some some problems for people so I do very strongly urge you to do that I think it will benefit both of us another very important thing to do is once you've downloaded the the extra material is to review the interfaces there is full headed dock for all the KPIs and you should bear in mind when you look at these is that these seed interfaces are evolving you know we're porting our own kernel extensions that as we speak we've done some we haven't done all so as we continue porting we'll we'll be spotting some limitations or some things that are missing or or awkwardness in some of the interfaces and so we will be changing things hopefully not radically but you know just fine-tuning as we go so please review those interfaces have a look at the examples that we have in those extras both the the file system ones and the networking ones depending on what your text is doing and see what changes we've had to make I apologize for some of the others we have fixed bugs in there so there'll be some extraneous changes not strictly related to KPIs but it should be pretty obvious what changes are are there for purpose for the purpose of using the well-defined interfaces versus not and finally well not finally then I'd like like to encourage you to actually try to port your text using the kpi's it's very important that that we understand your experience with these interfaces because we want to be able to ensure that it will be able to fully support your functionality and finally we would like you to give feedback there's the KPI - feedback at Apple comm mail alias which we are all sitting and watching and we would like to know your experience is both good and bad in other words if you found that moving your text over was simple then we'd really like to understand which sets of kicks were not having problems and if you find that there are some things missing or there's some awkwardness we would be very interested in that too because we'll do our best to make this experience as as easy as we can for you so more information the contact at Apple is Jason yo he's the Mac OS technology manager there's email please let him know of questions that you may have already feedback importantly for us in engineering the KPI feedback mail list I say I encourage you to to let us know your experiences with using the KPIs there's some additional documentation you can find on the DVD I was referring to the network kernel extensions API references as some 64-bit transition guides a kernel programming document and there's some additional documentation for kernel extensions this is on the ADC home website going down through darwin some descriptions of lib can and there's another document on on network kernel extensions you