WWDC2001 Session 116

Transcript

Kind: captions Language: en thank you and welcome to session 116 Mac OS file systems the Mac OS is supported multiple file systems for a long time now and now with the UNIX foundation in OS 10 we support whole plethora of file systems and to tell you all about those different file systems and and how you should be using them I like to introduce the manager of the core core OS file systems group clock Warner I'm really glad to see all of you here I know it's early and I appreciate your coming I hope we're gonna have some very useful information for you about the Mac OS 10 file system we'll start with this our welcome and a little idea of some of the stuff that we're going to cover today we're not going to talk as much about futures obviously you've probably heard that as a theme throughout the Developers Conference the present is so much more exciting this year and we're not going to talk as much about file system internals because we know that a lot of you are now in the process of bringing your apps over to Mac OS 10 so we're going to concentrate on some of the things that your apps need to be prepared for when they're using the file system as Jason mentioned we have a number of different file systems in Mac OS 10 and they do present some occasional wrinkles we also are going to give you some tips into how to improve the performance of your app especially with regard when it's to its use of the file system but we will talk a little bit about some of the key issues in building file systems yourself to add to Mac OS 10 if you went to the Darwin overview session or probably any of the core OS sessions you've seen this chart already this is the key you are here graphic if we run an airplane now you know we'd be saying this is the San Jose if you're going to Hawaii get off now we are part of the bsd kernel inside of the file system here's a blow-up of what the file system looks like internally we support basically at the core OS level the bsd berkeley standard software distribution system calls with some extensions inside of the file system there is a big switch we call the virtual file system layer which separates the part of the file system which is dependent on the underlying volume format or network protocol from the part of the file system which is independent so the stuff above the vs VFS layer is independent the stuff below is dependent and that's why you see our list of file systems below underneath the virtual file system switch u FS h FS NFS and we'll talk to you about all the various file systems that are supported in Mac OS 10 here's the outline of today's talk there'll be something of a status update basically an indication of the file systems that we ship we're gonna do a demo of a couple of the new ones we're going to talk only briefly about some of the different file system interfaces we've covered that a number of years and there are a lot of sessions describing the application frameworks for use in Mac OS 10 but we will spend a fair amount of time talking about the differences between the various file systems that Mac OS 10 supports and again how that might affect your application development we'll talk about security because that's been a large issue for a lot of folks Mac OS 10 is a multi-user system with perleval file permissions and in that way it differs greatly from Mac OS 9 and you need to be ready for permissions our errors potentially in your applications we're going to talk about performance considerations as I mentioned and we'll talk a little bit about building new file systems first status slide I think this is my third or fourth Worldwide Developers Conference session talking about the interesting things we were going to do in Mac OS 10 and the recent things we had done in interim build a B and so forth this is the first time I get to say this I am NOT above cheap applause this is why I love this crowd alright so Mac os10 we have three primary file systems and when we say primary what we mean is they're fully read and write file systems and we can boot and route off of them so we can boot and route off of the Mac OS extended format file system best known to those who know and love it as HFS+ we can boot in rudolph of ufs that the UNIX file system that we support based on the Berkeley fast file system and we can boot and Rudolph of NFS in fact we do that internally all the time for installs we also have some readwrite file systems that we don't boot and Rudolph of but are otherwise first-class citizens and that includes the Mac OS standard format otherwise known as HFS for legacy data that you may have on your Mac the Apple file protocol a client was delivered by the server team into Mac OS 10 we have support for an estas file system and specifically we mean fat16 and fat32 here which we did largely for digital cameras and so forth but also zip drives and removable media coming from your Doc's machines and WebDAV and we'll talk about web dev in more detail a little bit later on in the talk there are also some read-only file system supported in Mac OS 10 including ISO 9660 which is used on lots of CD formats especially to interchange between Mac and Windows we support the universal disk format actually I don't think it's call bet anymore I think they just use the initials UDF sort of like KFC and Kentucky Fried Chicken and CD da FS a file system written by the CPU software group to support CD audio files CD audio drives CD audio discs rather about music ISO 9660 I should mention we can also boot and root from I didn't mention in the primary file systems because it's not read and write so I'm going to do a demo I'd like to bring up demo machine number one and I'd also like to bring up my lovely and talented assistant Scott Roberts you might have seen a demo similar to this in the keynote avi got his digital camera up and took a picture of the audience I'm gonna have Scott do the same thing here thank you Scott now it turns out I'm not a vice president and as a result I don't I don't make as much money as avi - Vania and so I can't afford a really nice fancy digital camera that has USB hot plug and so forth and so on I have this camera that was given to me as a gift by a friend of mine actually a kodak DC 210 plus and but the nice thing ello doesn't have USB connectivity and all that it does have this little compact flash card that it uses for memory storage and we happen to have a SAN disc reader here attached to our demo machine and so I'm going to put this little card in here let's pray a little bit and you'll see the image come up on the system now we're doing something differently than avi did because we're not here to show you image capture or any of the fancy camera applications I just want you to know that this little compact flash card is formatted as an ms-dos disk and so we're looking at an ms-dos volume that's just loaded on our desktop and if I bring up the preview application and I go here to the image that Scott just took and open it up there's a picture of you folks I must have said something wrong I don't know what okay for our other demo I wanted to show you the web dev file system I've got over here on my other demo machine I pre-configured it as a web server and I actually changed the configuration file so that it would run the davit actually ships in Mac OS 10 as part of our Apache installation so host 2.1 62 is running web dev and i'm going to bring up internet explorer here which I should have pre-launch sorry to waste your folks this time and I'm gonna type in oops the URL and you can see we have our little Apache page now what I'm going to do is since I'm bored with looking at things in web browsers I'm going to mount this particular website as if it were a volume on my desktop I do that by hitting command K going to the connect to server dialog and typing in the URL here instead and now you notice I now have the Apache web site actually mounted as a volume of my desktop well why is that interesting let me show you one thing that we can do we took a movie actually last year's file system session and we edited it and made a small movie out of it and we put it on the web server and now instead of going to the browser and trying to deal with the QuickTime plugins and so forth I'm just gonna double click that movie and we'll show it for you now oops let me back it up a little bit here thank you the stage police are watching me and I've heard from my spies that when I turn around to jump on the stage something Bad's gonna happen to me I have to be careful I've noticed that the stage is higher this year than last so I think I'm ready I'm getting older I can't do this kind of stuff every year okay let me show you one more thing that I think is pretty interesting I'm gonna go into BBEdit here now I haven't actually completely rehearsed this down I was fooling around with it before the show but I thought it would be kind of an interesting thing to do so I'm gonna bring up BBEdit and I'm gonna open up a file on the WebDAV server and it's going to be the english HTML index file and I see some text here that says let's see if you can see this it means the installation of the Apache server went okay I don't like that text the text I like is file systems rule I like that much better now I'm going to go to the web server here and refresh and you can see file systems rule up here in the top so an interesting way to edit your website is to enable dev on it and then you can just use your favorite editing tools and do whatever you like inside your favorite editing tool when you save it's automatically populated right back up on the server you know I had to have a plant for that for that applause okay let me talk just briefly about some of the file system interfaces in Mac OS 10 this is the little file system interface chart you can see the three major application environments classic carbon and cocoa on the top they all have their own file manager or file object interfaces and but they all come through the BSD layer and we've extended the BSD layer to allow access to filesystem metadata that's not typically available in UNIX especially catalog information finder and create a file type and creator type and things that you find on HFS+ and underneath all of that is our virtual file systems which I mentioned before and that's where additional file systems that you might develop in Mac OS 10 or stacks would layer into our system so I won't go again it's detail here but suffice it to say the carbon file manager is still available to you it has the mac OS interface file system interface is carried forward there's access by volume reference number directory and ID still available and carbon does an interesting thing Carbon provides HFS style semantics on file systems that don't normally support it so if you're using ufs but your app is using carbon you'll still see resource Forks and you'll still see file IDs the file IDs don't last across mounts they're they're done in memory and kept for you only while the mount exists but nonetheless they're present if you do a call and you want to see them so you're somewhat insulated if you're using carbon from some of the differences across file systems we also of course have the cocoa environment with an object-oriented API and some file objects that you can use and as I mentioned the Berkeley UNIX interfaces are available and can be used by your app as well even in a mixed environment with the other application frameworks we want to just briefly mention a couple of the new calls that we added to the BSD layer for accessing HFS file metadata and sorts of things that aren't typically available under UNIX two major ones get a tour list and set out our list which roughly stand for get attribute list and set attribute list these are flexible calls designed to retrieve various types of metadata various different formats back to your application normally you wouldn't call these you'd probably call get cat info said CAD info and carbon but they're available to you if you need to they are in our system we also implemented a call called search FS which whose job is to do fast catalog searching it's supported in HFS+ although most file systems don't support it but it does allow for fast catalog searching and it was designed to support the PD cat search functionality and carbon we also have a call for exchanging data between files those of you who've done carbon apps before are probably familiar with the exchange files call we created a BST level call called exchange data to do that same atomic transfer of data between two files and finally we've added some options to F control which is a standard UNIX file system call but we have some extensions to allow behavior like allocating storage in advance of the elio F those data that's not part of your file per se but is part of the file allocated storage which can be extended later without additional allocation now we're at the file system differences part of the talk and I'm gonna grab a little cup of water here because this is gonna be a little longer and this stuff is important there are a station mentioned many file system supported Mac OS 10 and while Mac OS 10 tries to be as file system agnostic as possible the underlying volume formats and the underlying network protocols do impact the behavior of file systems and in some ways it's just unavoidable and your app may need to be ready for it so we wanted to give you an idea of some of the differences that you'll need to watch out for kind of a set kind of a you know warning about some stuff the first set our naming differences our file systems often behave differently with regard to name and two of the biggest issues are the name lengths that the file system will support and their whether or not they are case sensitive or case insensitive as a lot of you know HFS+ is a case insensitive file system the Mac has historically worked that way and people are accustomed to it but the UNIX file system is historically case sensitive and people who are used to using a UNIX environment are accustomed to that and we wanted people to have an option in what kind of file system that they supported because some of these differences are actually valued by people especially ufs for example supports holes so you can have very large files with not so much data storage and it doesn't take up much room on your volume HFS+ doesn't that maybe Han hfs+ supports full unicode characters anyway well we'll talk about some of those differences as we go access by persistent ID or path all of our file systems support path based access some of our file systems also support access by identifier and HFS+ of course is among the file system that supports access by identifier we also have a call that allows for an error to be returned if you try to delete a file while it's held open deleting an open file has historically been a no-no on the Mac and it's historically returned an error however unlinking which is the delete call on UNIX a file typically does not return an error even if the file is open and we support that semantics but we wanted Carbon to be able to get an error back if someone tried to delete a file which was open because apps are expecting that behavior so we built a special call we nicknamed it delete I think it's system called 227 and it just checks for a reference and returns an error likewise some of our file systems full Unicode support some local bytes some utf-8 and some of our file system support permissions and some don't and what I mean by that is the volume format or the network protocol will have provisions for permissions data to be stored and or transmitted some file systems like HFS for example this dental Mac OS standard format have no storage for permissions information and so we will use default permissions for the entire volume and often substituting the logged in user as the owner for all of the files on that volume with a default permissions man some of our file systems support hard links which is the ability to create separate nodes in the file system pointing to the same data where the nodes are essentially equal not an alias from one to the actual object some of our file systems have storage for catalog data catalog information some of them don't some of our file systems are multi-fork some of them are I wanted to talk to you about web dev in particular because web dev is a really good way of highlighting some of the odd differences you might see between file systems it's interesting that we have this sort of file so the Magnox tech architecture because it allows us to do things like mount to web server as a file system but there are going to be some gotchas the WebDAV protocol does not have any sort of dates besides modification dates so if you wish to get the access date of a file on the server it's actually impossible we have to sort of fake that information so that's a one classic difference inode numbers aren't a concept that is supported in the WebDAV protocol either webdev I should mention stands for web-based distributed authoring and versioning the reason it came into existence was to allow collaborative authoring on the web I think it's in visual it's a sorry original envisioning was for people who are doing development in a web-based authoring tool to be able to move things to and from the server and affect the files on the server but it wasn't originally a file systems protocol that's something Apple decided we could do because the protocol added enough support in the way of a consistent hierarchical namespace some synchronization with locking and property management but those properties do not include inode numbers and so very much like carbon and file IDs when we see a file in WebDAV we generate an inode number and we remember that inode number for the life of your Mount but if you unmount the web dev volume and mount it again the inode numbers for files will not be the same also we can't set live properties in WebDAV through the protocol live properties are the ones that are actually supported by the server there's also a notion of dead properties in web dev which are you can make up a property and store value in it but real live properties like for example the access time those those properties are actually on the side of the modification time those properties are actually on the server and we want to respect those and return those to you but the server's will not let us change them so you can't do a set mod time on web dev and have it work we'll have that silently failed to keep apps from killing over but be advised that you can see a silent failure of a set mod time the security model for web dev is entirely different from what you'd expect from a file system it's HTTP security it's basic authentication for us generally which means that you try a request and if the server doesn't like you it gives you back a message that says authorization denied and it's your job to find out the users username and password and try again sending it across but there is no way except for testing an application to determine if the user is going to be able to do that operation or not if you send a put across to the server which is the mechanism for taking a file and moving it up you don't know if it's gonna succeed or fail there isn't a pre-flight call to fuel you just have to do a put so what happens in WebDAV is if we get an authorization error the daemon that supports the system puts up a dialog box that says who are you please type your user name and password in and then we'll send that across to the server we'll keep doing that until the user either gets it right it times out after about five minutes or the user hits cancel if the user hits cancel Annie access error comes back from the filesystem likewise unlike AFP or NFS which are typically run over local area networks and are therefore usually reasonably fast maybe not compared to local Williams but in the absolute sense WebDAV can be quite slow it may be running over 28k modem link you never know because we're talking about an Internet file system after all so we're not going to go through all of the items on this chart I just wanted to scare you a little bit to give you an idea of how some of these file systems actually differ classic example hfs+ supports privileges it has storage in the volume format for privileged information supports the ability to get an error back when you delete an open file supports access by ID ms-dos supports none of these things WebDAV supports none of these things either NFS is sort of a mix same story with naming differences some file systems are case-sensitive like u FS some are case insensitive like h FS plus some support unicode fully and some don't we make a particular mention of the Unicode characteristic hfs+ supports unicode names in a canonical form where the characters are decomposed it's possible in unicode to have say an e with an accent represented as u with an accent as one character or e followed by an accent character there aren't that many decomposed characters but there are some we always store all our filenames decomposed on the volume so that there will only be one element in a directory that looks the same to the user and so that will easily be able to do name comparisons but not all file systems do this ufs does not ufs does not interpret the bits and we wind up storing things on ufs as utf-8 characters and so on you FS you actually could have two names that look identical to the user but are actually slightly different in their byte representation what this means for your application though is if you have a composed character in a name that you send to a create call when you look at it again through a directory listing at HFS+ it's going to be different and on you FS will be precisely the same so having said that let me give you some tips as to how you can handle some of the specific differences we've talked about today in your application number one be consistent in your use of case internally we found with the Macintosh Applications environment which was a Mac environment that ran on UNIX operating systems that lots of Mac apps would keel over when ma II was running on a case sensitive file system and that characteristic bled through reason being they would have files like preferences that they would open in one part of their app with a capital P and they would open in another part of their app with a small P and they wouldn't be the same and that would confuse the app HFS+ they would be the same ufs they wouldn't be you'd either see a different file or you get it not exist error when you tried to access the file but different in case so make sure that in your app when you're referencing a hard-coded file if you do that that you're using the same case in all instances always use D composed names in your application that way you'll never be surprised by a name being slightly different on the way back out than it was on the way into the file system when you created it be prepared for access errors at random times as I mentioned the web dev file system has a very odd permissions model and we're gonna do things like put a file across which may happen in an eff sink or a flush files operation or on a closed operation and we're gonna discover at that time and not in advance that the operation isn't permitted if the user doesn't have a username and password that allows them to do it or if the server administrator has cut their access your you can get back any access error on a close call or on a flush files call with something which probably hasn't ever happened to you before so be prepared for access errors at strange times and you may want to be able to put up a dialog that says access denied you're going to probably see them if you're running a Carbon app as an AFP permissions error because that's how Carbon maps are access denied error codes also do not rely on inode numbers and this is also true for file IDs across mounts if file system is unmounted and remounted all the inode storage on a web dev volume all the file ID stores on ufs volume is kept in memory in big tables and we'd there's no effort made to make that persistent across different mounts we have one way for you to help you deal with some of these differences and that is the path comps simple path system call path comp is designed to give you characteristic information about the file system you're running on it just takes a path and you give it a selector that says you'd like to know if the file system is case-sensitive or if the file systems support how long the names are that it supports those are the only two selectors that we support right now in HFS+ on path conf perhaps someday we'll expand that list but you can use this call especially in HFS+ to know that you're on a case insensitive file system so I'm going to now bring up Pat Dirk's the core OS file systems technical lead to talk to you about some security issues and some performance issues in Mac OS 10 thanks Ari well the first thing to realize is that in Mac OS 10 it's a whole new world it is multi-user to the core their permissions everywhere and the core kernel enforces those permissions there's no path around it or some access path that's gonna be different that's not going to be affected by it the whole system is fundamentally multi-user the permissions in the system are the standard UNIX permissions if you're familiar with those the next few slides are just going to be review for you hold on there are a few gotchas in the permission handling of HFS but if you are familiar with UNIX you should be very comfortable with the permissions model on our system and we'll see that diagram again in a moment the permissions for those of you who may not be familiar with UNIX are in some ways similar to Apple share Apple shares permission when we were designing the AFP protocol are based on the UNIX permissions model and we made a few changes from there to allow them to work on a on a folder only basis but they're fundamentally inspired by the UNIX model so instead of sea-fowl see folders and make changes you have read write and execute and you have that for files and you have that for directories the catch is it applies to files as well as folders on Apple share file server the only permissions you ever had to worry about was the permissions you had on a given folder in Mac OS 10 you have to worry about the permissions on individual files as well and the other differences with AFP is that in AFP you can have separate permissions for the world the group that a particular folder and the owner of the folder and whichever category you fit in you got those rights so everybody started out with the everyone permissions and then if you were part of the group you also got the groups Commission's and if you were the owner of the object you also got the owners permissions in Mac os10 there's only one group that is matched to if you're the owner you get exactly the owners permissions and if you're in the group you get exactly the group's permissions and if you're everyone else then you get the other permissions so those UNIX permissions consists of an owner ID that is saved with the object a group ID that is saved with the object a set of permission bits which you see there and a few extra bits that are not divided up in separate categories for owner group and other and we'll cover in a moment what exactly those bits mean in different cases but that's basically read/write/execute in three groups and three special bits and some flags that we'll cover in a moment as well so so there every user is categorized in one of three possible groups either the owner of the object the group that there is associated with the object or everybody else and whichever group is most specific determines the access as you get as I said I can unlike Apple share so for each group there is read write and execute write translates very directly to make make changes in AFP read is the right to read a file or list the contents of a directory it's a bit like C file see folders that's where it gets a little weird and execute applies most directly to files obviously if it's an executable and you have execute permissions then you can execute it it also applies to directories as we'll see in a way which is kind of a subtle case for files if you set me set UID bit then when you execute that binary and it only makes sense on executables the program will run with the ID of the owner of that file so you'll commonly hear set UID root binaries those are files that are placed on the system that will run as a privileged user in the system when they're executed and there's also said GID which is used less often which runs with the group that is associated with the object now in directories it's probably easier to make sense of these permissions if you think of directories as files listing the contents of the directory because that's how they were originally came about read is explicitly the permission to enumerate the contents of the directory to type LS and list the contents right is the right to make changes it's very analogous to a FPS make changes this is make changes to the directory so these are operations that would require changes in the directory file creating a new file renaming a file that's in there deleting a file that sort of thing you can set execute on a directory and that limits access a little bit without read normally you have read and execute together if you give execute without read you retain the ability to open files in there provided you have permissions on the file itself but you lose the ability to list the directory contents so it's almost sort of a security by obscurity if you know what the file name is or your program has built in you know some file that it's referencing execute is enough to get it open but you need to read in order to enumerate the contents and look at it so that's that's sort of an edge case that you may run into and it's it's unusual finally setting the sticky bit one of the special bits that is associated with an object means you can give right but the ability to actually make the changes is limited to the owner of the object itself so it imposes one additional test before you can actually make use of the write permission that you would otherwise look-look to be granted now there are a special group of flags associated with every object as well and these exist only in one form there's not a separate group of flags for own your group and others there's only one set of flags and most common one you'll see is the immutable flag that takes the place of the lock bit that HFS always had and that is in fact what set F lock and reset F lock in the carbon interfaces news to lock or unlock a particular file you can see these flags if you use the - o option in LS if you find yourself in the shell it will list you change if you if the immutable bit is set so that's a quick way that you can tell if things are locked and there's a chi Flags command that changes the flags and there's at your flag system call that will manipulate them at the BST level all these things are accessible at the bsd level there's nothing in carbon or cocoa or something that is that a special above and beyond this everything is enforced at the BSD level and everything is accessible at the BSD level the other gotcha you may run into is that when something has been marked immutable it can't be moved that used to not be true on Mac OS you can lock a file and still take it and move it somewhere on servers that was actually sort of an awkward thing to do because you could lock something down and somebody who had to make changes to make it disappear from you that's no longer the case when something is immutable it doesn't go anywhere and it doesn't change now these flags have the immutable and the append Flags have special variants of them that can be set only if you are especially privileged user and it can't be unset in the normal running of the system so if you are trying to protect some particularly important file in the running of the system you can set a special system only immutable bit that is sort of stronger even than the regular immutable bit and that you can't turn it off so be careful if you try this on your machine at home you have to take the system down to single user before you can clear that bit know all the UNIX aficionados wake up this is the part where things get different again there is some special handling on permissions for each of us plus volumes we had a problem in that we wanted people to be able to take discs and move them all around from system to system and retain the same ease of use that they had in Mac OS 9 they could take a zip disk from your system take it over to somebody else's system and you wouldn't suddenly find that the permissions were all wacky just because the numbers that were assigned for the user ID and group ID on your system made no sense on the other system so the system very carefully uses the permissions only on those discs that it knows are local or were specifically requested that they be used by default if you have an H of s plus disk that the system has never seen and you connected either by plugging in a zip disk or by plugging in a firewire drive even the permissions will fall back to a scheme where the owner and the group are ignored and you can get that same behavior on request for any system for any disk in the system through the finders ignore permission as a bit I'll talk about that so every every disk is identified not by name but by a special 64 bit identifier that we write on there when a disk is being mounted the HFS code checks to see if this ID is one of a disc that it has seen before and for whom permissions should be enabled and if it finds that then it will enable the permissions and it will be used just like you would see a you of fastest you'll see owners you'll see groups you'll see everything if there is no entry for that system if it's completely unknown or if the entry in there says the user asks that the permissions not be used then the handling switches over to ignore all the user and group IDs on there make them unknown and replace the owner with the login user and that's done completely dynamically if you have such a disconnected you log out somebody else logs in they are now the owner of all the objects in that system so it's not a static mapping it's whoever is currently logged in owns all that so it's a very convenient way to not trip over user or group settings that make no sense on your system so the ignore permissions checkbox in the finder lets you elect to ignore this and get the same sort of foreign disk behavior that you get and it's the same underlying mechanism what the ignore permissions bit does is basically turn off the recognition of that disk in the system and it will treated without regard for the users and groups it's called ignore permissions really the best way to think of it is to think of it as ignore ownership we'll take questions and answer questions on this later I wanted to bring up a few points about performance in the system and in particular the different ways that you can do I owe a few general words that will touch in a moment but I always say we want to cover the differences between doing buffered filesystem i/o doing direct memory mapped i/o in the system and using unbuffered filesystem IO and the differences between them and the implications of those things I'll say a few words about zero fill which your application may have run into which is something that you see on Mac OS 10 that you never saw on Mac OS 9 before in general this shouldn't be news to anybody the few where iOS you do the better the more you can aggregate Jerry OS into a few large operations the faster things will go even if you're doing small transfers the system will try to aggregate these on your behalf if you're sequentially reading through a file the system will pick up on that and it will read larger and larger chunks even ahead of where you currently are and as you're writing it will it will save up rights to do single large writes out to the disk to maximize the efficiency of your i/o so that is why sequential operations are so much better than random operations because even if your application is only ever asking for 4k at a time you'll be doing very large transfers to the disk and the zero fill that we'll cover in a moment is triggered when you are leaving areas of the file unwritten but you do become the owner of them and it's best to avoid that because it's really just wasted effort you might as well write the data sequentially don't skip ahead of the end of file for instance so this is basic buffered file system i/o you see the device driver in the system which is doing the actual data transfer the buffer cache and the virtual memory system which are part of the kernel that govern all the data in the system and in Mac OS 10 those are actually integrated and they coordinate with each other so where there are cases where a particular piece of a file or page in the system is the same the buffer cache and the virtual memory system coordinate access to that page so there's only ever one copy of the page that means if you have something mapped and you write that you'll see those changes in the mapping right away and vice versa if you if you if something gets paged out the readwrite path will see that right away it seems obvious in hindsight but it's an awful lot of work to make that work correctly and finally there's a user application drawn there and you see the user application with a page of memory and there that's really the appearance of a page that has managed on the user's behalf in the virtual memory system you can think of it either way you can think of the page as owned by the user or you can think of the page is managed by VM on the user's behalf it's really all the same thing so in basic buffered i/o the data is first copied from the device driver into a buffer that is set aside by the file system to hold blocks for that V node and those blocks are shared between VM and the buffer cache as necessary so that's the first copy then the filesystem will copy whatever the user asked for to be copied either read or written into the user application and there's a second copy made into the user pages that hold the user buffer so it's completely flexible you can read at any offset in the file you can read any amount of data but you do end up making two copies first a large page aligned copy for the convenience of the system into the buffer cache and then a separate copy from there into your user address space where the data really lives and by the way the user page ends up being dirtied as a result and we'll see in a moment why that's important but if you're going to be reading a file over and over or you're reading back and forth through a file it's a cost that's well worth making in addition to the flexibility you gain from the ability to align the data or the size anywhere you want the fact that the copy remains in the buffer cache means the next time you hit either in that same page or somewhere right around there you'll probably find it in memory and you'll only end up doing the last copy from the buffer cache into the user page so there's an extra copy but it may be worth it under some circumstances and this is probably what most of you are always here I always like this is ordinary open closed readwrite i/o now instead of that you can do memory mapped i/o and it's something that you should consider as an option when you're just reading files it's a very efficient way to get data in and are some advantages although it requires a B as the VM call to set it up so it may be tricky to do from a CFM Carbon application it's a very nice way to get the data in because you only end up doing a single copy essentially it goes straight from the device driver into the VM system and from there it's visible to you user application so there's only one copy mate so we save a copy in addition the VM page is not marked dirty all that's ever happened to that page is something was read into it unlike the user page that was copied into a moment ago so when the system needs more pages it doesn't have to go copying that user page out to swap storage it's all set it can just throw this away and I can read it in later if it should get page faulted in again so there are some disadvantages every transfer is at least a whole page worth so if you've got a file with ten bytes of data in it that's obviously not worth mapping it but it's a nice way to get some data in and read through a lot of data the VM system does the same clustering of i/o operations that I mentioned earlier as you're touching through pages it will start paging in more and more in advance so it's a very good way to read in a sizable data file that you're just going to read sequentially it's not good for right because you can't extend the file by mapping it but it's a very good way to read data in that you're only going to read and that you're reading sequentially and it'll save you a copy now finally there's something that's almost a mixture of the two you can choose to do unbuffered IO and it's actually very easy from carbon because it's the exact I Oh pause mode no cash bit that you can set on a reader right transferring carbon and if carbon does the work to the system on your behalf if you're not using carbon you can make s control calls to enable this mode for your IO but it basically skips the intervening buffer cache altogether and the data travels directly from the device driver into your buffer now that imposes some of the limitations that the file system previously took on on your behalf on your application so the data has to be page aligned it has to be a multiple of pages but if you're you know reading in a QuickTime movie that you're going to play once and never necessarily touch again it would be a waste to fill up the buffer cache with all those pages and it's perfect to just read that indirectly that way you have total control over the amount that is transferred in a single transfer so if there's something about your data that you know that the system wouldn't know this is a very good way to do it if you need to grab a whole frame of data or for some reason know that 64 K is exactly the right size to transfer and you don't care until you have a full 64 K available this is the kind of i/o you should consider unlike memory mapping the page the page is dirtied just like ordinary i/o would be the page in the user space is marked dirty because it's been copied into and it will be swapped out if necessary but it's a good way to do i/o and not fill up the buffer cache if you're not likely to read or write to read the data again it's a good thing to do and you can write files this way so if you're writing an output file that your application isn't just about to reread this may be a good way to do your AO now the zero fill I mentioned the Mac OS 10 kernel tries to be very careful not to let you read data that you haven't previously written if you would call cases where some major word processing application would inadvertently shift pieces of your hard disk out along with your documents you'll see why this is a really nice feature you have to be careful though because if you have a file that you're writing randomly you'll end up if the first transfer is some distance into the file you'll end up 0 filling the whole intervening space basically anything that you can potentially read you should consider the you should either write where the file system will write with zeros on your behalf as part of the write transfer so those cases are basically where you you said EOF to make the file larger or where you do a write that skips ahead past the end of file some distance and starts a transfer there creating this gap that gap will be 0 filled so for those reasons sequentially writing a file aside from all the benefits I mentioned earlier of clustering the i/o is is far preferable no word about the cost of caching you should be careful when you decide to cache data in your application because in Mac OS 10 you are constantly running with virtual memory enabled and what you think of as setting aside some memory for this particular cache is really just that much more paged memory in fact you may end up doing a number of i/o operations just to read the data and you may have to page out some other dirty page in the system to free up a page for your cache you end up incurring the cost of the actual transfer to read the page in and if this day that turns out not to be referenced you may end up having to page out a page and you dirtied by this cache in addition you have to be very careful about how you structure the cache this is not wire to memory that's sitting there for your behalf if you have a cache data structure that is just laid out very conveniently in memory but ends up skipping around from this page to that sort of randomly you end up touching all these pages and you may end up doing page ins with every new page that you touch so you were very careful that you structure your cash in a way that minimizes the number of potential page hits to get you to your data altogether it's very easy for an application cache to become much more expensive than simply reading the data right back in from disk especially if the data is something that is mapped directly into memory for instance so think about it carefully and only use caches for things that are truly hard to reconstruct or where you are sure that the hit rate is actually very very high so finally Mac OS 10 is a good time to rethink some of the fund elías options behind your application think about the kind of data that you're reading and the pattern that you're reading or writing the data in and think about what mechanism you might best use to get that IO in and out of the system look at your application as it's happening and figure out where the real bottlenecks are before you decide where to spend your time and effort and trickiness and what to optimize if the bulk of your application is reading and writing files it's obviously worth thinking about if the bulk of the time is spent waiting for the user to click on some cell somewhere or something that it may not be an issue at all look at the underlying assumptions that went into your application because some of them may well be changed in Mac OS 10 some system calls that used to be almost free on Mac OS 9 because they came straight out of memory all the time may be reasonably expensive on Mac OS 10 all of a sudden and again that's the reason to go back and look at your application in action and see where the time is being spent because you may be surprised to find that you're spending a lot of time doing things that you assumed would be almost free and finally try to avoid making assumptions about how fast something will be to read because you might be surprised what's actually somewhere remote over on a network in somebody's home directory and the preference file you thought was cheap actually turns out to be a very lengthy operation that might involve automatically mounting some volume getting access to the data etc so don't make assumptions about what's fast what's local what's remote it could be on a web dev volume for all you know so finally there are some tools that you should look at there are some classic UNIX tools top is a very nice tool for seeing the size of your application the amount of virtual memory that it has allocated to it how much of that is shared how much of that is private and it gives you a little peek into the system and will show you how fast paging IO is being done how busy the system is what it's doing what in your system is using the most CPU time all kinds of things I recommend it highly you should you should run it off there's a time command which can be very interesting it's limited to command-line things but it will tell you how much system time and how much user time was spent executing this particular application so along with the number of i/os that were done on behalf of your application so you can easily tell when your application suddenly starts doing fewer reads or fewer IO transfers or more larger ones or smaller ones or whether the percentage of system time versus user time is interesting if the system is spending most of its time in system time you should think about what system calls it's doing to cause that to happen and similarly don't worry too much if most of the time is spent in the system because your applications algorithms may not be as relevant so time can be interesting sample is a gather a long standing next step tool it's it dynamically probes your running application and takes a peek at where the system is currently running and the stack at that time and you can tell it to take a number of samples over a certain period of time and it will tell you what percentage of time was spent in what routines and that may tell you where the hot spots in your application are tell you whether your application is constantly waiting for IO to come off disk or waiting for the user to do something or all sorts of things so sample is interesting and finally FS usage which you may have seen demoed in other sessions as well is a wonderful tool for getting down to the real nitty gritty of exactly what your application is doing and what the system is doing on behalf of your application you may be making carbon calls and be unaware of the number of system calls that go on under the covers to make that carbon API happen so at this point I like to bring up our resident expert in bad demo code my manager Clark Warner Thank You Pat all right let me pull down an explorer here and BBEdit window all right I'm gonna bring up a copy of TextEdit which you've many of you imagine it probably used by now and let me bring up a copy of the process viewer no I have to do this the are we all right okay first I'm going to do my little UNIX command here to find out the pin number of the process that is TextEdit and it looks like 278 let me change the font here to make this little bit more readable for you that's probably better okay this is not a command we want you to run too much at home I just made myself route what's that oh thank you okay now I'm now monitoring all the behavior of TextEdit and when I go back into whoops let me bring it back you can see as I click around various things are happening one of the things I'll do is open up a file that I put on our demo volume you can see a lot of things are happening now there's a 116 demos data file okay so here's my opening of the data file you'll notice there were a few page ins of some open some F stats some reads but basically one read call of a fairly large size so that's not too bad I'm gonna close up this file here let me just show you oops here we go again sorry you think I would have woken up by now the man page for SF FS usage one of the most interesting things about the FS usage program is the ability to see all of the actual carbon file system calls that are happening while the read calls are happening if you notice here there's this temp file tracing if you create in slash temp this file called file tracing then you will actually see all the carbon calls as well as all of the bsd calls that are coming through and to show you that briefly I'll turn that on and now I'm going to launch an application that I call dumb text dumb text is the standard simple text text editor hacked up to do one byte file reads just to give you an idea of if if you guys wrote after this way this is how you could figure it out before your boss does let me talk you a little bit about some of the key issues and building your own file system one is it's really not recommended and the reason we say that it's not because there's some you know fundamental thing wrong with the with the api's or anything like that but what the basic issues are kernel extensions like in Mac OS 9 have the ability to create kernel instability and while I know all of you right perfect apps that's not true for the people that don't come here and so we want you to bring the word back to them don't try to throw stuff into our kernel unless you absolutely positively have to and if you do you're gonna have to contact us building a file system extension requires deep internal knowledge of our kernel not just the VFS stack but also the various calls you might make to the kernel to make to use kernel services in your system and they change and we change as we change internals in the kernel think there are things in your file system that may have to change as well and so basically if you write a file system extension your rev lock to our kernel if you went to Deane rhesus talk you'll note he talked about kernel versioning if you don't version your kernel correctly in the future we won't load your file system extension if you don't version it at all in the future we won't load your file system extension so not only is there the implicit rev lock because kernel interfaces are changing internally and you may be using those interfaces but there's the explicit rev lock that if we know we've changed those kernel interfaces we will change the version of the kernel what's more which ain't will change the version of the kernel that that kernel is compatible with and your file system may not load I want to give you one of these changes that was made recently inside of our kernel something that happened between public beta and Mac OS 10 GM to give you an idea of what kinds of things were doing we wanted Mac OS tend to be a fully preemptable system the Mach microkernel was already fully preemptable but the bsd kernel was not historically in bsd when you make a system call you run all the way until you reach a voluntary yield point that is to say you do i oh you try to acquire a lock allocate memory and so forth or you'd run all the way to completion so we invented a mecca I'm called funnels to wrap all the BSD code so that that assumption of non preemption would would be held inside of the code but otherwise bsd system calls could be preempted funnels are required when a thread enters a system call they're released when the thread returns to user mode they're also released when a system call reaches a voluntary yield point like IO allocating memory and so forth but they're held across kernel preemption so a bsd system call now can be preempted in the kernel and something else can run a user thread or a mock thread or an i/o cue thread and so forth but the bsd structures won't change out from underneath the bsd kernel system call so it's happy we also split the funnel after we developed the first one so that now networking operations in the kernel are handled in the network funnel and all other operations including file system operations are handled in the kernel funnel we found that we actually could separate network activity in the kernel from filesystem activity in the kernel what this means though is if you are writing a network file system say every time you went to use the networking infrastructure in the kernel you'd have to change your funnel switch from the kernel funnel to the network funnel and when you went back you'd have to switch back and switching funnels is a blocking call and so the entire world can change from out from under you when you switch from the network funnel of the kernel funnel and vice versa all things you would have to know network funnel is for things like socket IO and find and accept calls and so forth kernel funnel for everything else and there are some calls of course that can be called either from either funnel or from no funnel memory allocation and free etc so here are some need to knows if you wanted to build a kernel extension for Mac OS 10 one as we mentioned last year we built this thing we call the unified buffer cache which if you had built the file system prior to that would have had to change to support it likewise between public beta and now we introduced the split funnel and of course we're going to be doing things to improve the performance of our kernel and the functionality of our crenel on into the future and some of those things are going to require changes in the file system and if you have one written you're going to have to be inside of the loop you're going to have to contact Apple there's other stuff that may be involved but we can only tell you so much in an hour so the primary message is talk to Jason you if you're thinking about building a file system contact Apple you're gonna have to be in the loop now we do a little demonstration here because we like to bring concepts home at the file system session and so I am a rogue kernel file system extension and my compatriots here Pat Derek Scott Robertson you mish-mosh and Pyun are the kernel and this is me an inappropriately version kernel file system extension attempting to load I [Laughter] [Applause] think you get the picture here's some other sessions you may be interested in at the show to help you with building applications that are filesystem centric or even building filesystem extensions open source at Apple is happening at 10:30 right after this session in Hall a2 there's a session on AFP server and the Apple share client file system in Mac OS 10 that's happening tomorrow in route in this room at 3:30 there's a carbon performance tuning session happening and halt to tomorrow at 2 o'clock and a Apple performance tool session happening in room a to Thursday at 5:00 where you may get to look at your third demo of FS usage we think FS usage is so important that if you come to the world by developers conference you should see it at least twice possibly three times likewise leveraging bsd services will happen in the Civic Auditorium Friday at 2 o'clock the Darwin kernel presentation which will give you an idea of how the kernel is structured internally the mach kernel and some of the bsd kernel services outside of file systems and networking that's going to be at the Civic Center at 3:30 and the Darwin feedback forum will be Friday at 5 o'clock that's all we have for you today I'm gonna ask Jason now to come up and he's going to moderate our question-and-answer I'm gonna bring Pat Dirk's Scott Roberts emission fire champion and Don Brady up on stage from the file systems in kernel team and we'll take your questions you