WWDC2004 Session 106

Transcript

Kind: captions Language: en [Applause] alright good morning this is the service management for Mac os10 talk or as i like to call it writing modern damon's and mac OS 10 so what does that mean let's jump into some terminology what is a daemon a daemon is a long-running background process people don't tend to see it it's not in their immediate view like the finder or Safari well we don't just have damage on the system we have super damon's to what is it super Damon a super Damon is a daemon that proxies work on behalf of other damage to make their lives easier we also have a notion of an agent which is more specifically a daemon that runs in the context of a user after you login something like the ichat agent might run on the background do a help i chat out finally something that's very important for this talk is the notion of a communication handled we talking about this a lot communication handles a file descriptor or a mock board something used to talk to somebody else be it near or far also some assumptions I'd really like to assume that most of you have prior experience writing a mock or UNIX Damon also some familiarity with the mock or UNIX system calls to be very helpful let's jump right in so what is this session about it's all about background processes you do work on behalf of a user either directly let's say like someone telnet then the telnet server is launched or indirectly a thing like look up d which is doing queries on behalf of applications to look up users or internet hosts or whatnot and finally your code needs to get running at some point you have work to do how do we know when to run us at all now what's wrong with the status quo I believe and we believe the daimons deserve better treatment and both the UNIX and mac OS world Damon's were just prophecies which disassociated themselves from the user a might say core graphics nope don't need you a call functions to remove yourself from the users at context in the mac OS 9 world we call these spaces background applications again they were just programs which said nope don't need this don't need that oh yeah no don't need this subsystem over there that's not the way to go the solution we're going to provide a new super Damon to manage all the daimons on the system it's designed to do work for you it's designed to be flexible it's designed to support messaging and control and finally we're going to call it launched deeb a good major component of this since we want all damage to be use this is that launched II will be open source this is a critical objective of ours we want UNIX damage to adopt this technology things like Apache and bind and you name it any unix Damon we'd like to adopt this technology we want to have it the open source we want to have it run on freebsd and linux and solaris and all those communiqués is it's in everybody's best interest so we're going to learn in this session issues a modern Damon writer faces things that affect you as a developer what launch T will provide for you what it won't provide for you and how to port an existing Damon to launch deed also I'll talk briefly about how to write a savvy Damon so let's let's talk about some history why do we need to write a new Damon well in the UNIX world the classic super Damon is ina d ina d is responsible for things like running the telnet server or the ftp server you name it well why was it created it was created because there was the observation was made that every Damon bit did basically the same thing it opened up the socket and sat there waiting on accept and then forked a child and then had the child deal with the client well if they're all doing the same thing we can have one Damon do that for them and not have 20 Damon's all running at the same time now in hindsight I nandi was IP centric it also assumed that there's only one communication handle / Damon this is definitely not true these days with the invention of things like SSL almost every day min these days is listening on at least two ports one to handle SSL traffic London to not handle non SSL traffic also in the UNIX world we had something called an it and it was centric to a different kind of technology it was centric to TT wise things like the console login session or a login session running on a serial port off on the side that was you know I'll find a dandy but it was only specific towards the specific technology also in to handle different kinds of daimons we had this call-out starting from etsy RC when the system booted we had a shell script run it would then in turn run other things and eventually a script of yours would be run that would let your daemon be launched well this is very very static it only happens at boot there's no notion of you know the world might change we turn the system off I mean we put it to sleep and woke up back up somewhere else and also didn't d able to shut down very well finally and they also had cron at and batch there were time centric that's great but maybe you want to run for a different reason maybe you want to be like send mail and run once an hour to deal with spooled mail but you also want to deal with incoming connections on port 25 in the mock world they had something similar to ina d called maken it mocking it can launch damon's on demand based on mach port IPC that's great but it's only from awkwards what about file descriptors maybe need both so that yes that brings us to today's problems there's missing functionality first of all I nadee doesn't handle UNIX domain sockets despite them being file descriptors we have file system Bates events to trigger a daemon launch maybe your daemon is this sits waiting for a male spool to fill up and then try and drain it periodically well we'd like to be able to run your daemon when and now that school directory gets files in it also ina d and enid don't support user-supplied jobs you know why can't a normal user just submit a job to inet d saying hey when somebody connects to port five four three two one run my little custom Damon you can't do that today because the config file is in flash etsy and only route can write to that file that's a shame also multiple event sources like we talked about with I net D it's only allows you to have one communication port and that's just not practical these days we're still some Damon's listen on both mocking UNIX file descriptors something like the multicast DNS responder is listening on a mach port to handle requests from the GUI abs but it's also listening on file descriptor to handle the network event there's no reason why we can't launch that Damon on demand today except that our super Damon's only support one technology ultimately we believe that time file system IPC events all these various events sources need to be supported in the sames it's super Damon we need to be able to launch on demand whenever any of your events happen also a critical request by many people is that their Damon automatically be restarted if they inadvertently exit or die for whatever reason so that's something else we'd like to support that not all the super dam and support so the future we're going to have one Damon to rule them all its support for transferable based event sources at the moment we support most file descriptors and we support we will support mock ports also support for user-supplied jobs is we believe a very critical feature we want users to be able to run their own damage to support their own needs and not need to run their own super damage just this when we already have one running on the system so what does this mean first of all we want to save you work in the UNIX world you need to fork you need to call set said you need to close stray file descriptors reopen standard IO is Devon all it's just a mess it again it goes back to the we were a user process now we need to disassociate ourselves from the user input and these are some of the items you need to do and nobody likes trying to keep on top of any new API and you need to call just to disassociate themselves from the user input what are we going to do when you hit main your program will be pre demonized you can check in and go we're also going to support launching on demand this helps you help us save system resources we will have your communication handles open and alive even when you're not running and should someone connect to them we will then launch you it also improves the system boot-up we don't need to run your entire executable if we just need to open your communication handle that's less work we needed to do at boot and our users will appreciate a faster boot-up it also means we can do a parallel load of boot we plan or actually we do implement the registration of all the communication handles of all the damage all at once that means that if Damon a needs to talk to Damon be we don't really have a dependency problem because the dependency is implicit by their communication through a file descriptor 0 mach port we register them at the same time we don't need to worry about it which one gets launched first which helps us in too if you're used to start up by them as little external dependencies we've been listing there you don't need to track them anymore you don't even need to list them also since we want to support user-supplied jobs we can support something called user agents we're going to standardize the way background processes run for user prot users and allows them also to launch those on demand we've never been able to launch user agents on demand in any standardized way this will help our login performance which our users will appreciate finally this is something I'm really excited about we're going to support messaging things like you know hey Apple file server how many by yourself tell me how many people are connected tell me you know who they are tell me about them we'd also like to support controlling damage things like hey AFP servers you disconnect that user they're done now or please shut down and the daemon can come back to us and say okay I'm ready to shut down now I've written out my log files I've closed and cleaned all my stayed thank you I'm done now this we believe this is very important for things like database servers which might have a lot of work to do before they shut down the old way in the UNIX world was just to send a signal cross your fingers and hope that they exit someday and you know that's that's not great we need to be able better than that but you know the number of messages that we can support or potentially endless and we look forward to feedback from you on what met kinds of messages you'd like to see sent to you or messages you'd like to send to us what would you like to tell us about yourself now let's talk about some case studies the real world is what matters we have something called cups beyond the system it's a common unix printing system they have a daemon today what they did to support restarting is they use some mock API is to check in with mock ahead and tell it hey restart me if I die that was extra code they needed to add it just to get restarted in case they exit with launch d there's no extra code we know you want to keep running your a daemon you exist to serve other people this is a simplifies their port to Mac os10 and makes life a lot easier for them something I brought up earlier is the multicast DNS responder it uses both mock ports and UNIX file descriptors we'd like to be able to launch that on demand but today we have an ID on the right hold just supporting file descriptors I'm mocking it on the left holding just mock ports they don't know anything about each other one could launch mdns responder based on communication on one type of technology but not the other leaving problems for the mtns responder if they want to launch on demand launch the handles both nothing else does for launch on demand and we're very excited about this support for multiple technologies also back to the user side let's bring another case study example the ssh agent is a classic example of a command-line process that people want to run to support their SSH activities a lot of people usually end up creating these complicated statements at the beginning of their shell startup just to make sure that they have only one SSH agent running on the system so they'll create lock files then they'll try and see if they already have an agent running and if not run it and then unlock the whole you know state that they built up well with launch Steve you can just drop an XML plist into a directory in your home directory and at login time we will register the ssh agent along launched on demand the next time you talk to it and only one of them for your login session which means all that complicated locking and checking to see if one's running nope don't need to do that we're very excited about this for user agents we hope to see more cases where people use it also the changes to ssh agent were very simple this is something we're excited about because it means less work for you guys we know pretty much how demons operate their state engines we know what information they need in order to get running we can get that to you right at the beginning of your process and again your pre damon eyes so you just take your check-in information and you can get running so okay I'm totally about what launch T does but what doesn't it do well there's a lot of other kinds of events on the system we have config DS database of key value pairs or config these events we have the net info database of key value pairs which can come and go as the system runs we also have the rendezvous service advertisement some people want to launch on demand just by virtue of another entity showing up on the network we also a very popular one is the io kid namespace which is built on top of mock ports and I okay at events again up on top of infrastructure already which are is built on top of mock boards now we wait you know not i okayed events well we have things like the Bluetooth Damon to launch one bluetooth devices show up we Apple have our own needs we know you have your needs so this list that I just gave a things we don't support is subject to change we're probably most likely add those subsystems in the future as we figure out how to deal with technologies that are built on top of each other now porting let's start jumping into the meat of things we have a simple very simple IPC API and a very simple rtti based object system to support the message passing that we want to use to communicate between launch d and your daemon now what does the IPC APA it looked like well it's kinda sorta corefoundation well why not core foundation portability just about what we talked about earlier launch d is open source this is a very important goal for us to drive adoption and minimizing our dependencies helps make adoption easier also a stumbling block is the mach port and file descriptor passing is not supported by core foundation at the moment it's very few people requested it was just easier for us to implement a simple object graph ourself and support serializing and sending it across to the other side finally really all we need is our TTI dictionaries and arrays we're just sending messages back and forth so to give you an example of how simple it is this is what the two basic CPAC api's look like we have launched message which supports sending an object graph and getting an object graph and response and if you're going to be doing any kind of asynchronous programming we have a simple API to get the file descriptor that we're using on the backend to communicate with launch key and add it to your own run loop if you're using case you select it's up to you we can give you the file descriptor you can add it and you can call launch message when you get a when messages are available to read now to jump more into the meat of it a launch data t represents an object graph it could be anything you know we have the rtti that's what makes it fun and you can introspect it as you get a response back or you can send just about any you want also launch meet message while it appears synchronous for the common case is actually what launch method is synchronous for the common case it returns null and sets error no on failure plain and simple for the simple case now if you do send the message was requesting that asynchronous messages we sent in response you can then call launch message with no message to send by using null as a parameter and then you'll get any asynchronous message back and you can keep calling and keep calling and keep calling until you get all back and if errno is equal to zero then there's no more asynchronous messages and if it's obviously equal to something not equal to 0 then there's a problem now let's talk about those messages we're sending back and forth you know rtti and container classes are fun we have dictionaries we have a raise and we have some basic objects to match the see basic types we have file descriptors mock boards integers real numbers boolean strings and finally opaque data should we need it now again it's just enough for IPC and no more we have getting set operations for the basic types and in the case of dictionaries we allow insertion lookup removal and iterate again all we need for IPC for a raise gettin set index and get count it's enough to introspect and set up an object graph that's all we need now to understand the object graphs are going to be sending back and forth let's talk about the XML plist that you'll use to configure a daemon for us to run into the later point in the system you're going to have something like a label to uniquely identify your job and that really is the unique key this is in case someone wants to control your damn and the label is what they're going to use to send it can turn it on turn it off change some attribute of it that's the important key you can have things like the username you want the damn and run as the group name program name and pretty much any kind of environmental setup you need to get the daemon running you know I want the daemon to run a networking directory I want to have this argument and environment variable any kind of setup detail we plan on supporting now if you notice at the bottom of the list back there we have something called a vent sources they're all about the details of how to setup file descriptions and mock boards you know who to connect to you know where to listen particularly file descriptors or can be very complicated since they're different technologies we can back them you've got ipv4 ipv6 UNIX domain sockets it could be a new type of technology that we don't know about the requires different API is to set up the file descriptor and we need all these details put in one place so this is what the event sources are now before we send an XML plist over to launch key we actually do some data distillation we're going to turn those usernames into you IDs the group names into G IDs and we're going to take all those events sources and actually translate them down to the file descriptors and mock ports the back some and then the resulting object graph is much much simpler it's and requires no lookups by launch d to find out the actual details of anything and the great thing is since we've translated things to follow the Scriptures any kind of security check the colonel would do we don't need to replicate now let's look at some example messages life's pretty simple you might want to submit a job and then you're going to have that you know translated xml p list you might want to remove a job well then you just need the label you might want to get the jobs you want to introspect launch do you want to find out about what damages are running on the system I don't get you a big dictionary coming back you might want to check in your daemon you can call it send a check and message and cut out the file descriptors and mach port so you have allocated to you will come back and you can proceed with them as you wish and finally something that supports for things like the ssh agent is we need a notion of getting unsetting and querying and environment that isn't directly tied to your shell now something that we're still working on is becoming launch T savvy if you remember I talked to you about sending messages to control it tell me about the users you have connected to you please i want to disconnect this user over here we're still figuring this stuff out we look forward to your feedback and definitely some updated documentation for you in the tiger time frame so that way you can understand what introspection and control operations we supply now let's hash things out again so pre demonization we believe this is very very important for you it means again you're going to hit main and your demons ready to run you don't need to worry about disassociating yourself from any kind of sub system on the system via the graphics or maybe applescript i'm actually don't program at that level but at least in the UNIX world no forking no call set said no closing straight file descriptors left behind by who knows what she'll life's good now you just need to check in and go also again we automatically start support restarting so should you decide that your configuration file change do you want to rerun just exit will launch again should next time someone connects to you also something we hope you will take advantage of is the multiple event sources you might have a file descriptor or 20 file descriptors we can listen on all of them on your behalf and run you whenever activity happens on any one of them if you listen on a few mock ports sure tell us about them well launch you when activity happens on them too we also the user agents are very excited about should you have a foreground / background application combination well maybe you don't want maybe the user doesn't want the foreground application to necessarily run every time it log in but they still want the background activity to run a the example I like to continuously point back to is the ichat agent you have the menu lang it supports showing you which users connected but maybe you don't want I chat running a plugin but you do want the menu you do want to log in well with launch d we can run the ichat agent at login without running the foreground application so to give you an example demo I would like to show the ssh agent running under launch d and i would like to show our debug damon running under launch d and acting as a pseudo web server to tell you about launch d itself by doing the introspection that i talked about so to show you let's show you the web server I have this is a representation the object graph coming back from launch Deeb shows you that here's our key we have the combat apple to launch debug d we run it on demand there's a boolean set to true it's got a pit at 328 if I reload this web page you'll see it you know we keep launching on demand because this number is changing we have some you know various setup details in which you ID to run it as if you notice it's running as me the user finally there's two file descriptors we're listening on they happen to represent the ipv4 and ipv6 sockets or set up by the user and also look at this we have an SSH agent it's also running on demand it doesn't have a pig set yet because no one's tried to talk to them yet and this is the unix domain socrates listening on so if we go back to the terminal we can run ssh ad and now if we go back here and run it again we see that the ssh agent launched on demand how did this all happen well and goes back to the Simple pea list we talked about dropping on the system if you see until the library launch agents we have a couple p list there's a launch debug d that i talked about and the ssh agent and they're pretty simple files we got com openssh all we have is again here's that label here's the program arguments and the listeners which event sources which get distilled down to an actual file descriptor so in this case we have a little secure socket with the environmental key which if you're using ssh agent if you're familiar with and the socket type we need now that can show you how simple it can be too oh I'm i will show the the apple so here's the debug dammit for example it just listens on a port 1 2 3 4 5 happens to be a passive socket and so it's like sock stream and what ends up happening if you're familiar with the get a tour info API is we take these details we throw it a get a tour info and take whatever at our info structures come back and allocate a file descriptor for each one of them and then send it over to launch T and it just listens on them so in this case ipv4 and ipv6 sockets came back and you know that could change in the future and your daemon wouldn't even need to know about it a new type of communication technology might come out and you don't care as long as it's a file descriptor and it comes back to you you can get events off of it and handle them so that we're very excited about this and I believe it's time for Q&A now this is yeah oh also for more information you talk to Jason and yeah and this source will be on the developer website at the determined developer website