WWDC2001 Session 407

Transcript

Kind: captions Language: en hey everybody so I'm Chris Lacroix I manage the engineering group that does the QuickTime streaming server and the Darwins streaming server at Apple and some of the things we're going to talk about today are we're gonna go through I'm gonna have some of the engineers come out and show you some of the different ways that you can modify and extend the server using the server module API let's use an API that we've built into this River it allows you to build modules that actually add functionality to the server we've added a new protocol in streaming server 3 that allows you to using stray HTTP requests make request to get and set values out of the server of an engineer talk about that and we've added a new web-based admin which is also extensible and we'll talk about that with a third engineer so let me briefly tell you what's going on with QuickTime streaming server we just released this week quicktime streaming server 300 fail available now you can download it for the web to install on Mac OS 10 it's also included as a standard service in Mac OS 10 server which was also just released this week and all of the darwin versions of the server are available for download as well 3o versions of those of those servers and there's also a tarball of the source code that's downloadable as well as the CVS repository posite ori being fully up to date ok so Steve s3 is going to come out and talk to you a little bit about how to write modules I'm using the module API Steve good morning everyone this hon yes it is as Chris said my part of the talk is going to be on how you extend quicktime streaming server with the module api's and more specifically I'm going to talk about following areas first thing I'm talk about is the module architecture kind of a real high-level overview of what the module architecture is I'm going to talk very briefly about what's new in the API for qts 3.0 and I'm going to walk you through a very simple source code example of what a module extension would look like and then I'm going to give you a really brief demo of that module so I should warn you before I move on that there's a lot of material here and I'm going to cover it very very quickly if you would like more detail the place to look is to download our source code and look at the top level you'll find a documentation folder and inside that documentation folder you'll find a streaming server modules documentation files a PDF file of between 300 and 400 pages I believe so there's a lot of detailed information in that if you're more in if you're interested in this so let me talk a little about the module architecture for those of you who are familiar with the Apache web server our module architecture is very loosely modeled on that architecture the concepts are the same but the details are different in other words it's the implementation is completely different but but the overall concepts are the same there are two types of modules static and dynamic a static module is compiled in with the with the streaming server source code itself so when you build the server you have to compile in a static module a dynamic module on the other hand is an external code fragment or shared library that's placed in a streaming server modules folder that is identified as one of the preferences in streaming server preferences file so once the server starts it will detect modules in that modules folder and load them and from that point on they behave exactly the same as the static module would so ap the API is implemented as what we come in what we call module roles and what module roles are their callbacks with the welding that when the server's in a well defined State and they're your passed and environment through a parameter block that's defined for every role and each role has a specific parameter block format to show you how powerful this this feature of the server is all of the features of QuickTime streaming server all of the ones that are interests I should say are implemented as modules within the server itself so I'm going to bring up a diagram that's meant to be illustrative of a kind of overall architecture of the module itself of the modules architecture itself the topmost box you'll see what's called it the QuickTime streaming server module itself and that's the main module that handles all roles in the absence of any other module to handle that role and most of those role handling girls in that module are do-nothing roles in other words they just return no error and do nothing immediately below that in the six boxes below that are not actually modules but those are collections of cross-platform C++ code that you can call and use within your module there's all sorts of things in there that are platform independent ways of doing networking of threading of new texting all sorts of things that you would need in fact on the only right side of the upper right side of that you'll see something called QT file that's of interest because that's the code that knows how to parse a quick time the QuickTime file itself and take it apart in and see what's in it in all the green area down below those are typical static modules that are compiled in with the streaming server itself and I'm not going to describe all of them but I'll pick out a few just kind of at random here and talk about them QT SS access what that module does it handles all the authentication for the server let's pick another one the QT SS reflector module is interesting it's one of the more complex modules in the server and it what it does is it takes a broadcast stream from an external broadcaster of some sort and replicates that stream out to multiple clients so let me pick one more QT SS admin module what that one does is it handles our web-based admin protocol which you will hear John file in the following discussion after me describe in detail and I should also mention that many of the C++ classes up above are based on something we call QTEs objects and are actually registered in a dictionary structure within the server itself so you can actually access them through the web admin and change and set parameters within those classes dynamically down to very bottom of the diagram you'll see two blocks in lavender those are meant to be illustrative of modules that somebody else wrote that perhaps you that are loaded at the time the server starts up to provide some kind of new functionality that's not available in the core server so let's move on and talk about module roles okay so the very first role you'll see on the list is the register role that's the one that your module will be called in no matter what that's the one role that's always called within a module and it's only purpose is to say what other roles your module supports so that callback is called immediately after the server starts and your module is loaded in memory and it registered it your your module should be responsible for saying what other roles it supports the next one on the list initialized and shutdown role those are useful if you need to allocate global data or allocate memory or set up global data if you do set up global dave data you'll probably want to remember that the server is multi-threaded and your your global data can be called in multi-threaded contexts so if it's critical path stuff be sure the mutex protect it the next roll down is one that's only of interest if your module needs to read preferences so that's your callback to say okay it's time to reread the Preferences file which is XML based and grab the Preferences out of there and set some data within my module based on those preferences the error log roll is mostly not of interest to most people but it will be of interest to us today because that's the one a very simple one I've chosen to use in the demo because it's really easy the only thing the error log role does is any time the server generates an error log message and to be written to the log if your modules registered for this role it's also called back with a pointer to the error message so you might use this say for localizing error messages perhaps RTP rolls RTP is the standards-based protocol that's used to carry a stream pipeline from the server to a client it's a connectionless protocol meaning that's like a Datagram or packet level protocol and it's the primary transport for sending a stream from the server to the client so that's a very powerful way to actually get into and look at the stream or do something with it so if you maybe wanted to do something like server-side encryption of the stream itself you could use that role so let's continue on with the roles that I think you'll find the most interesting and that's the RTSP roles are TSP unlike RTP is a connection-oriented protocol protocol built on top of TCP and it is very powerful and that it manages all of the communication between the client and the server and it's a responsible man all the streams within that context so most of the opportunities for doing interesting things to the server lie within the RTSP roles so I'm going to take a very first one there the RTSP filter role whenever the client starts a session with the server it sends an RTSP request in what we call an RTSP describe function and that's very similar if you're familiar with HTTP that's very similar to an HTTP GET so within that describes all sorts of parameters to describe the session that the client wants to see so what the filter role can do is grab any of those parameters and change them or modify them or augment them so it's a powerful way you can use to to augment what the clients asking the server to do the next one the route role someone yesterday asked me how would i from a client I want to have my students type in a URL and once it gets to the server I want to look up that URL in a database to substitute some other URL so you could use the route role for doing that that's primarily the kind of things that's it suited for the authorized role provide allows you a chance to get in and do authentication other than if you're not happy with the way the streaming server does a syndication you want to write your own authentication level module that would be the the role you would want to register your module for and use RTSP preprocessor role and is your last chance to get into the stream and do interesting things once the client session is set up and all of the D variables and internal things have been set up to begin the stream itself that's your last chance to do it so that lends itself to interesting things like you could perhaps do a redirect to another server at that point as long it but if you do do something like a redirection you have to notify the client by sending by answering replying to the request yourself and in that case you will not call the next role the request role which I'll describe momentarily the post-processor role is primarily interesting if you wanted to do something like gather statistics like for logging the final role probably you probably won't call because only one module can call this role the RTSP request role that's the one that actually manages the real extremely session between the client and the server and that's going to be done for you automatically by the main queue TSS module so you probably will not want to use that but it's there if you do want to use it if you want to manage the session yourself because you don't like the way the server does it feel free to write the code yourself okay here's some new API that's in QuickTime streaming server 3.0 so one of the things that we were asked to do is provide callbacks we're doing file i/o so anytime your server reads a media file and now has the opportunity if it's register for these roles to be called back at interesting times the very first one is the Open File pre-process role that's used to say yes I support this media type or no I don't open this kind of file type or no this I don't have file permissions to open this file type so you can do those kinds of things to say yes or no I handle that file the open and closed file roles are interesting if you want to do something like instead of opening a POSIX file you want to open a media type from a database or in the internet and when you run down on the other side close the database connection you can do it there the read and write roles are read from the media file right from the media file and the requested hit file role are only if interest if you want to do asynchronous i/o and you can use that for being called back when your reads are write complete so as I said I was gonna show you a very simple module and the module I'm going to show you argument C error logging routine and it's what it's going to do is instead of in addition to writing to the error log it's going to email an administrator the error message and so I'm going to walk you through the code real quick to do that it's really short the very first thing your module would do would be have a main function and the only thing every mod must have a main function that's the only entry point that your modules called in every anywhere else it any anything else goes through that that function and the only thing it will in turn do is call the dispatch function a dispatch function is just a switch and case statement switching on the roles that you support and the other than the register role the only one I'm showing here is something called the error log roll which I have written a routine called handle error that is my own routine within my module that's going to do something interesting with that error message and I'll show you that code in a moment and below that you would see other roles that I was registered for such as initializer shutdown so in order for those roles the net to be called in that dispatch function you also have to have a register roll and the register roll as I said previously it's called when your module loads it's a it tells QuickTime streaming server what roles your module supports as you see in this slide I support initialize the shutdown the error log rolls the only thing my initializing shutdown roles do is I've written a very simple SMTP class which I won't show you it's kind of boring and it initializes an SMTP class and then on the other side when the server shutdowns it it closes down the STP class the code that I did write is the is the error is the handle error role and and you saw that in the dispatch slide on the previous slide so in order for that dispatch function to be called I have to register that that I support the error log roll and this is where I do it so let's look at my really complicated function for handling error messages there it is the my send mail function is just cost my little simple SMTP class to send I grabbed from the parameter block pointer that's pointed in to me the string the error message string and I send it on its way so now that we've seen all this wonderful code let's see how it works so one of the things you would normally do on to make this work would be you would drat you would drag your module into the modules folder and restart the machine I didn't want to do that so I just killed the server process which runs all the time in the background prior to this demo to make things a little easier so I'm going to open my demo folder here and here is the compiled module I built and I have this alias on the desktop to my streaming server modules folder I'm going to drag it in okay it's in place now my server functionality is extended the next thing I want to do is bring up the web-based admin which my feeling will show you that's the last talk so I won't go into it but let me click on that so I'll bring up the web-based admin it tells me the server server is not running so I'm going to click on start server now and if all goes well I'll see the page that tells of statistics that tells me the server is up and running and there it is so I'm gonna stall a little bit because I'm waiting for my mail to be routed from my SMTP host to to my pop server but I'm going to bring up this really nifty mail program from Mac os10 which I really like I thought it was pretty cool and you'll see suddenly I magically appeared is it an error message from the server so in addition to writing this error log message to the server to the error log and and I did something to make this error occur I intentionally made this error occur by not having a quick time groups file in my in my server configuration area and so now you've seen me extend the capabilities of server the core server without actually touching the server code itself and that concludes my demo so I'm going to ask Chris to come back up xD so I think the important thing to kind of get from that demo is that it's really simple to write these modules I mean that that's obviously simpler than most but what was it 15 lines of code something like that - the male stuff we won't talk about that anyway we're gonna have John Marauder come out and talk to you about the admin protocol and the admin protocols something new that we added to three oh that allows you to administer the server remotely through variety of means here's John okay so one of the great new features of the 3.0 server is the ability to control and access information in the server from a remote or local application and we do this through something called the admin protocol I'll be covering that and showing some examples at the end of my talk this first diagram is just a simple overview of where the admin module which implements the protocol sits inside of the server it's as Steve was mentioning a built-in static module it uses the QT SS API to access the data in the server and turns around and speaks to any of the applications who are talking to it over HTTP the data that the module has access to is the same as for all modules and that's information concerning the server as in the server object there is a client session object for every client that's connected there's the preferences for the server and a lot of other objects that are all defined inside of the QT SS API so the admin module can go ahead and get set delete or add data attributes and the values for those attributes inside of the server a application that's external to the server can go ahead and have access to that data on a par with any other compiled modules this diagram is an example of the hierarchy that's new in 3.0 card to this as a module you just received a set of objects each time your module was called what we've added is the ability for an object to encapsulate another object what this means is you can walk starting from the server object all the way down to the individual streams of each of the clients that are connected to the server so as an example we have our server object down at the bottom of the next square there's the client sessions and then underneath that the streams this is a example of the protocol it's very straightforward when we say HTTP it's that's what it is it's just to get there's a URL which is made possible by the hierarchy built into the server the URL specifies an object and its data and thanks for this example I'm using a get of the server container object and I'm passing in a wild card giving back all of the attributes for the server container and so this is just a brief overview if you want to in depth read on the protocol and the various options available to you download the source code there's a documentation directory you'll find admin protocol readme inside there and take a quick look at that and now I'm going to show you some examples of the protocol working okay so I've just created a simple web page here with some predefined URLs their urls are query individual queries to the server the first HTTP query is to the server object and the command is get so I'm just gonna click on this and I received the response from the server through the admin protocol and admin module it starts out with the container it says that I've asked for the server attribute inside of the admin container and I have a 0 error status code so this next query is the same as the first except I'm adding the attribute for the server version that's contained inside of the server object and once again it's a get and you can see that I'm receiving back the value for that attribute this once again this attribute the name may look a little odd to you but if you've seen the QT SS API you'll recognize that is one of the fields that inside of the server object in this next URL there is a asterisk wildcard which says take all the attributes inside this object and return them so we have a full list of data elements in this object including the objects down here at the bottom that are contained each of the values that are coming back are a name and value pair and the values are quoted so there's some of the other things that you can find inside of the server are the preferences this is the QT SS server preference object this is also the same information that you'll find in the XML reference file inside of the server so each of these values represents what's in that file one of the nice things to say if you were to change this any of the values here or add new attributes through the protocol it also updates the file on the server itself if you have a module and you've compiled it and added it to the server you'll show up here in this modules list and you can see here we have the file some of the example modules their mail module that Steve showed you file module which handles the on-demand movies reflector module that handles broadcasts to the server and some of the various other modules built in so the way you would access provide a URI perhaps for your module is to have the protocol access your preferences and in this case I've just said give me all the preferences for all the all of the modules all in one query the way this query was executed was to have two wild cards first wild card represents the name of the different modules and the last wild card asterisk here represents all the attributes at that object okay so for this next example I'm gonna bring up QuickTime Player and we're gonna add a couple of movies okay so this particular query is going to go and look at the client sessions that are currently playing I've I've asked it to get each of the sessions there's two of them playing right now you can see the URL the movie duration for the first movie and also for the second movie there's the interesting thing about this particular queries that I've asked for specific fields inside of each of the sessions so I've narrowed it down from just a basic wildcard search to specific these three attributes inside of the sessions now one of the features that you'll find inside our web admin is the ability to turn it on off access to the port and way we do that is we have a list of ports and I have two right now that the server is listening on so what that means is I can bring up one of these movies and if I ask for a port that doesn't exist it will fail the connection but one of the nice things about the protocols that allows you to control the behavior of the server so in this case I'm going to tell it to add the port value 8080 we got it back a status to zero you can see that it's been added and now we're playing so we just changed behavior of the server so this last query that I have for years pretty much starts out at this server level as we started but I'm adding all of the parameters that are currently defined inside the protocol so the R is a recursive search verbose means give me the full path for each of the attributes T is the type give me the type of every attribute a is for access privileges and the last one D is any debug information that you might have it's asking the Dakota admin module return that in the protocol so here's an example of all of the information that you can actually pull out of the server from an HTTP application so it starts out here at the server keeps on going through the modules and keeps on running and running and all of these are you can see that we have different the values the data types access privileges this one's read right and here we have a list of error messages that the server will be sending back at various times and finally we have our arrow status which is zero and a verbose message of okay and that's that's our demo thanks John so pretty obvious that that protocol is really powerful we're hoping to see people write a lot of tools around it we obviously use it for the web-based admin and my sling is going to show you a little bit about the web-based admin in ways you can extend that mightily Deveny you've heard all about the Qt SS API in the admin protocol so now I'll talk about how you can use this knowledge and a little more to be able to customize the web admin the web admin is one of the new features in QT s3o as you would have probably gathered if you were in the earlier session and I'll talk a little bit about what it is and how it works also I'll give you an example of the special tags that we're using the admin and ended with the demo the web-based admin is a simple web server written in Poland has some CGI scripts and server side includes it is intelligent pearl and the CGI's handle all the HTTP response headers the other thing that's different from a regular web server is that the authentication is provided by Q TSS so the web the web admin server just acts as a pipe between the browser and QT SS and it does not do any authentication of its own of course it does SSL encryption and that's part of what the web admin server itself it's based on the admin protocol and that's how it talks to QT SS using the admin protocol now why would you want to customize the web admin that's probably the question that you're asking your now if you want to be able to configure or modify advanced server preferences or add advanced monitoring capabilities to the server then you'd want to extend the web admin in otherwise if you're a module developer and you've always wanted to have an easy way to configure and monitor your module then extending the web admin is a way to go you'll find that it's a very you can have a very easy-to-use interface now how does the web-based admin work what happens when you actually get an HTTP request it's very similar to a regular web server because when a request comes in it looks for the file and it's in this case it's a CGI so it executes the CGI the CGI looks for the template HTML file which is the hoos file with filename is passed in as the argument to it now the template HTML file has a bunch of special tags which are kind of like server side includes and they tell the server admin server what to do with the tags that is whether to go fetch data from QT SS and process them and displays display them in the format that you want them to be displayed so wild cards on the template when the CGI find such tags it will send a request to the qts server using the admin protocol well it'll actually talk to the QT SS admin module because the admin module is the one that implements the admin protocol it gets a response back and the response is used in the form that you probably have seen when John demo his the admin protocol the admin protocol response is parsed and the values extracted and processed based on the information in the special tags and once the tags are parsed the attacks are removed and the data the processed data is put into the HTML and the last step is to append the CGI headers and then send the HTML to the browser so it's very simple very similar to a web server the only difference is that there's an interaction between the admin server and QT SS now I've mentioned these special tags all the time so you're probably wondering what these look like and what they are because this is what you need to know if you have to use extend or extend the web admin server so one example of the tags is echo data and echo data basically tells the server the admin server to fetch some data which is actually the value associated with the attribute the parameter and process it strip out everything else and just get the value and replace the tag with the fetched value so if a tag such as this is found in the HTML template the admin server translates is to an admin protocol request which is again a simple HTTP GET as you can see all it did was append modules admin to the actual parameter that was in the template HTML and once it gets the response back it parses the value and replaces the tag in the template HTML with this value so it's very similar for the other tags there might be more process information like formatted into radio button or things like that or take an array and display it as a table there's a lot of things that you can do with these tags now to illustrate this the use of the special tags I'll well do a short demo and you'll find that the admin server handles all you know can talk to QT SS and it handles all the other details like authentication encryption things like that for you already so it's only a matter of using writing simple HTML to extend the web admin ok now earlier you got a glimpse of how cool the web-based admin is but I find that that wasn't enough for me so I decided to actually have my own page that only shows some of the server statistics and it refreshes them every five seconds I don't want to see all this that you're seeing right now I only want to see certain data so I started off by writing a simple HTML and I'm going to show that to you using Apple's HTML editor as you can see the static HTML shows up as you know the form elements are all formatted and the only thing that's interesting here for us is that special tags they look very strange but those are the ones that actually do all the job for you and the example that I gave you eco data is up there and then there are other tags they'll get value and form and float and things like that and these are the ones that you'll find very useful when you're extending the web admin and if you see there's nothing other than just these tags and these tags are doing all the work where they're talking to the admin protocol you know authentication is taken care of you don't have to do anything yourself ok and I also wanted to add have a way to get to my page from the current web admin so I modified the navigation page the current navigation page to include a link to my page someone dragged this in to the admin HTML folder and my snapshot page and let me go to the web admin and see if it shows up there you go there is a link to my snapshot and there it is so here is my custom snapshot page it has only certain data that I care about which gets refreshed every 5 seconds let's see if we can connect a couple of clients to it and see if it gets refreshed and you have the right output as you can see the cpu load and the current RDP connections all the data is being refreshed every five seconds and it's only data that I care about which I can very easily get to and all that is authenticated I don't have to write anything else no other scripts or any code to get this information okay so it's very easy to customize the web admin and all you need to know is how to write some HTML which pretty much everybody knows so well hopefully everybody knows that it's very easy to get started if you don't and well the cool thing is that we've done everything for you so that you don't have to go and redo all the work well thank you [Applause] thanks mightily so obviously you guys can see how powerful that is and how simple it is once you understand the tags so let me talk about some development opportunities some during the prior session you've already heard these but there are a lot of ways to modify or enhance the server the first one is since learned open source project obviously you can modify the source code directly that's great if you really need to do you typically don't need to do that unless you're going to be doing some really bizarre things or maybe fixing a bug those types of changes fall under apples Apple public source license which means that if you make changes to the to the flourish code directly you need to provide those changes back to Apple and the reason for that is that we can provide let those let those changes out to the rest of the world mmm as Steve showed you can write a module modules they're relatively simple to write depending upon what you're doing but at the same time you can also write really complex modules to that do some pretty amazing things you can extend the UI using simple HTML possibly a little perl script that does some additional parsing if you've got some complicated UI element you need to deal with or you can write a monitoring tool using the protocol that John showed which is basically just straight HTTP which HTTP that allows you to get to every attribute in the server basically and you can also implement our caching protocol and this is not something we showed today but the caching protocol is basically an extension that we've offered up to the IETF to RTSP and RTP for providing additional information to caching streaming servers for giving them additional metadata so that they can make sure that the stream from the caching proxy to the client is actually as good as the stream from a server directly to a client so we encourage you guys to take advantage of all of these like when you can or where you can for more information on all of this stuff the probably the best place to go is make sure you're subscribed to our developer mailing list and you can go to lift a palom just searched for the word stream and you'll find a developer's list and a user's list as well and these lists were actually really good the content tends to be very focused the the traffic is actually not so high that it's annoying between five and twenty messages a day and all of the engineers and and part of marketing are actually on this list and we respond to questions or requests on the list and it's a great way to just get a hold of us directly to if you have any questions or need some help with anything the open-source projects including the Darwin streaming server are all available at WWF in Florida Apple comm so you can go get the source code there get all the documentation for these for everything as well as sign up to be a Darwin developer and for any kind of non server QuickTime related things such as how to hint movies out of author movies QuickTime api's for broadcasting things like that you can go to the developer site for QuickTime at developer.apple.com slash QuickTime and the one thing I'd like to ask you guys is if you guys plan to do plan to do anything around around any of the ways to extend the server let us know let us know on the mailing list and we actually are really excited to help people do think get things done we want to make sure that you get it done right we want want to make sure it's kind of painless for you to because we want to see some really cool things out there today's Thursday so we've only got a couple of things left in relation to QuickTime there's a feedback forum at 3:30 in room j1 and that's a feedback form for QuickTime in general so it'll be both client-side feedback and server-side feedback will have both server and client people there and there's also a beer buffs tonight at Apple and we will have server engineers at the beer best as well their way that better to organized we're kind of broken into different areas so check out either the server group and/or the QuickTime area you