WWDC2004 Session 709

Transcript

Kind: captions Language: en thank you so yeah as Steve said we're going to have a cast characters up here and you forgot to mention one which is John Anderson at the very end is going to do a demo of the next version of QuickTime streaming server publisher which has some great new features that for the project that we're going to talk about mi stream that they're very you know excited about the new product because some of the some of the new capabilities so the agenda is first me talking about some of the how-to stuff setting up authentication streaming through firewalls etc and then we're going to go into this case study on the merit network so merit network they are a non-profit provider of Internet access for most of the schools and universities in Michigan and so they're going to go ahead and and go through details on the history of the project some of it objectives how they set up relays how they've set it up so that to redirect clients to the right server some of the testing and configuration and then future plans so let's dive right into some of the specifics on how to so one of the things that a lot of people want to do is is set up authentication and there's two main reasons to do this one is you want to authenticate who can broadcast through your server so with a live streaming product who can actually send streams through your server and we have this thing we call automatic announced broadcasts that save a lot of the hassle of the way you used to have to be done in terms of doing a live webcast through the server so one of the reasons for setting up a foundation is for for providing a friendly way to do these automatic announced broadcasts then there's also just the authentication to the media itself and authentication that we've implemented RTSP authentication is very similar to HTTP digest authentication so the protocols are similar and the implementation is also similar to what you would in Apache so just like an Apache where you have an HT access file for authentication in QuickTime streaming server you have a QT access file for setting up authentication to a specific directory on the streaming server and just like Apache with HT access you have files flat files on the disk that have the users and groups in them so let's just take a look at this for announced broadcasts in QuickTime streaming server we have a GUI for creating this password and I've highlighted them out here on the screen you would click on accept incoming broadcasts and click on set password so that allows you to create a username and password for an announced broadcast from a live encoder to the server on the the client-side I have two different screen shots here these are three graphical broadcasting clients available today that support announced broadcast QuickTime broadcasters the one from Apple it's free and it's you know it's very simple in its capabilities one that I'm really excited about there's a new product and Simon Clark is in the audience here I encourage you to maybe catch him afterwards he's got a great new product called Wirecast they uses quartz transitions supports multiple video inputs it's an incredible piece of web casting software and their screenshot is on the right so they also support this idea of announced broadcasts so basically what you do is you provide a host name a file location you don't really have to put the SDP extension in I like to put it in you know it depends on if you it's a good idea to add the SDP extension the user name and password that you've set up on the you know in the previous area and then when you webcast what this does is it automatically generates this file on the server that's necessary for a live webcast this SDP file session description protocol file so in the old days which still works the old way you can do not an automatic unicast manual unicast which means you have to manually export the FDP file and manually transfer it to the server that way still works and that's the way that for instance if you are using one of the hardware encoders be brick jvc both make hardware mpeg-4 encoders that work great with our streaming server but they don't automatically generate the STP file you have to get the FTP file from the device and transfer it to the server to reflect the live webcast so this way is a is a more is a friendlier way so if you're out in the field with your power book and you want to do a live webcast through the server this is a very easy way to set it up now under the hood this all uses these files that we talked about before so in this example we'll try this laser pointer I've done this cat cutie access so I'm in the movies folder on the server and I've keep I'm looking at this cutie access file this is what's in it limit write require a group broadcaster slash limit and then require any user so this one means that to write to the server you have to be this username broadcaster require any user means any other user can just connect to the server so there's no authentication anybody else can freely connect to the server to access the streams and then this cat command is I'm tatting the Qt groups file and actually here's required group not user so anybody in the group broadcaster will be able to webcast to the server so I've catted can this QT groups file and piped into grep being a UNIX weenie here and look for this name broadcaster and these are all the users that are often are going to be able to broadcast through the server on you know on in this setup so this is the if you want to set up your own authentication you're going to have to do some command line stuff and that's just the way it currently works so basically what happens if you want to authenticate access to media you are going to have to modify these files okay so again I've looked at the QT access file this time I'm saying require group users that means in this folder I have the special directory off of my movies fault directory called authenticated so now it's just like Apache it's anything in this directory will now be required n ocation because I've got a QT access file in it says require group users so now you have to be in the users group to be able to access media and it's going to put up this login dialog box when you try so if I look through my QT groups file grab users these are the users that are in that group called users and then here I've shown what's the contents of the directory are so what happens with QuickTime Player you launch the time player you type in the URL to that authenticated media and then it's going to ask you for a username and password before it streams the content to you so to create these users there's a command QT password which allows you to create new users in the QT users file to create the groups it's just a text editor it's very simple format it's just the user the group name followed by a space delimited list of users so these files you can have other QT groups and QT users files they don't have to be the default files that are in the config folder of QuickTime streaming server they can be other files so you know you can set up your own little we see in higher education in particular where they want to give a faculty member the capability to create their own their own users and groups file and that works as well so this is kind of the old way to do things with that we've been going over which is what apache has and what quicktime streaming server has a lot of folks want to tie into a more organizational authentication architecture and quicktime streaming server like Apache is written in modules so there are modules that can be created or modified to do things like customize authentication so one group that has done this is NYU so NYU is involved in to project called Shibboleth Shibboleth is about cross realm authentication which means that for instance the NYU library might want to trust people from another University will say Cornell University they might say well we've got an agreement with Cornell library so that we can both access each other's stuff and so should well if that's what it's about is this cross realm authentication and if you go to the Internet to webpage you can read all about Shibboleth they have written an Apache module because it's modular - for Shibboleth that allows you to do this with websites so what NYU has done is taken that infrastructure and extended it to the streaming server they didn't do it directly with the streaming server because the Shibboleth authentication they use it's not just a simple challenge in response kind of dialogue between the web server and the client there's a lot that goes on there and that sophisticated dialogue the RTSP authentication was just too simple for that it was didn't provide them with all the infrastructure they needed so they set it up with Apache so basically the flow here is you know you request the URL from Safari your web browser it goes through this Apache module to authenticate you against Shibboleth which does this cross realm authentication so Shibboleth can talk to basically anything LDAP Kerberos whatever you want and then what they do is once you've authenticated it talks to a my sequel database stores the IP address of the client the cookie the ship cookie and timestamps it and then redirects the to this custom module they've written for QuickTime streaming server because it's modular they've written this custom module that understands these special requests these special URL is the special URL comes in and then this custom module goes back to the my sequel database and says okay what IP address are they coming from what's the ship cookie and when did they do this one so you can't replay these things and then it says yes or no and delivers the streamer doesn't deliver the stream to the client so this is an example of how a customer has taken the open source part of the streaming server the Arwen part of streaming server customized the module so that they can deliver a custom authentication and we're working with NYU to try to make this piece available through the Darwin CDF so it would be available to any QuickTime streaming server or Darwin streaming server but you have it I have all this other stuff working too for it to work so we're in the progress the process of trying to figure out how to make this available to a wider audience then then NYU right now so pretty interesting stuff next topic so we're done with authentication let's talk about firewalls if you have a publicly accessible streaming server you should turn on for it ad streaming and this means that you can't run a web server on that same port on that same IP address on that server so it means that you might want to bind your streaming server to one IP address and the web server to a different IP address if you want them both to work over port 80 or just run your web server on 8080 or some other port so what this does for daily streaming is the tunnels all the RTSP RTP traffic through HTTP port 80 and that means it gets through most firewalls QuickTime 6 supports automatic configuration so it will try RTSP over the standard you know streaming ports and if they're not available it will you know go back to port 80 it's all back to port 80 and this is very recommended for these publicly accessible servers because it allows you to get through firewalls and you don't know where the clients going to be in your organization another thing that you might want to do is open up certain ports so this allows you to access streaming servers that might not have for dady streaming enabled so minimally you can open up TCP 45 54 and 70 70 70 70 is what real servers use and we can use it too but by default we use TCP port 554 for RTSP which is a signaling protocol to the server and then UDP is the protocols used to actually stream the data and you can open up a range of ports here it depends on you know how busy your server is and how many clients have you have connected to it so that's what sets that range of how many how many ports you actually need a typical range is 69 70 through 9999 and then if you want to administer the server via a web browser you need to open up TCP port 1220 and if you want to administer it through our server admin application you need to open up TCP port 311 so those are the port's you need to have open another way that people set up again inside an organisation security through firewalls they have a special partner network called a DMZ where they put proxy servers and there is an RTSP proxy server that allows you to do this and basically what you're doing here is you're saying the proxy server can talk to any server on the Internet and any client can talk to the proxy server but clients can't directly talk to the servers out on the internet so it puts this you know go-between box in there and we have you know a version of this that's posted to our Darwin site that you can download and use as well so this is another option more of an institutional option again for people to create firewall that you know to work with a firewall probably the least common this is the least common of the three so next we'll talk about the really I think the meat of this presentation which is scaling streaming through relays and replication so streaming relays are our way to conserve bandwidth really so when you're doing a live stream or a playlist stream or your streaming stuff out in real time to a lot of different clients what you can do is set up a relay so that that a client doesn't have to go to the source server so this is what we've used in Michigan typically a school district doesn't have a big you know oc3 line or something like that to the internet they have a t3 or a t1 even which isn't a very big pipe so you're not going to be able to get a lot of 300 kilobits streams through a 1.5 megabit t1 it's just there's not a lot of bandwidth there so instead of trying to get 50 people to connect with t1 on 300 kilobits which won't work because you'll way overload the capacity of the t1 you could put a relay inside the school then what happens is the main server sends a stream to the relay and all the clients inside the school tuning in on the local relay where they have plenty of bandwidth inside the school so they provide these streaming points of presence so this is the kind of model that Akamai is based on so Akamai has this capability built in you might have heard of Akamai which is a industrial-strength replication and service out on the Internet but they also support this live capability these relays can support unicast or multicast so what I'm doing right now is I'm multi casting hopefully you're all listening I'm speaking so it's one person lots of listeners so that's what multicast is about whereas unicast is if we're all on a conference call we'd be tying up I don't know a 100 phone lines so in the unicast everyone makes a separate connection just like we do on the phone network when we when we make phone calls these can be used with live are typically used with live or playlist' streams they can actually be used with on-demand media but it's not a very common use of them another way to save bandwidth is to replicate your content and this is what Akamai does in a big way but there are little ways you can do this with a streaming server as well that are pretty interesting and it's commonly used for video on-demand moving that content closer to the client so relays the basics of relays is every relay has one source and one or more destinations the source for a relay can be one of these FTP files it comes from a live encoder it can be an actual piece of media and it needs to have an IP address and port numbers for that source where is it going to get that information from and then you can have one or more destinations where do you want to send this stuff out to so that can be an IP address with specific you know port numbers or it can be this thing called in an announced destination which means it will do same thing the broadcaster does it will announce the broadcast to that destination automatically generate the FTP file figure out the port numbers etc so a very simple configuration for for a relay is where you have a main campus in a room Old Campus on the main campus again you're trying to make most effective use of bandwidth so you have a lot of clients out on the remote campus and then a lot of clients on the main campus that want to view this webcast and then you have an encoder I've labeled as a broadcaster in this slide so the broadcaster sends creates this SDP file sends a stream off to the main campus server the clients on the main campus would just build a tune in on an FTP file they're seeing what's called a reflected stream they're getting a reflected stream off of the broadcaster this allows you to scale up because most of these webcasting products you know encoders support say one to ten clients at the most so you really need a streaming server there to scale just internally to more than you know a handful of clients and in this case what we've done is we've sent we've created a relay on the main campus server to send the stream to a revolt campus and that remote campus server reflects the stream to all the clients and the value of this is we only have one stream going over the Internet instead of all those clients on the remote campus creating their own connections back to the main campus server they're just connecting locally so that's the value of a relay and they're simplistically I described two basic kinds of relays pull and push so a pull relay what this does is the relay is pulling a stream from a remote source and relaying it locally so this would be in this case that look the relay is set up on the remote campus the remote campuses we're pulling it from Merritt in this case that's what I call it from Merritt from a server called mi scream that one and I give it the path to the FTP file when you do with this kind of poll rate you have to authenticate to the main server using the administrative username and password of the streaming server anybody that's in the admin group so it doesn't have to be the administrator of the server which would be I think a bad idea it can be anybody that's in a QT groups file listed in the admin group so it doesn't that user can be not a real user on the system this is one of those times when having it the users not as real users comes in handy actually so this this is the remote campus server it's pulling in this content and relaying it locally so that's the source the destination is itself it's very common for either the source of the destination to be the loopback address so in this case the destination is the server itself and we're calling the giving it the same file name because it is the loopback address we don't need to user name and password we don't need to authenticate because we're talking to ourselves we figure that's secure okay then the other kind of relay is a push relay and this is also very common this is what we've used in Michigan so this is when local streams are pushed out to another server so you have a stream coming in to the server and we want to push it out to multiple destinations so in this case the source is the loopback address and this file name the destination is a remote server and there are a few different types of destinations we can use here putting out like little crib notes here there's unannounced UDP is what this is setup so a specific IP address and port number this destination dress can be a multicast address as we'll see in a little bit it can be a unicast address or it can be announced UDP announced UDP is just like the broadcaster so it needs a username and password it's automatically going to generate the SDP on the destination so it acts basically just like a live broadcast would act so beyond the basics a relay can pull from a source and push to a destination a real common scenario for this would be let's go back to that scenario where we have the remote campus on a remote campus we have this server out there and if you saw my picture it what it was doing was it was pulling the content from the main campus and relaying it locally as a unicast to itself so it was just sending it back to itself so that people could tune in on that but what if we wanted to relay it as a multicast to make even more effective use of bandwidth then what we would do is we would pull it from the remote server and push it to a multicast address on the local network so you can pull and push but as I said it's very common for either the source of the destination to be the mission itself we have found that when you respect when you're using multiple destinations as they're doing in Michigan that this unannounced UDP provides the most robust configuration the main reason is if if the relay server becomes unavailable for whatever reason and then becomes available again clients automatically reconnect when you use when you set it up this way so the unannounced UDP because the packets are just flowing out on the network if there's an interruption in the flow as soon as the flow is restored the clients will continue to receive the stream without any with that you know hands off whereas if you're using an announced webcast the clients have to reconnect because with an announced webcast that SDP file gets deleted when the flow is interrupted and it goes away so that's why the clients need to reconnect you can also reflect a multicast source for the streaming server so this means if you had if you had a rack of servers each one serving a different subnet you could multicast one stream to all those servers from your encoder your broadcaster and each one of those could reflect those as unicast off to different parts of your campus and the relay destination can be a multicast address and we're going to take a look at that one so this is the most sophisticated set up that I'm going to go through and then I'll turn it over to merit so in this case what we have is the blue lines our unicast and the green lines our multicast so in this case what we want to do is take advantage of multicast to to again reduce bandwidth usage on our networks so we have these great multicast enabled networks you have to have this your network multicast enabled bringing this stuff to work the internet in your one is not multicast enable so this won't work over the Internet it does work over net two however it is a multicast network so in this case what we're having is the broadcaster is sending a unicast stream to the main server the main server then is reflecting that unicast or relaying it actually relaying that unicast as a multicast on the main campus so now instead of sending out in this case three different streams we're sending out one stream and people are just tuning in on that multicast we're sending another unicast stream to the remote campus server and the rural campus server is doing the same thing it's reflect it's relaying that incoming webcast as a multicast so let's take a look at the configuration on this so again using server admin this is how you would configure it on the main campus server we have a source the source is ourselves and it's this file coming in from the it's a stream coming in from the encoder so the encoder has created this OTO UDP FTP file and we're just bringing that in as a source the destination is a multicast address so I've used one that's in the experimental range here so there are assigned multicast address and there are the 239 addresses are basically wide open and you can just use them to play with so I've used the 239 address and I'm sending it as unannounced UDP on a specific port number so now I'm multi casting as soon as I turn on to simulate a multicast about how the heck do you tune in on multicast well this SDP file from the broadcast source this is what it looks like this is a typical SDP file there's a couple things you need to change in this SDP file to make it a multicast SDP file so what you need to do is there's a line here the C equals in line you have to change the address which was the address in this case of the broadcast or you have to change it to the multicast address and you have to change the port numbers book for the first M equals line change it to the base port number you selected as your destination and then add two for the next track the video track so now we have a multicast FTP file if we put this on the streaming server and people tune in on it the streaming server reflects the multicast as unicast which is a useful thing but not what we want in this case to actually tune in a multicast you need to put the FTP file on any web server and that's the easiest way to do it and you can just drag and drop it on quicktime player too but or email it or whatever you want to do but the easiest way is to put it on a web server and then people use the HTTP URL to connect to the multicast FTP file on so that on the main campus server we're relaying also to the remote campus so in this case what we're doing is we're sending a unicast stream from the main campus to the destination which is the remote campus so the destination is that remote server and we've decided again we're using unannounced UDP because it's more robust and we know that if the stream somehow gets interrupted this will continue to go as soon as the things start start back up again on the remote server we're going to set up the same thing we have on the main campus over we're going to tune in on that incoming stream so the source is ourselves and it's just on that IP address listening to specific ports so that it's unannounced UDP we're saying look at ourselves look at these two specific ports that's our source we know we're getting a stream there we're acting as basically as a client to that stream and then we're going to set the destination and in this case I'm reusing the same multicast address because these net are two separate networks I can do this and I'm using the same multicast address that's a good thing to do in this case because then we can use the same SDP file we don't have to edit it again we can just use the same file we already edited put it on a web server and it just works so that's a more sophisticated one I have a website that tries to untangle all this stuff for people and to help you you know do more sophisticated with this stuff with the streaming server it's just sound screen comm so if you go to that website you can get touch me through that page also I've got a lot of instructional information on how to the how to's on all these things so with that though I want to turn it over here to Lawrence or Lori Kurt Meyer who is the senior engineer of networking R&D for merit network and Patrick by mold who is the assistant engineer that supports education in the state of Michigan and I've worked closely with them on their statewide streaming and initiative hello where as as we get into this Patrick and I have worked very closely over the last year putting this together and so we thought that we would sort of tag-team this presentation because there's parts that we both worked on so we'll we'll see how this goes as I say I'm a senior engineer with merit network merit is a statewide network for to the educational institutions in Michigan and the 15 universities are so that the board members of the merit board and we have over 200 affiliates connected to the statewide network those affiliates being the is DS School District's hospitals libraries and so the objectives we have here were to provide a set an infrastructure for statewide live webcasting the impetus for this was from one of the what's called the regional educational media centers were had a broadcast studio and they were doing satellite broadcasts of administrative broadcast to school administrators on various topics and like No Child Left Behind and that's one example and they wanted a way to provide that broadcast to users without them without without the administrators having to drive to a central location to watch this satellite broadcast or have it or having to go to receive it over a h.323 stream to a to a conference or a video conferencing room so the so what was a what happened about a year and a half ago was Patrick came in and there was one broadcast where they wanted to explore new streaming technologies and so Patrick suggested using a photo streaming server so it's actually kind of a funny story we were doing broadcast statewide we're using Akamai cause apples got a great relationship with Akamai and we couldn't fully support the projects the way we wanted to some merit not merit Oh Wayne RESA set up a streaming server during their broadcast and it was working just smashingly up until the point that the network manager for the organization walked in and he's got a map of the network you've bandwidth usage and he's seeing you know ups and downs and then there's a huge spike when the broadcast come on because they've got like 60 or 70 people that are turning in the broadcast and its network manager goes that's unacceptable dad needs to change which we all do is where in our work manager because we have to make sure the traffic and flow perfectly so we started having conversations about the possibilities and I knew the quick times got to me one of the most amazing streaming services available and I said cool let's set up a statewide Akamai and yeah this is where we're going to start talking right so what happened was we'll go to the next slide this is the set up where Patrick installed a streaming server they try and point with this thing over at what's called Wayne RESA this region educational service association they installed one QuickTime streaming server for one of these educational broadcasts that was being put out by satellite and they simultaneously took a feed and put it on a quick toast broadcaster and started this was a combined machine running the streaming server as well and so this is the merits network mish net it's the name of network and we have various affiliates around the state of Michigan who were tuning in to this and so what happened was the the stream went across this this one link from Wayne RESA to mich net which happened to be a ds3 which was fine initially until they started to get many people tuning in to this point and completely overloaded their outbound connection to to mich net and again their network administrators were hammering on the door saying what the hell are you doing or what is what is causing all this famine so it turned out but also with this solution where this stream was going out the posish net there were similar trope points adverse affiliate networks where multiple people were tuning in some of our some of the school districts that are connect them go back some of the school districts that are connected get a thank you oh well get that I'm pressing too fast I'm sure so we copy you in this point no that's right okay that as I say some of these school districts are connected with a t1 and so you have four users on there and you overloaded they're inbound connection to their network so the solution was to deploy a hierarchy of streaming servers with let's go to this line and so that's what we did and so how now we can have a broadcaster at any location on the mission on on the network on the internet basically there's one single this green dotted line is a single stream that is central master relay server and merit network we we're providing the colocation space and so the bandwidth free bandwidth basically for this master relay server and so this master relay server in turn just read redirects the broadcast out to a number of relay servers in the different school districts and around the state of Michigan so that's minimizing the bandwidth going across those expensive links or for the school districts the expel links to our network so again with regards to multicast and unicast there missionaries the unica multicast enable network and so wherever we can we would like to to be able to send a multicast stream to the relay service and we're working with our affiliates to see whether that can be done so now we're going to talk about leave start talking about the things we've learned this has been an amazing journal of journal journal over the past year ok more yep we I'd like to take the opportunity now to thank the person that has more information and is head of a QuickTime streaming server which is George Koch than I think anybody does so if you get a chance to take a look at its site but we ended up with right now we have 50 16 locations 15 locations in the state of Michigan including copper country it was a nice seven-hour drive up to Houghton Michigan to set up that server and the thing that's been most informational about this whole project is working with 16 land managers right because now I have 16 people they're responsible for their own networks every network has been completely different and so we've had some serious growing that we had to do so go on does it so ok are you yeah there's something you want to add just that talking about working with these 16 where manages the other the other interesting piece to this is that if you tend to tend to have several people wanting to deliver the content on a campus you have the content folks or the who are wanting to get this stream in and then you have the networking why'm folks and deploying a quicktime streaming server tends to bring those two groups closer together very quickly and that's something which is good I mean it's definitely an education for the network administrators so and as a Patrick mentioned he has been doing the legwork driving to all these locations installing these service has done an amazing job ok quickly about the the architecture some of the things some of the things we want to do is make sure we could broadcast from anywhere we want to be able to eventually have students that are working on projects and in the educational institution anywhere in Michigan broadcast right we've got a program we're going on at 11 o'clock or program going on at noon we have actually had conversations about universities doing school nights so instead of students having to go to the University they tune into the school night event is taking place in their local school district from em I assume that so broadcast from anywhere where we will have the parent node what does that say push information out to the relays regarding the content of the video which you will talk about in a minute but I've got this thing now that I call the Microsoft effect and it's bad for me to say that nobody's from Microsoft right we're an apple developer cops like to say this point but this whole notion of servers do a good communication in terms of announced relays over ICMP right ICMP huh no the pinging between the yep and so the some organizations were actually ICMP has been turned off right and so we have servers that can't talk with each other anymore and they just oh I'm going to stop broadcasting now because I know large if you talk to it to the child replicate we're going to talk about replication of content over the server's over our sink and then the thing that Laura's going to get to is how I made all this thing come together the Akamai effect with the redirection script the children notes we wanted them to be able to have access to the data meaning the video coming in from the streaming server via the FTP file and when you starting is anybody running streaming servers everybody the thing that I have to say is don't make this any harder than it actually is for some reason the hardest time we had in setting up the service we were setting up is we thought it had to be harder math we thought maybe we needed a pocket protector or some incredible slide rule something that you know what would allow this to make sense to us but really this is just very very simple in the way that it can be configured ok the last thing that I think is from merits perspective being a ISP and we were concerned about the whilst we've got these 15 relay servers around the state and users get redirected to pull their streams from these local points of presents in their various school districts we were concerned about how many other users might tune in who I don't have that but don't have a local relay server and so what we put in as a second QuickTime streaming server in the merit data center which would take which would be the catch-all server for users who are not redirected to a local streaming server in there is D and so we're sort of working to see how much how much bandwidth we're willing to donate for that and how much demand there is but again this is a useful tool because we can begin to see if more users come in from another school district we can begin to go to that target that school district and say hey if you put a streaming server in your district then you wouldn't incur this or the bandwidth costs across your you'll link to our network and there'll be a great benefit okay so the one note that we have is we've limited the shunt servers so with allows up to 50 users or anybody that's outside of the service area and really what we want to do is if it's a 50 first person goes and clicks on that street and that shunned server what we should really do is is put up a web page where it says you should go by a quick time still get your network first or to buy a QuickTime streaming server and put it in your school district and then you could watch this so so some of the stuff we learned the first time I set this up will use denounce relay which is the easiest way to set up a relay network and if anybody's looked at expanding the networks that are doing hierarchical broadcast broadcasting but announced relay very simple put in from QuickTime streaming server I'm simply setting up a actually my travel that nemac in a minute the first time we set this up we'll use announce relay so let's go on to the next line okay and in announce relay the parents responsible for getting the content out to the children servers what I call children servers so they know what the broadcast is coming from so and parent announces the FTP file and each one of the children since tune into the STP file because it's sitting inside their movies slash library slasher movies directory next and this was a great solution it's very very simple in setting up a distributed streaming server network we also found a very interesting feature built into a streaming server in that if if one of the children was not available the streaming server would simply say I can't broadcast anymore and restart itself every two minutes so we ended up with these live broadcasts where the video is being restarted every five minutes over and over and over again yeah we just don't assume it so you tune in you see a little bit more and then it would stop yeah and so we had two things that were taking place firewall configurations right or the again the Microsoft effect which goes back to firewall configurations because there was no ICMP so one of the things I would recommend in setting up a distributed network if you're going over firewalls is make sure that you have a one to one relationship the easiest conversation for me with the network managers the way managers was set up your parent set up your child and set up the firewall in between so that the two just can do anything they want to do with each other there is no restriction there is no port mapping it's just a completely open connection between those two devices on the network so we worked at this for a little while and we continued having problems so we switched to unannounced basically give a name give a port number so the panthous is on the parent server the parents simply get a relay set up and we'll see some just a sec and it simply says I'm sending data to this port number at this machine just it doesn't even care if the machines out there it's just here's video you're going to tune in to it you can watch the video to your heart's content if not I don't care really I'm just giving you a video go on so this is the way the network is currently configured I didn't mention the first part we have quick temps broadcaster running on a machine a twink on your ISA right now and we've simply taken the FTP file off QuickTime broadcaster exported it and put inside the movies directory on the and each one of the children have been set up so that they simply tune into the SVP file and we're going to look at the SUV file in a minute so you can see what that looks like it's again very simple we thought it had to be hard Patrick white and I hardly ever use Excel I use Excel and I got the spreadsheet that just would cover this whole stage and it's got this flowing set of formulas where you can figure out port mappings oh my gosh I would open sources saying or so amazing and it didn't need to be that hard rank so I I still have it always have a unique last nighti file for all right go thought about printing it but I really don't need it now so anyway so we copy the broadcaster file up to the server each one of the relays are configured inside the streaming server and we'll look at that and then video goes down to the children notes so this is what we're currently configured and this has been the best experience that we've had fun one of one of the things to note I like to think of it the way this is set up is that we thought that the route the SDB file is the key if you don't have the SCP file on the child relay server you know the the master relay the parent server is sending the data down and it's only when you get the key that key on that local relay server that but the stream actually gets into that local relay server and users can watch it and so it's a good sort of analogy well I went to myself what's up oh so some of the things we like about this the fact that if the parent ends up going away the children don't lose the connection to the viewers that are watching the broadcast when a parent comes back online if for some reason it goes down the the children will automatically pick up in the broadcast that they were showing to the to the viewers and the same SC this is a huge note with the little pointer up on the third line up there the same FTP file we use for children to tune into the broadcast can be distributed throughout the children so with our sink we're actually looking at the ability to say we're going to have 20 channels and channel 1 equals and with the information that we're going to have regarding localhost address 127.0.0.1 we will distribute out throughout the entire network affecting all the servers and we see this thing growing we've got 17 servers 16 servers now we get two more coming online so it'll be nice feel to have this easy configuration be replicable throughout the state next so let's talk about the way this works on the left hand side inside the streaming server admin is the brought the relays themselves and there's some information you associate with the relay itself and then next is on the right hand side the actual servers that you're redistributing this broadcast to so go on okay I have a clicker but it doesn't work with that thing what I wanted this so if you're doing oh so we set up the service so we have a really named doesn't really effect in terms of what the user is going to be tuning into we have a relay point basically zoda's is at STP and this is the mount point of the broadcast of connection point and the server ends up pointing it back to itself because we have the broadcaster sending video to a specific IP address on the network we're not going to share it with you because it's it's not fair for us to do that but we tell the server tune in to yourself to watch the broadcast because it knows that this is near to do it that STP file has all the information it needs to build to get to the broadcast after sending one to the next one and then when it's been within the relay setup we have each one of the servers defined either by IP address or by name this actually doesn't have to be a name that is a resolvable by DNS it can actually be a name is sitting in the host file and the port that this is going to be broadcast to and as I mentioned before we're talking about channel 20 this could be that all the servers for channel 20 are sitting at 21 326 throughout the entire network so the server simply says I'm broadcasting to this UTB base port going to the next one so remember this number you d be base port actually ends up being two ports 21 to 24 inch 326 is the beginning port and we notice we assign this to audio we didn't QuickTime bit and then video is assigned to 21 328 this does not mean you can start messing around with the port 21 327 alright basically three ports for broadcast is what you're going to be using and I tried to get crazy with the spreadsheet I tried to figure out a way to maximize my number of cords because when I was thinking I had to have a look at George laughing he had a fun time with this he really didn't tell me this and knew he didn't tell me I'm making that part up but I started to get crazy with how is configuring the pores and it turned out that the 327 is a I don't know Cinque Ports something that is used by the server in terms of communication so this information really isn't pertinent to what we're talking about regarding the broadcast talk to myself look at these ports and then start spreading video to the viewers that are wanting to tune in one of the excellent the cost outlines I mean we can delete those out of the SPP file which were the unique it turned out that go back one it turned out that these three lines right here are extremely important and George can talk about that I didn't have them and we didn't have a broadcast and of course he's like oh you're messing me three lines guys scary and okay okay I'll talk about live every direction - as we said we want to redirect our users to the closest streaming server local in their to their school district and so what we've put together is the CGI poor script and what will be a database right now is a flat configuration file which contains a mapping between a set of IP addresses IP address ranges and for that School District and then the load the address of the local relay server to redirect the user to and so the users can open up a URL causes the CGI to the post script and they automatically get xml data back to quicktime player or to quicktime embedded in a browser and telling it to open up the stream from the local server we also added what i worked on was some additional logging so we could lovely as users get redirected or do not get redirected so we could begin to see how how successfully we were on how well the coverage was of local relay servers and again we could have the ability to redirect users to a live stream to an SDP file on a local relay server or to replicated movies or mp4s or whatever media we have replicated on the local relays this this redirection script will work with and with any of those this is a just a one-line example we've just got the the child IP address that the relay address and then the client address range start and end and as the script is called essentially this is you can see there's a curl statement we're just showing that when the script gets called by the browser you can use curl command line 2 to make this call this is what gets returned at XML and this is passed to good time either the QuickTime Player or to QuickTime inside of the browser and does the necessary redirection passing the redirection to the local relay IP address followed by the the name of the FTP file and we just this is showing the embed come on how you would embed the call of the object embed command to to embed the call this is this is our master relay server and we're calling this poor script my stream net dot pearl with the BSD name of the SPV file and what results from that call is this is this redirection to the local server and [Music] we're going to talk about what so what we wanted to do was as we rolled out these servers we wanted to do a whole set of testing and load testing as much as possible to make sure we that it would run under under load certainly we wanted to stress the redirection script and so we developed or there's a one a Python script which George put together to to stress test the redirection script itself and then we also use streaming low tool which is part of QuickTime streaming server is it part of the release yeah it's on every server isn't it I don't know so John talk a little more about so two things I wanted to add so we've been talking very positively about in my stream net and it worked incredibly well the last two broadcasts that we had but we actually were at a point where we had four four broadcasts and and all of them ended up with some part of the network falling apart so we we were needing meeting to find ways brought George out to figure out what the heck was taking place and if anybody thinks that my closed ten server and QuickTime streaming server and a neck serve is not an incredibly powerful tool to use I just have to let you know what we saw these things capable of doing em I assume that our parent streaming server we were simulating five hundred five hundred broadcasts along with seventeen relays and it just ran for like hours and hours and hours and hours and hours and this is great we thought this was amazing we got a 10 megabit connection to the network or a hundred mega 10 mega we got 100 megabit connection to the network and we're you know we got 500 users and watching this video plus we're doing video out to relays I think overall we simulated two thousand two thousand three thousand users on the net because we started stream loading each one of the relay servers or the children's service out on the out on this day and it was great they worked the only problem was we had not gotten to the point where the server failed which is what we were trying to find until the day that we got Ken Tramp who's been the most wonderful person to work with he grew up in the video world right in the video world if something fails you get rid of it no there's no three strikes and you're out if something fails in the video world has just done dead don't talk to me anymore and we were bringing them up in this electronic world which is a little bit different and as soon as I got count on the phone and I can check us out we're doing these broadcasts working is great we got things figured out cuz we reconfigured some stuff and as soon as he got on the phone it's like oh that looks wonderful BAM and my student 1 goes to help just and we had to go actually restart the machine well it turned out and the reason they're bringing this up is assuming load tool is a great application that comes a streaming server you can simulate one hundred nine hundred you know 25 to 50 users per connection hitting the server in real time so it's sending out the bandwidth each one of those connections and so long story short turned out that we end up end it up with a motherboard or a motherboard on our server that had problem only related to power changes that were taking place in the environment that it was in so it was like one of those things you would never find but we ended up with a guy putting a new server in the server room that we had our MIT no one in and he turned on the server and that little switch changed the power enough for the for that configuration the server just like come out of here done see you tomorrow so we got a new motherboard things have been working perfect ever since but I think that what going back to you talking about working with this this video production person I mean with any of this when you pull out a service like this I mean you it's important that it that it works I mean I did it it's hard you don't get many chances with this you want to make sure when you roll this out that it runs because if you're dealing with video production folks from that world they really have a very low tolerance for for failure coming in so it's important to test and so assuming roll to low till we came a great solution the Python script available at the site that you have so basically the Python script is what we use to simulate a hundred users per second hitting the redirect script to make sure we weren't having some kind of memory problem with Apache or the script we've written to do the redirection and the thing that ended ups in the end giving us the information we needed was the server monitor tools anybody using a neck serve the server month monitor tools are incredible for what the information it can provide some things that I would recommend all services we're running on the servers anything that's not necessary turned off so anything that's taking place in terms of file copies we're doing over SCP we've actually set up firewall rules so that the things that we need to be public like 12:20 we're gonna have that open to the world so we get to it from anywhere 660 is what we actually have for server settings on 10 - server we're going to be just we're going to be up to 10 tree by the end of the summer which would take us back to 311 and then a tweet because we're using SCP and SSH we actually have the firewall set up so that there's only one machine on the internet that has the ability to administer these servers over SSH which we're not going to tell you because you might try and get into it so I would highly recommend it to not run as much on the serve as you possibly can lock this thing down because it's sitting on in a public place and this thing doesn't work what's the next one this is where George talks it is what Oh sight looks like it's with with both Darwin and QuickTime streaming server and it's a great tool for stress testing and these folks said the main thing here was testing you know and - and I see some other folks I know any audience and and they know that that's that that is the critical component here is that when you're rolling out something a new set of services on the network especially something new like streaming important thing is do some stress testing you know make sure that you've got things under control before you know you do a broad broad scale deployment so some of the things we did was we each instance of streaming load tool we've set up to do 50 concurrent clients we have multiple instances running we have three different machines running multiple instances and we've loaded the heck out on the server and we're amazed at the performance and I've recently assignment again from Wirecast was showing the Swedish newspaper that is really loading the heck out of a server and again I noticed wanted to talk to about what they're doing there even more amazing really ok so long do some 10 runs replication these are the next steps so the next things we're working on is replicating content necklace 10 the UNIX layer has great replication technology built in called our sink so this is an our sink command that you can that we're using right now in a test mode to synchronize content between the servers the important thing here is that this synchronized stuff all works with the redirection script so the redirection script doesn't care if it's an FTP file a movie file mp4 file whatever it is it does the redirection we're also working on some monitoring tools and Laurie is really excited about putting some of the stuff into a sequel database I've just been doing flashy QuickTime stuff so far so this is something we're working on for local administrators the idea is the local ran guys that are up on these uh the in these school districts we want to provide them with a tool that they can use to monitor the server but not give them full administrative privileges to the server I think again monitoring especially during the testing phase being on a monitor and know how many streams are actually running on this little child relays if something does go wrong you've got some logs that you can go back and really begin to figure out what was the state when yeah we're also Patrick and I are planning on setting up Moodle which is open source course management system very nice one beautiful macula sensor for all these people that are becoming involved in the project so as more people become involved the faculty member want to know how to produce content places on the server there will be a website for them to go to to get educated on that and so that's another work in progress cute QuickTime trimming server publisher we're really excited about the new version one of the new capabilities that that's coming in the next version is the ability for end-users to use the tool right now it requires an administrative user which limits the scope of its use but with the capability for end-users to basically publish streaming content or downloadable content to their own site streaming folder through this tool is just phenomenal we're very excited about that use because it will it will extend the use of these child's own servers so that they can be used for locally hosted content so we think that's a really great thing so that we'll do a few demos and how are we doing on time here hold on we had worked 15 okay let's bring up the demo machine we have it so might stream that this is a the folks at Wayne RESA who are the third partner of my stream net with with merit and Apple have got an amazing web designer who's putting together this with this website and this is the live streaming page of the website and so we're giving the users ability to view the stream within within a quick time in a browser window my only cat let me make sure so ya know that definitely come see what we're streaming here right now is as a sort of a testament making the the Annenberg video at Annenberg educational video which is being streamed in over satellite and they're taking this feeding it through QuickTime broadcaster at Wayne RESA and then streaming it out over QuickTime over my stream man and so we have a great ability to test at any time and this is this is it's a good promotional piece for for new users in the state of Michigan to see that also education and so the other way you can also click and view it in a quick time player window the other piece we're also saving archives maybe this isn't working and so we have all the archives our previous broadcasts that we've done over the last year and users can view these at any time so you want to show the air so are we working here get a dinner night here myself so another thing that we've been looking at is some of these Administrative Tools so what I did was I created some some wired QuickTime movies to talk to the thing called the admin protocol on the server so on the back end of the servers is this thing called the admin protocol which is used to do all the administration and so I wrote them see guys that talked to the admin protocol and then wired them into some QuickTime movies this kind of sounds crazy right so the first thing we'll do is we'll bring up this this little the graph thing so this is graphing bandwidth usage coming from the server in real time and I can change the interval of the graph and I can change the bitrate but while it's doing that let's go ahead and we'll start off a stream from the server so we have this movie so we'll go ahead and turn it up and you see it as a fan which jumps up now it jumped up a lot there in the beginning that's because of Chris because right because we're we're doing the skip protection and we're sending a whole bunch of packets down and then it will start to settle down as it you know as the buffer fills so you know it's kind of a cool thing to be able to watch the bandwidth in real time right so that's one little hack the other one that that and what's neat about this is it the admin protocol with this all this stuff is talking to Lori's really excited about logging all the stuff into a database I like the cool woodsy demos myself so so how's the demo as well so here's another one that we've done so I'm going to go ahead and set this to one minute so it's pulling and now every one minute this is the number of connections RTSP connections or we can look at HTTP tunneled connections we can look at the RTP bandwidth so let's crank this back up to five seconds so now we're looking at the individual nodes around the network and seeing what the current Pamela is in this on the main this is the law the main server here so you know just interesting things that you can do with wired quick time so thank you totally hip for your tool you can squash you to do it and then we actually have a this web page but I left the password over there if you want to do the web page it's just talking to see guys and you can actually drill down on the individual session you can see what IP address somebody is coming from and what piece of content they're watching with the admin protocol so you can get that level of detail with the Stabbin protocol very exciting in terms of how we log this stuff how we tract in the future okay so I think that's it for us for right now and what I want to do is invite up John Anderson and John is on the streaming server engineering team and he is largely responsible for QuickTime streaming server publisher and there's a new version coming as I said what's exciting to us about this tool is to really leverage these child node servers to be able to take those servers and in the School District allow faculty to give them a very easy user tool to publish their own content locally hosted content to these streaming servers which is something we found very effective at a lot of schools and now we have a great new tool to support it so thank you that's a little weird with the microphone it's like karaoke or something I'm going to go ahead and close this as it was mentioned before one of the big features and cue CSS publisher two is hello okay as that's was mentioned before one of the major features in chief justice publisher 2 is that anybody can log in to their home directory and publish content they don't have to be at an administrator so that's a great new feature I'm going to go ahead and start this up in login there's also been a lot of great performance improvements in the application and it has a new UI now so if anybody is familiar with you just as publisher one you had all these different modes that you had to go through where you were uploading your media in one mode and then you go to the settings and set that up and you go to links at this point everything is all in one place another thing that's that's nice this new is if you're binding QuickTime streaming server to specific IP addresses so that for example you can run on port 80 and then run any web server on another IP address you can actually pick which IP address you want to publish files through so that's really nice also if you have multiple virtual hosts set up in server admin for the web server you can pick any of those sites here as well and then of course if your administrator you get a choice you can work either out of this the system library or you can work out at your home folder as well so that's kind of a nice feature and so what I'm going to do here is I'm going to go ahead and open on this up and upload one of these movies and it's real simple all you do is drag it into the library that uploads the movie and in addition it will hint the movie as well so labs interactions a movie automatically on the server so now what I can do is I can click on this movie now when we put files into the library they actually go into a location outside of the QuickTime streaming server folder and that reason for that is that you may want to make a movie available only via live simulated live broadcasts or playlists and so that gives you the opportunity to do that but what I'm going to do is I'm going to go ahead and check this checkbox which is publish media to QuickTime streaming server and when I hit apply what this is going to do is it's actually going to create a symlink in the QuickTime streaming server folder it will also go and create a referee that points to the to the to the movie on the streaming server so does that all that automatically you get down over here you get a URL that you can copy and paste into the player and you can also just click play and the movies have been running so it's very simple streaming and then of course if you want to set up a simulated live playlist which is great for relays something that you can then relay off to other servers all I have to do is click on this plus button and I'll just name it the latest much like iTunes you just create the playlist and drag the file to it and that's it if you want you can change weights on this so you can do a weighted random playlist and that will actually play some some tracks more than others and then just start it now the cool thing is I'm going to go back to the file not only is it publishing those files and making them available you can change the annotations anytime you want and it already has a movie when it's done and in addition you get down here you get all of the HTML to embed the page to embed the movie in a page so that basically you can choose whether you want to for example open the movie include templayer have it autoplay and once you choose that you have HTML right here that you can copy and paste into Dreamweaver or BB edit or any other web page editor in addition to that I can actually create a web page directly from here this is the development LS that is may now work a lot to see but I just go ahead and create a page and we'll drag the file to it and this will actually generate a page based on a nexus LT template that has the movie in it now of course I haven't set a poster frame so it's just defaulting with the first frame if I click on that then that automatically brings up the movie so so that's publisher [Applause]