WWDC2004 Session 709

Transcript

Kind: captions
Language: en
thank you so yeah as Steve said we're
going to have a cast characters up here
and you forgot to mention one which is
John Anderson at the very end is going
to do a demo of the next version of
QuickTime streaming server publisher
which has some great new features that
for the project that we're going to talk
about mi stream that they're very you
know excited about the new product
because some of the some of the new
capabilities so the agenda is first me
talking about some of the how-to stuff
setting up authentication streaming
through firewalls etc and then we're
going to go into this case study on the
merit network so merit network they are
a non-profit provider of Internet access
for most of the schools and universities
in Michigan and so they're going to go
ahead and and go through details on the
history of the project some of it
objectives how they set up relays how
they've set it up so that to redirect
clients to the right server some of the
testing and configuration and then
future plans so let's dive right into
some of the specifics on how to so one
of the things that a lot of people want
to do is is set up authentication and
there's two main reasons to do this one
is you want to authenticate who can
broadcast through your server so with a
live streaming product who can actually
send streams through your server and we
have this thing we call automatic
announced broadcasts that save a lot of
the hassle of the way you used to have
to be done in terms of doing a live
webcast through the server so one of the
reasons for setting up a foundation is
for for providing a friendly way to do
these automatic announced broadcasts
then there's also just the
authentication to the media itself and
authentication that we've implemented
RTSP authentication is very similar to
HTTP digest authentication so the
protocols are similar and the
implementation is also similar to what
you would
in Apache so just like an Apache where
you have an HT access file for
authentication in QuickTime streaming
server you have a QT access file for
setting up authentication to a specific
directory on the streaming server and
just like Apache with HT access you have
files flat files on the disk that have
the users and groups in them so let's
just take a look at this for announced
broadcasts in QuickTime streaming server
we have a GUI for creating this password
and I've highlighted them out here on
the screen you would click on accept
incoming broadcasts and click on set
password so that allows you to create a
username and password for an announced
broadcast from a live encoder to the
server on the the client-side I have two
different screen shots here these are
three graphical broadcasting clients
available today that support announced
broadcast QuickTime broadcasters the one
from Apple it's free and it's you know
it's very simple in its capabilities one
that I'm really excited about there's a
new product and Simon Clark is in the
audience here I encourage you to maybe
catch him afterwards he's got a great
new product called Wirecast they uses
quartz transitions supports multiple
video inputs it's an incredible piece of
web casting software and their
screenshot is on the right so they also
support this idea of announced
broadcasts so basically what you do is
you provide a host name a file location
you don't really have to put the SDP
extension in I like to put it in you
know it depends on if you it's a good
idea to add the SDP extension the user
name and password that you've set up on
the you know in the previous area and
then when you webcast what this does is
it automatically generates this file on
the server that's necessary for a live
webcast this SDP file session
description protocol file so in the old
days which still works the old way you
can do not an automatic unicast
manual unicast which means you have to
manually export the FDP file and
manually transfer it to the server that
way still works and that's the way that
for instance if you are using one of the
hardware encoders be brick jvc both make
hardware mpeg-4 encoders that work great
with our streaming server but they don't
automatically generate the STP file you
have to get the FTP file from the device
and transfer it to the server to reflect
the live webcast so this way is a is a
more is a friendlier way so if you're
out in the field with your power book
and you want to do a live webcast
through the server this is a very easy
way to set it up now under the hood this
all uses these files that we talked
about before so in this example we'll
try this laser pointer I've done this
cat cutie access so I'm in the movies
folder on the server and I've keep I'm
looking at this cutie access file this
is what's in it limit write require a
group broadcaster slash limit and then
require any user so this one means that
to write to the server you have to be
this username broadcaster require any
user means any other user can just
connect to the server so there's no
authentication anybody else can freely
connect to the server to access the
streams and then this cat command is I'm
tatting the Qt groups file and actually
here's required group not user so
anybody in the group broadcaster will be
able to webcast to the server so I've
catted can this QT groups file and piped
into grep being a UNIX weenie here and
look for this name broadcaster and these
are all the users that are often are
going to be able to broadcast through
the server on you know on in this setup
so this is the if you want to set up
your own authentication you're going to
have to do some command line stuff and
that's just the way it currently works
so basically what happens if you want to
authenticate access to media you are
going to have to modify these files
okay so again I've looked at the QT
access file this time I'm saying require
group users that means in this folder I
have the special directory off of my
movies fault directory called
authenticated so now it's just like
Apache it's anything in this directory
will now be required n ocation because
I've got a QT access file in it says
require group users so now you have to
be in the users group to be able to
access media and it's going to put up
this login dialog box when you try so if
I look through my QT groups file grab
users these are the users that are in
that group called users and then here
I've shown what's the contents of the
directory are so what happens with
QuickTime Player you launch the time
player you type in the URL to that
authenticated media and then it's going
to ask you for a username and password
before it streams the content to you so
to create these users there's a command
QT password which allows you to create
new users in the QT users file to create
the groups it's just a text editor it's
very simple format it's just the user
the group name followed by a space
delimited list of users so these files
you can have other QT groups and QT
users files they don't have to be the
default files that are in the config
folder of QuickTime streaming server
they can be other files so you know you
can set up your own little we see in
higher education in particular where
they want to give a faculty member the
capability to create their own their own
users and groups file and that works as
well so this is kind of the old way to
do things with that we've been going
over which is what apache has and what
quicktime streaming server has a lot of
folks want to tie into a more
organizational authentication
architecture and quicktime streaming
server like Apache is written in modules
so there are modules that can be created
or modified to do things like customize
authentication so one group that has
done this is NYU so NYU is involved in
to project called Shibboleth Shibboleth
is about cross realm authentication
which means that for instance the NYU
library might want to trust people from
another University will say Cornell
University they might say well we've got
an agreement with Cornell library so
that we can both access each other's
stuff and so should well if that's what
it's about is this cross realm
authentication and if you go to the
Internet to webpage you can read all
about Shibboleth they have written an
Apache module because it's modular - for
Shibboleth that allows you to do this
with websites
so what NYU has done is taken that
infrastructure and extended it to the
streaming server they didn't do it
directly with the streaming server
because the Shibboleth authentication
they use it's not just a simple
challenge in response kind of dialogue
between the web server and the client
there's a lot that goes on there and
that sophisticated dialogue the RTSP
authentication was just too simple for
that it was didn't provide them with all
the infrastructure they needed so they
set it up with Apache so basically the
flow here is you know you request the
URL from Safari your web browser it goes
through this Apache module to
authenticate you against Shibboleth
which does this cross realm
authentication so Shibboleth can talk to
basically anything LDAP Kerberos
whatever you want and then what they do
is once you've authenticated it talks to
a my sequel database stores the IP
address of the client the cookie the
ship cookie and timestamps it and then
redirects the to this custom module
they've written for QuickTime streaming
server because it's modular they've
written this custom module that
understands these special requests these
special URL is the special URL comes in
and then this custom module goes back to
the my sequel database and says okay
what IP address are they coming from
what's the ship cookie and when did they
do this one so you can't replay these
things and then it says yes or no and
delivers the streamer doesn't deliver
the stream to the client so this is an
example of how a customer has taken the
open source part of the streaming server
the
Arwen part of streaming server
customized the module so that they can
deliver a custom authentication and
we're working with NYU to try to make
this piece available through the Darwin
CDF so it would be available to any
QuickTime streaming server or Darwin
streaming server but you have it I have
all this other stuff working too for it
to work so we're in the progress the
process of trying to figure out how to
make this available to a wider audience
then then NYU right now so pretty
interesting stuff
next topic so we're done with
authentication let's talk about
firewalls if you have a publicly
accessible streaming server you should
turn on for it ad streaming and this
means that you can't run a web server on
that same port on that same IP address
on that server so it means that you
might want to bind your streaming server
to one IP address and the web server to
a different IP address if you want them
both to work over port 80 or just run
your web server on 8080 or some other
port so what this does for daily
streaming is the tunnels all the RTSP
RTP traffic through HTTP port 80 and
that means it gets through most
firewalls QuickTime 6 supports automatic
configuration so it will try RTSP over
the standard you know streaming ports
and if they're not available it will you
know go back to port 80 it's all back to
port 80 and this is very recommended for
these publicly accessible servers
because it allows you to get through
firewalls and you don't know where the
clients going to be in your organization
another thing that you might want to do
is open up certain ports so this allows
you to access streaming servers that
might not have for dady streaming
enabled so minimally you can open up TCP
45 54 and 70 70 70 70 is what real
servers use and we can use it too but by
default we use TCP port 554 for RTSP
which is a signaling protocol to the
server and then UDP is the protocols
used to actually stream the data and you
can open up a range of ports here it
depends on you know how busy your server
is and how many clients have you have
connected to it so that's what sets that
range of how many how many ports you
actually need
a typical range is 69 70 through 9999
and then if you want to administer the
server via a web browser you need to
open up TCP port 1220 and if you want to
administer it through our server admin
application you need to open up TCP port
311 so those are the port's you need to
have open another way that people set up
again inside an organisation security
through firewalls they have a special
partner network called a DMZ where they
put proxy servers and there is an RTSP
proxy server that allows you to do this
and basically what you're doing here is
you're saying the proxy server can talk
to any server on the Internet and any
client can talk to the proxy server but
clients can't directly talk to the
servers out on the internet so it puts
this you know go-between box in there
and we have you know a version of this
that's posted to our Darwin site that
you can download and use as well so this
is another option more of an
institutional option again for people to
create firewall that you know to work
with a firewall probably the least
common this is the least common of the
three so next we'll talk about the
really I think the meat of this
presentation which is scaling streaming
through relays and replication so
streaming relays are our way to conserve
bandwidth really so when you're doing a
live stream or a playlist stream or your
streaming stuff out in real time to a
lot of different clients what you can do
is set up a relay so that that a client
doesn't have to go to the source server
so this is what we've used in Michigan
typically a school district doesn't have
a big you know oc3 line or something
like that to the internet they have a t3
or a t1 even which isn't a very big pipe
so you're not going to be able to get a
lot of 300 kilobits streams through a
1.5 megabit t1 it's just there's not a
lot of bandwidth there so instead of
trying to get 50 people to connect with
t1 on 300 kilobits which won't work
because you'll way overload the capacity
of the t1 you could put a relay inside
the school then what happens is the main
server sends a stream to the relay and
all the clients inside the school tuning
in on the local relay where they have
plenty of bandwidth inside the school so
they provide these streaming points of
presence so this is the kind of model
that Akamai is based on so Akamai has
this capability built in you might have
heard of Akamai which is a
industrial-strength replication and
service out on the Internet but they
also support this live capability these
relays can support unicast or multicast
so what I'm doing right now is I'm multi
casting hopefully you're all listening
I'm speaking so it's one person lots of
listeners so that's what multicast is
about whereas unicast is if we're all on
a conference call
we'd be tying up I don't know a 100
phone lines so in the unicast everyone
makes a separate connection just like we
do on the phone network when we when we
make phone calls these can be used with
live are typically used with live or
playlist' streams they can actually be
used with on-demand media but it's not a
very common use of them another way to
save bandwidth is to replicate your
content and this is what Akamai does in
a big way but there are little ways you
can do this with a streaming server as
well that are pretty interesting and
it's commonly used for video on-demand
moving that content closer to the client
so relays the basics of relays is every
relay has one source and one or more
destinations the source for a relay can
be one of these FTP files it comes from
a live encoder it can be an actual piece
of media and it needs to have an IP
address and port numbers for that source
where is it going to get that
information from and then you can have
one or more destinations where do you
want to send this stuff out to so that
can be an IP address with specific you
know port numbers or it can be this
thing called in an announced destination
which means it will do same thing the
broadcaster does it will announce the
broadcast to that destination
automatically generate the FTP file
figure out the port numbers etc so a
very simple configuration for for a
relay is where you have a main campus in
a room
Old Campus on the main campus again
you're trying to make most effective use
of bandwidth so you have a lot of
clients out on the remote campus and
then a lot of clients on the main campus
that want to view this webcast and then
you have an encoder I've labeled as a
broadcaster in this slide so the
broadcaster sends creates this SDP file
sends a stream off to the main campus
server the clients on the main campus
would just build a tune in on an FTP
file they're seeing what's called a
reflected stream they're getting a
reflected stream off of the broadcaster
this allows you to scale up because most
of these webcasting products you know
encoders support say one to ten clients
at the most so you really need a
streaming server there to scale just
internally to more than you know a
handful of clients and in this case what
we've done is we've sent we've created a
relay on the main campus server to send
the stream to a revolt campus and that
remote campus server reflects the stream
to all the clients and the value of this
is we only have one stream going over
the Internet instead of all those
clients on the remote campus creating
their own connections back to the main
campus server they're just connecting
locally so that's the value of a relay
and they're simplistically I described
two basic kinds of relays pull and push
so a pull relay what this does is the
relay is pulling a stream from a remote
source and relaying it locally so this
would be in this case that look the
relay is set up on the remote campus the
remote campuses we're pulling it from
Merritt in this case that's what I call
it from Merritt from a server called
mi scream that one and I give it the
path to the FTP file when you do with
this kind of poll rate you have to
authenticate to the main server using
the administrative username and password
of the streaming server anybody that's
in the admin group so it doesn't have to
be the administrator of the server which
would be I think a bad idea it can be
anybody that's in a QT groups file
listed in the admin group
so it doesn't that user can be not a
real user on the system this is one of
those times when having it the users not
as real users comes in handy actually so
this this is the remote campus server
it's pulling in this content and
relaying it locally so that's the source
the destination is itself it's very
common for either the source of the
destination to be the loopback address
so in this case the destination is the
server itself and we're calling the
giving it the same file name because it
is the loopback address we don't need to
user name and password we don't need to
authenticate because we're talking to
ourselves we figure that's secure okay
then the other kind of relay is a push
relay and this is also very common this
is what we've used in Michigan so this
is when local streams are pushed out to
another server so you have a stream
coming in to the server and we want to
push it out to multiple destinations so
in this case the source is the loopback
address and this file name the
destination is a remote server and there
are a few different types of
destinations we can use here putting out
like little crib notes here
there's unannounced UDP is what this is
setup so a specific IP address and port
number this destination dress can be a
multicast address as we'll see in a
little bit
it can be a unicast address or it can be
announced UDP announced UDP is just like
the broadcaster so it needs a username
and password it's automatically going to
generate the SDP on the destination so
it acts basically just like a live
broadcast would act so beyond the basics
a relay can pull from a source and push
to a destination a real common scenario
for this would be let's go back to that
scenario where we have the remote campus
on a remote campus we have this server
out there and if you saw my picture it
what it was doing was it was pulling the
content from the main campus and
relaying it locally as a unicast
to itself so it was just sending it back
to itself so that people could tune in
on that but what if we wanted to relay
it as a multicast to make even more
effective use of bandwidth then what we
would do is we would pull it from the
remote server and push it to a multicast
address on the local network so you can
pull and push but as I said it's very
common for either the source of the
destination to be the mission itself we
have found that when you respect when
you're using multiple destinations as
they're doing in Michigan that this
unannounced UDP provides the most robust
configuration the main reason is if if
the relay server becomes unavailable for
whatever reason and then becomes
available again clients automatically
reconnect when you use when you set it
up this way so the unannounced UDP
because the packets are just flowing out
on the network if there's an
interruption in the flow as soon as the
flow is restored the clients will
continue to receive the stream without
any with that you know hands off whereas
if you're using an announced webcast the
clients have to reconnect because with
an announced webcast that SDP file gets
deleted when the flow is interrupted and
it goes away so that's why the clients
need to reconnect
you can also reflect a multicast source
for the streaming server so this means
if you had if you had a rack of servers
each one serving a different subnet you
could multicast one stream to all those
servers from your encoder your
broadcaster and each one of those could
reflect those as unicast off to
different parts of your campus and the
relay destination can be a multicast
address and we're going to take a look
at that one so this is the most
sophisticated set up that I'm going to
go through and then I'll turn it over to
merit so in this case what we have is
the blue lines our unicast and the green
lines our multicast so in this case what
we want to do is take advantage of
multicast to to again reduce bandwidth
usage on our networks so we have these
great multicast enabled networks you
have to have this your network multicast
enabled bringing this stuff to work the
internet in your
one is not multicast enable so this
won't work over the Internet it does
work over net two however it is a
multicast network so in this case what
we're having is the broadcaster is
sending a unicast stream to the main
server the main server then is
reflecting that unicast or relaying it
actually relaying that unicast as a
multicast on the main campus so now
instead of sending out in this case
three different streams we're sending
out one stream and people are just
tuning in on that multicast we're
sending another unicast stream to the
remote campus server and the rural
campus server is doing the same thing
it's reflect it's relaying that incoming
webcast as a multicast so let's take a
look at the configuration on this so
again using server admin this is how you
would configure it on the main campus
server we have a source the source is
ourselves and it's this file coming in
from the it's a stream coming in from
the encoder so the encoder has created
this OTO UDP FTP file and we're just
bringing that in as a source the
destination is a multicast address so
I've used one that's in the experimental
range here so there are assigned
multicast address and there are the 239
addresses are basically wide open and
you can just use them to play with so
I've used the 239 address and I'm
sending it as unannounced UDP on a
specific port number so now I'm multi
casting as soon as I turn on to simulate
a multicast about how the heck do you
tune in on multicast well this SDP file
from the broadcast source this is what
it looks like this is a typical SDP file
there's a couple things you need to
change in this SDP file to make it a
multicast SDP file so what you need to
do is there's a line here the C equals
in line you have to change the address
which was the address in this case of
the broadcast or you have to change it
to the multicast address and you have to
change the port numbers book for the
first M equals line change it to the
base port number you selected as your
destination and then add two for the
next track the video track so now we
have a multicast FTP file
if we put this on the streaming server
and people tune in on it the streaming
server reflects the multicast as unicast
which is a useful thing but not what we
want in this case to actually tune in a
multicast you need to put the FTP file
on any web server and that's the easiest
way to do it and you can just drag and
drop it on quicktime player too but or
email it or whatever you want to do but
the easiest way is to put it on a web
server and then people use the HTTP URL
to connect to the multicast FTP file on
so that on the main campus server we're
relaying also to the remote campus so in
this case what we're doing is we're
sending a unicast stream from the main
campus to the destination which is the
remote campus so the destination is that
remote server and we've decided again
we're using unannounced UDP because it's
more robust and we know that if the
stream somehow gets interrupted this
will continue to go as soon as the
things start start back up again
on the remote server we're going to set
up the same thing we have on the main
campus over we're going to tune in on
that incoming stream so the source is
ourselves and it's just on that IP
address listening to specific ports so
that it's unannounced UDP we're saying
look at ourselves look at these two
specific ports that's our source we know
we're getting a stream there we're
acting as basically as a client to that
stream and then we're going to set the
destination and in this case I'm reusing
the same multicast address because these
net are two separate networks I can do
this and I'm using the same multicast
address that's a good thing to do in
this case because then we can use the
same SDP file we don't have to edit it
again we can just use the same file we
already edited put it on a web server
and it just works so that's a more
sophisticated one I have a website that
tries to untangle all this stuff for
people and to help you you know do more
sophisticated with this stuff with the
streaming server it's just sound screen
comm so if you go to that website you
can get touch me through that page also
I've got a lot of instructional
information on how to the how to's on
all these things so
with that though I want to turn it over
here to Lawrence or Lori Kurt Meyer who
is the senior engineer of networking R&D
for merit network and Patrick by mold
who is the assistant engineer that
supports education in the state of
Michigan and I've worked closely with
them on their statewide streaming and
initiative hello
where as as we get into this Patrick and
I have worked very closely over the last
year putting this together and so we
thought that we would sort of tag-team
this presentation because there's parts
that we both worked on so we'll we'll
see how this goes as I say I'm a senior
engineer with merit network merit is a
statewide network for to the educational
institutions in Michigan and the 15
universities are so that the board
members of the merit board and we have
over 200 affiliates connected to the
statewide network those affiliates being
the is DS School District's hospitals
libraries and so the objectives we have
here were to provide a set an
infrastructure for statewide live
webcasting the impetus for this was from
one of the what's called the regional
educational media centers were had a
broadcast studio and they were doing
satellite broadcasts of administrative
broadcast to school administrators on
various topics and like No Child Left
Behind and that's one example and they
wanted a way to provide that broadcast
to users without them without without
the administrators having to drive to a
central location to watch this satellite
broadcast or have it or having to go to
receive it over a h.323 stream to a to a
conference or a video conferencing room
so the so what was a what happened about
a year and a half ago was Patrick came
in and there was one broadcast where
they wanted to explore new streaming
technologies and so Patrick suggested
using a photo streaming server so it's
actually kind of a funny story we were
doing broadcast statewide we're using
Akamai cause apples got a great
relationship with Akamai and we couldn't
fully support the projects the way we
wanted to some merit not merit Oh Wayne
RESA set up a streaming server during
their broadcast and it was working just
smashingly up until the point that the
network manager for the organization
walked in and he's got a map of the
network
you've bandwidth usage and he's seeing
you know ups and downs and then there's
a huge spike when the broadcast come on
because they've got like 60 or 70 people
that are turning in the broadcast and
its network manager goes that's
unacceptable
dad needs to change which we all do is
where in our work manager because we
have to make sure the traffic and flow
perfectly so we started having
conversations about the possibilities
and I knew the quick times got to me one
of the most amazing streaming services
available and I said cool let's set up a
statewide Akamai and yeah this is where
we're going to start talking right so
what happened was we'll go to the next
slide this is the set up where Patrick
installed a streaming server they try
and point with this thing over at what's
called Wayne RESA this region
educational service association they
installed one QuickTime streaming server
for one of these educational broadcasts
that was being put out by satellite and
they simultaneously took a feed and put
it on a quick toast broadcaster and
started this was a combined machine
running the streaming server as well and
so this is the merits network mish net
it's the name of
network and we have various affiliates
around the state of Michigan who were
tuning in to this and so what happened
was the the stream went across this this
one link from Wayne RESA
to mich net which happened to be a ds3
which was fine initially until they
started to get many people tuning in to
this point and completely overloaded
their outbound connection to to mich net
and again their network administrators
were hammering on the door saying what
the hell are you doing or what is what
is causing all this famine so it turned
out but also with this solution where
this stream was going out the posish net
there were similar trope points adverse
affiliate networks where multiple people
were tuning in some of our some of the
school districts that are connect them
go back some of the school districts
that are connected get a thank you oh
well get that I'm pressing too fast I'm
sure so we copy you in this point no
that's right okay that as I say some of
these school districts are connected
with a t1 and so you have four users on
there and you overloaded they're inbound
connection to their network so the
solution was to deploy a hierarchy of
streaming servers with let's go to this
line and so that's what we did and so
how now we can have a broadcaster at any
location on the mission on on the
network on the internet basically
there's one single this green dotted
line is a single stream that is central
master relay server and merit network we
we're providing the colocation space and
so the bandwidth free bandwidth
basically for this master relay server
and so this master relay server in turn
just read redirects the broadcast out to
a number of relay servers in the
different school districts and around
the state of Michigan
so that's minimizing the bandwidth going
across those expensive links or for the
school districts the expel
links to our network so again with
regards to multicast and unicast there
missionaries the unica multicast enable
network and so wherever we can we would
like to to be able to send a multicast
stream to the relay service and we're
working with our affiliates to see
whether that can be done so now we're
going to talk about leave start talking
about the things we've learned this has
been an amazing journal of journal
journal over the past year ok more yep
we I'd like to take the opportunity now
to thank the person that has more
information and is head of a QuickTime
streaming server which is George Koch
than I think anybody does so if you get
a chance to take a look at its site but
we ended up with right now we have 50 16
locations 15 locations in the state of
Michigan including copper country it was
a nice seven-hour drive up to Houghton
Michigan to set up that server and the
thing that's been most informational
about this whole project is working with
16 land managers right because now I
have 16 people they're responsible for
their own networks every network has
been completely different and so we've
had some serious growing that we had to
do so go on does it so ok are you yeah
there's something you want to add just
that talking about working with these 16
where manages the other the other
interesting piece to this is that if you
tend to tend to have several people
wanting to deliver the content on a
campus you have the content folks or the
who are wanting to get this stream in
and then you have the networking why'm
folks and deploying a quicktime
streaming server tends to bring those
two groups closer together very quickly
and that's something which is good I
mean it's definitely an education for
the network administrators so and as a
Patrick mentioned he has been doing the
legwork driving to all these locations
installing these service has done an
amazing job ok quickly about the the
architecture
some of the things some of the things we
want to do is make sure we could
broadcast from anywhere we want to be
able to eventually have students that
are working on projects and in the
educational institution anywhere in
Michigan broadcast right we've got a
program we're going on at 11 o'clock or
program going on at noon we have
actually had conversations about
universities doing school nights
so instead of students having to go to
the University they tune into the school
night event is taking place in their
local school district from em I assume
that so broadcast from anywhere where we
will have the parent node what does that
say push information out to the relays
regarding the content of the video which
you will talk about in a minute but I've
got this thing now that I call the
Microsoft effect and it's bad for me to
say that nobody's from Microsoft right
we're an apple developer cops like to
say this point but this whole notion of
servers do a good communication in terms
of announced relays over ICMP right ICMP
huh no the pinging between the yep and
so the some organizations were actually
ICMP has been turned off right and so we
have servers that can't talk with each
other anymore and they just oh I'm going
to stop broadcasting now because I know
large if you talk to it to the child
replicate we're going to talk about
replication of content over the server's
over our sink and then the thing that
Laura's going to get to is how I made
all this thing come together the Akamai
effect with the redirection script the
children notes we wanted them to be able
to have access to the data meaning the
video coming in from the streaming
server via the FTP file and when you
starting is anybody running streaming
servers everybody the thing that I have
to say is don't make this any harder
than it actually is
for some reason the hardest time we had
in setting up the service we were
setting up is we thought it had to be
harder math we thought maybe we needed a
pocket protector or some incredible
slide rule something that you know what
would allow this to make sense to us but
really this is just very very simple in
the way that it can be configured ok the
last thing that I think is from merits
perspective
being a ISP and we were concerned about
the whilst we've got these 15 relay
servers around the state and users get
redirected to pull their streams from
these local points of presents in their
various school districts we were
concerned about how many other users
might tune in who I don't have that but
don't have a local relay server and so
what we put in as a second
QuickTime streaming server in the merit
data center which would take which would
be the catch-all server for users who
are not redirected to a local streaming
server in there is D and so we're sort
of working to see how much how much
bandwidth we're willing to donate for
that and how much demand there is but
again this is a useful tool because we
can begin to see if more users come in
from another school district we can
begin to go to that target that school
district and say hey if you put a
streaming server in your district then
you wouldn't incur this or the bandwidth
costs across your you'll link to our
network and there'll be a great benefit
okay so the one note that we have is
we've limited the shunt servers so with
allows up to 50 users or anybody that's
outside of the service area and really
what we want to do is if it's a 50 first
person goes and clicks on that street
and that shunned server what we should
really do is is put up a web page where
it says you should go by a quick time
still get your network first or to buy a
QuickTime streaming server and put it in
your school district and then you could
watch this so so some of the stuff we
learned the first time I set this up
will use denounce relay which is the
easiest way to set up a relay network
and if anybody's looked at expanding the
networks that are doing hierarchical
broadcast broadcasting but announced
relay very simple put in from QuickTime
streaming server I'm simply setting up a
actually my travel that nemac in a
minute the first time we set this up
we'll use announce relay so let's go on
to the next line okay
and in announce relay the parents
responsible for getting the content out
to the children servers what I call
children servers so they know what the
broadcast is coming from so and parent
announces the FTP file and each one of
the children since
tune into the STP file because it's
sitting inside their movies slash
library slasher movies directory next
and this was a great solution it's very
very simple in setting up a distributed
streaming server network we also found a
very interesting feature built into a
streaming server in that if if one of
the children was not available the
streaming server would simply say I
can't broadcast anymore and restart
itself every two minutes so we ended up
with these live broadcasts where the
video is being restarted every five
minutes over and over and over again
yeah we just don't assume it so you tune
in you see a little bit more and then it
would stop yeah and so we had two things
that were taking place firewall
configurations right or the again the
Microsoft effect which goes back to
firewall configurations because there
was no ICMP so one of the things I would
recommend in setting up a distributed
network if you're going over firewalls
is make sure that you have a one to one
relationship the easiest conversation
for me with the network managers the way
managers was set up your parent set up
your child and set up the firewall in
between so that the two just can do
anything they want to do with each other
there is no restriction there is no port
mapping it's just a completely open
connection between those two devices on
the network so we worked at this for a
little while and we continued having
problems so we switched to unannounced
basically give a name give a port number
so the panthous is on the parent server
the parents simply get a relay set up
and we'll see some just a sec and it
simply says I'm sending data to this
port number at this machine just it
doesn't even care if the machines out
there it's just here's video you're
going to tune in to it you can watch the
video to your heart's content if not I
don't care really I'm just giving you a
video go on so this is the way the
network is currently configured I didn't
mention the first part we have quick
temps broadcaster running on a machine a
twink on your ISA right now and we've
simply taken the FTP file off QuickTime
broadcaster exported it and put inside
the movies directory on the
and each one of the children have been
set up so that they simply tune into the
SVP file and we're going to look at the
SUV file in a minute so you can see what
that looks like it's again very simple
we thought it had to be hard Patrick
white and I hardly ever use Excel I use
Excel and I got the spreadsheet that
just would cover this whole stage and
it's got this flowing set of formulas
where you can figure out port mappings
oh my gosh I would open sources saying
or so amazing and it didn't need to be
that hard rank so I I still have it
always have a unique last nighti file
for all right go thought about printing
it but I really don't need it now so
anyway so we copy the broadcaster file
up to the server each one of the relays
are configured inside the streaming
server and we'll look at that and then
video goes down to the children notes so
this is what we're currently configured
and this has been the best experience
that we've had fun one of one of the
things to note I like to think of it the
way this is set up is that we thought
that the route the SDB file is the key
if you don't have the SCP file on the
child relay server you know the the
master relay the parent server is
sending the data down and it's only when
you get the key that key on that local
relay server that but the stream
actually gets into that local relay
server and users can watch it and so
it's a good sort of analogy
well I went to myself what's up
oh so some of the things we like about
this the fact that if the parent ends up
going away the children don't lose the
connection to the viewers that are
watching the broadcast when a parent
comes back online if for some reason it
goes down the the children will
automatically pick up in the broadcast
that they were showing to the to the
viewers and the same SC this is a huge
note with the little pointer up on the
third line up there the same FTP file we
use for children to tune into the
broadcast can be distributed throughout
the children so with our sink we're
actually looking at the ability to say
we're going to have 20 channels and
channel 1 equals and with the
information that we're going to have
regarding localhost address 127.0.0.1 we
will distribute out throughout the
entire network affecting all the servers
and we see this thing growing we've got
17 servers 16 servers now we get two
more coming online so it'll be nice feel
to have this easy configuration be
replicable throughout the state next so
let's talk about the way this works on
the left hand side inside the streaming
server admin is the brought the relays
themselves and there's some information
you associate with the relay itself and
then next is on the right hand side the
actual servers that you're
redistributing this broadcast to so go
on okay I have a clicker but it doesn't
work with that thing what I wanted this
so if you're doing oh so we set up the
service so we have a really named
doesn't really effect in terms of what
the user is going to be tuning into we
have a relay point basically zoda's is
at STP and this is the mount point of
the broadcast of connection point and
the server ends up pointing it back to
itself because we have the broadcaster
sending video to a specific IP address
on the network we're not going to share
it with you because it's it's not fair
for us to do that but we tell the server
tune in to yourself to watch the
broadcast because it knows that this is
near to do it that STP file has all the
information it needs to build to get to
the broadcast
after sending one to the next one and
then when it's been within the relay
setup we have each one of the servers
defined either by IP address or by name
this actually doesn't have to be a name
that is a resolvable by DNS it can
actually be a name is sitting in the
host file and the port that this is
going to be broadcast to and as I
mentioned before we're talking about
channel 20 this could be that all the
servers for channel 20 are sitting at 21
326 throughout the entire network so the
server simply says I'm broadcasting to
this UTB base port going to the next one
so remember this number you d be base
port actually ends up being two ports 21
to 24 inch 326 is the beginning port and
we notice we assign this to audio we
didn't QuickTime bit and then video is
assigned to 21 328 this does not mean
you can start messing around with the
port 21 327 alright basically three
ports for broadcast is what you're going
to be using and I tried to get crazy
with the spreadsheet I tried to figure
out a way to maximize my number of cords
because when I was thinking I had to
have a look at George laughing he had a
fun time with this he really didn't tell
me this and knew he didn't tell me I'm
making that part up but I started to get
crazy with how is configuring the pores
and it turned out that the 327 is a I
don't know Cinque Ports something that
is used by the server in terms of
communication so this information really
isn't pertinent to what we're talking
about regarding the broadcast talk to
myself look at these ports and then
start spreading video to the viewers
that are wanting to tune in one of the
excellent the cost outlines I mean we
can delete those out of the SPP file
which were the unique it turned out that
go back one it turned out that these
three lines right here are extremely
important and George can talk about that
I didn't have them and we didn't have a
broadcast and of course he's like oh
you're messing me three lines guys scary
and
okay okay
I'll talk about live every direction -
as we said we want to redirect our users
to the closest streaming server local in
their to their school district and so
what we've put together is the CGI poor
script and what will be a database right
now is a flat configuration file which
contains a mapping between a set of IP
addresses IP address ranges and for that
School District and then the load the
address of the local relay server to
redirect the user to and so the users
can open up a URL causes the CGI to the
post script and they automatically get
xml data back to quicktime player or to
quicktime embedded in a browser and
telling it to open up the stream from
the local server we also added what i
worked on was some additional logging so
we could lovely as users get redirected
or do not get redirected so we could
begin to see how how successfully we
were on how well the coverage was of
local relay servers and again we could
have the ability to redirect users to a
live stream to an SDP file on a local
relay server or to replicated movies or
mp4s or whatever media we have
replicated on the local relays this this
redirection script will work with and
with any of those this is a just a
one-line example we've just got the the
child IP address that the relay address
and then the client address range start
and end
and as the script is called essentially
this is you can see there's a curl
statement we're just showing that when
the script gets called by the browser
you can use curl command line 2 to make
this call this is what gets returned at
XML and this is passed to good time
either the QuickTime Player or to
QuickTime inside of the browser and does
the necessary redirection passing the
redirection to the local relay IP
address followed by the the name of the
FTP file and we just this is showing the
embed come on how you would embed the
call of the object embed command to to
embed the call this is this is our
master relay server and we're calling
this poor script my stream net dot pearl
with the BSD name of the SPV file and
what results from that call is this is
this redirection to the local server and
[Music]
we're going to talk about what so what
we wanted to do was as we rolled out
these servers we wanted to do a whole
set of testing and load testing as much
as possible to make sure we that it
would run under under load certainly we
wanted to stress the redirection script
and so we developed or there's a one a
Python script which George put together
to to stress test the redirection script
itself and then we also use streaming
low tool which is part of QuickTime
streaming server is it part of the
release yeah it's on every server isn't
it I don't know
so John talk a little more about so two
things I wanted to add so we've been
talking very positively about in my
stream net and it worked incredibly well
the last two broadcasts that we had but
we actually were at a point where we had
four four broadcasts and and all of them
ended up with some part of the network
falling apart so
we we were needing meeting to find ways
brought George out to figure out what
the heck was taking place and if anybody
thinks that my closed ten server and
QuickTime streaming server and a neck
serve is not an incredibly powerful tool
to use I just have to let you know what
we saw these things capable of doing em
I assume that our parent streaming
server we were simulating five hundred
five hundred broadcasts along with
seventeen relays and it just ran for
like hours and hours and hours and hours
and hours and this is great we thought
this was amazing we got a 10 megabit
connection to the network or a hundred
mega 10 mega we got 100 megabit
connection to the network and we're you
know we got 500 users and watching this
video plus we're doing video out to
relays I think overall we simulated two
thousand two thousand three thousand
users on the net because we started
stream loading each one of the relay
servers or the children's service out on
the out on this day and it was great
they worked the only problem was we had
not gotten to the point where the server
failed which is what we were trying to
find until the day that we got Ken Tramp
who's been the most wonderful person to
work with he grew up in the video world
right in the video world if something
fails you get rid of it no there's no
three strikes and you're out if
something fails in the video world has
just done dead don't talk to me anymore
and we were bringing them up in this
electronic world which is a little bit
different and as soon as I got count on
the phone and I can check us out we're
doing these broadcasts working is great
we got things figured out cuz we
reconfigured some stuff and as soon as
he got on the phone it's like oh that
looks wonderful BAM and my student 1
goes to help just and we had to go
actually restart the machine well it
turned out and the reason they're
bringing this up is assuming load tool
is a great application that comes a
streaming server you can simulate one
hundred nine hundred you know 25 to 50
users per connection hitting the server
in real time so it's sending out the
bandwidth each one of those connections
and so long story short turned out that
we end up end it up with a motherboard
or a motherboard on our server that had
problem only related to power changes
that were taking place in the
environment that it was in so it was
like one of those things you would never
find but we ended up with a guy putting
a new server in the server room that we
had our MIT no one in and he turned on
the server and that little switch
changed the power enough for the for
that configuration the server just like
come out of here done see you tomorrow
so we got a new motherboard things have
been working perfect ever since but I
think that what going back to you
talking about working with this this
video production person I mean with any
of this when you pull out a service like
this I mean you it's important that it
that it works I mean I did it it's hard
you don't get many chances with this you
want to make sure when you roll this out
that it runs because if you're dealing
with video production folks from that
world they really have a very low
tolerance for for failure coming in so
it's important to test and so assuming
roll to low till we came a great
solution the Python script available at
the site that you have so basically the
Python script is what we use to simulate
a hundred users per second hitting the
redirect script to make sure we weren't
having some kind of memory problem with
Apache or the script we've written to do
the redirection and the thing that ended
ups in the end giving us the information
we needed was the server monitor tools
anybody using a neck serve the server
month monitor tools are incredible for
what the information it can provide some
things that I would recommend all
services we're running on the servers
anything that's not necessary turned off
so anything that's taking place in terms
of file copies we're doing over SCP
we've actually set up firewall rules so
that the things that we need to be
public like 12:20 we're gonna have that
open to the world so we get to it from
anywhere 660 is what we actually have
for server settings on 10 - server we're
going to be just we're going to be up to
10 tree by the end of the summer which
would take us back to 311 and then a
tweet because we're using SCP and SSH we
actually have the firewall set up so
that there's only one machine on the
internet that has the ability to
administer these servers over SSH which
we're not going to tell you
because you might try and get into it so
I would highly recommend it to not run
as much on the serve as you possibly can
lock this thing down because it's
sitting on in a public place and this
thing doesn't work what's the next one
this is where George talks it is what Oh
sight looks like it's with with both
Darwin and QuickTime streaming server
and it's a great tool for stress testing
and these folks said the main thing here
was testing you know and - and I see
some other folks I know any audience and
and they know that that's that that is
the critical component here is that when
you're rolling out something a new set
of services on the network especially
something new like streaming important
thing is do some stress testing you know
make sure that you've got things under
control before you know you do a broad
broad scale deployment so some of the
things we did was we each instance of
streaming load tool we've set up to do
50 concurrent clients we have multiple
instances running we have three
different machines running multiple
instances and we've loaded the heck out
on the server and we're amazed at the
performance and I've recently assignment
again from Wirecast was showing the
Swedish newspaper that is really loading
the heck out of a server and again I
noticed wanted to talk to about what
they're doing there even more amazing
really ok so long do some 10 runs
replication these are the next steps so
the next things we're working on is
replicating content necklace 10 the UNIX
layer has great replication technology
built in called our sink so this is an
our sink command that you can that we're
using right now in a test mode to
synchronize content between the servers
the important thing here is that this
synchronized stuff all works with the
redirection script so the redirection
script doesn't care if it's an FTP file
a movie file mp4 file whatever it is it
does the redirection we're also working
on some monitoring tools and Laurie is
really excited about putting some of the
stuff
into a sequel database I've just been
doing flashy QuickTime stuff so far so
this is something we're working on for
local administrators the idea is the
local ran guys that are up on these uh
the in these school districts we want to
provide them with a tool that they can
use to monitor the server but not give
them full administrative privileges to
the server I think again monitoring
especially during the testing phase
being on a monitor and know how many
streams are actually running on this
little child relays if something does go
wrong you've got some logs that you can
go back and really begin to figure out
what was the state when yeah we're also
Patrick and I are planning on setting up
Moodle which is open source course
management system very nice one
beautiful macula sensor for all these
people that are becoming involved in the
project so as more people become
involved the faculty member want to know
how to produce content places on the
server there will be a website for them
to go to to get educated on that and so
that's another work in progress
cute QuickTime trimming server publisher
we're really excited about the new
version one of the new capabilities that
that's coming in the next version is the
ability for end-users to use the tool
right now it requires an administrative
user which limits the scope of its use
but with the capability for end-users to
basically publish streaming content or
downloadable content to their own site
streaming folder through this tool is
just phenomenal we're very excited about
that use because it will it will extend
the use of these child's own servers so
that they can be used for locally hosted
content so we think that's a really
great thing so that we'll do a few demos
and how are we doing on time here hold
on we had worked 15
okay let's bring up the demo machine we
have it so might stream that this is a
the folks at Wayne RESA who are the
third partner of my stream net with with
merit and Apple have got an amazing web
designer who's putting together this
with this website and this is the live
streaming page of the website
and so we're giving the users ability to
view the stream within within a quick
time in a browser window my only cat let
me make sure so ya know that definitely
come see what we're streaming here right
now is as a sort of a testament making
the the Annenberg video at Annenberg
educational video which is being
streamed in over satellite and they're
taking this feeding it through QuickTime
broadcaster at Wayne RESA
and then streaming it out over QuickTime
over my stream man
and so we have a great ability to test
at any time and this is this is it's a
good promotional piece for for new users
in the state of Michigan to see that
also education and so the other way you
can also click and view it in a quick
time player window the other piece we're
also saving archives maybe this isn't
working and so we have all the archives
our previous broadcasts that we've done
over the last year and users can view
these at any time so you want to show
the air so are we working here get a
dinner night here myself so another
thing that we've been looking at is some
of these Administrative Tools so what I
did was I created some some wired
QuickTime movies to talk to the thing
called the admin protocol on the server
so on the back end of the servers is
this thing called the admin protocol
which is
used to do all the administration and so
I wrote them see guys that talked to the
admin protocol and then wired them into
some QuickTime movies this kind of
sounds crazy right so the first thing
we'll do is we'll bring up this this
little the graph thing so this is
graphing bandwidth usage coming from the
server in real time and I can change the
interval of the graph and I can change
the bitrate but while it's doing that
let's go ahead and we'll start off a
stream from the server so we have this
movie so we'll go ahead and turn it up
and you see it as a fan which jumps up
now it jumped up a lot there in the
beginning that's because of Chris
because right because we're we're doing
the skip protection and we're sending a
whole bunch of packets down and then it
will start to settle down as it you know
as the buffer fills so you know it's
kind of a cool thing to be able to watch
the bandwidth in real time right so
that's one little hack the other one
that that and what's neat about this is
it the admin protocol with this all this
stuff is talking to Lori's really
excited about logging all the stuff into
a database I like the cool woodsy demos
myself so so how's the demo as well so
here's another one that we've done so
I'm going to go ahead and set this to
one minute so it's pulling and now every
one minute this is the number of
connections RTSP connections or we can
look at HTTP tunneled connections we can
look at the RTP bandwidth so let's crank
this back up to five seconds so now
we're looking at the individual nodes
around the network and seeing what the
current Pamela is in this on the main
this is the law the main server here so
you know just interesting things that
you can do with wired quick time so
thank you totally hip for your tool you
can squash you to do it and then we
actually have a this web page but I left
the password over there if you want to
do the web page it's just talking to see
guys and you can actually drill down on
the individual session you can see what
IP address somebody is coming from and
what piece of content they're watching
with the admin protocol so you can get
that level of detail with the Stabbin
protocol very exciting in terms of how
we log this stuff how we
tract in the future okay so I think
that's it for us for right now and what
I want to do is invite up John Anderson
and John is on the streaming server
engineering team and he is largely
responsible for QuickTime streaming
server publisher and there's a new
version coming as I said what's exciting
to us about this tool is to really
leverage these child node servers to be
able to take those servers and in the
School District allow faculty to give
them a very easy user tool to publish
their own content locally hosted content
to these streaming servers which is
something we found very effective at a
lot of schools and now we have a great
new tool to support it so thank you
that's a little weird with the
microphone it's like karaoke or
something I'm going to go ahead and
close this as it was mentioned before
one of the big features and cue CSS
publisher two is
hello okay as that's was mentioned
before one of the major features in
chief justice publisher 2 is that
anybody can log in to their home
directory and publish content they don't
have to be at an administrator so that's
a great new feature I'm going to go
ahead and start this up in login there's
also been a lot of great performance
improvements in the application and it
has a new UI now so if anybody is
familiar with you just as publisher one
you had all these different modes that
you had to go through where you were
uploading your media in one mode and
then you go to the settings and set that
up and you go to links at this point
everything is all in one place
another thing that's that's nice this
new is if you're binding QuickTime
streaming server to specific IP
addresses so that for example you can
run on port 80 and then run any web
server on another IP address you can
actually pick which IP address you want
to publish files through so that's
really nice also if you have multiple
virtual hosts set up in server admin for
the web server you can pick any of those
sites here as well and then of course if
your administrator you get a choice you
can work either out of this the system
library or you can work out at your home
folder as well so that's kind of a nice
feature and so what I'm going to do here
is I'm going to go ahead and open on
this up and upload one of these movies
and it's real simple all you do is drag
it into the library that uploads the
movie and in addition it will hint the
movie as well so labs interactions a
movie automatically on the server
so now what I can do is I can click on
this movie now when we put files into
the library they actually go into a
location outside of the QuickTime
streaming server folder and that reason
for that is that you may want to make a
movie available only via live simulated
live broadcasts or playlists and so that
gives you the opportunity to do that but
what I'm going to do is I'm going to go
ahead and check this checkbox which is
publish media to QuickTime streaming
server and when I hit apply what this is
going to do is it's actually going to
create a symlink
in the QuickTime streaming server folder
it will also go and create a referee
that points to the to the to the movie
on the streaming server so does that all
that automatically you get down over
here you get a URL that you can copy and
paste into the player and you can also
just click play and the movies have been
running so it's very simple streaming
and then of course if you want to set up
a simulated live playlist which is great
for relays something that you can then
relay off to other servers all I have to
do is click on this plus button and I'll
just name it
the latest much like iTunes you just
create the playlist and drag the file to
it and that's it
if you want you can change weights on
this so you can do a weighted random
playlist and that will actually play
some some tracks more than others and
then just start it now the cool thing is
I'm going to go back to the file not
only is it publishing those files and
making them available you can change the
annotations anytime you want and it
already has a movie when it's done and
in addition you get down here you get
all of the HTML to embed the page to
embed the movie in a page so that
basically you can choose whether you
want to for example open the movie
include templayer have it autoplay and
once you choose that you have HTML right
here that you can copy and paste into
Dreamweaver or BB edit or any other web
page editor in addition to that I can
actually create a web page directly from
here
this is the development LS that is may
now work a lot to see but I just go
ahead and create a page and we'll drag
the file to it and this will actually
generate a page based on a nexus LT
template that has the movie in it now of
course I haven't set a poster frame so
it's just defaulting with the first
frame if I click on that then that
automatically brings up the movie so so
that's publisher
[Applause]