WWDC2004 Session 632

Transcript

Kind: captions
Language: en
good morning so I'm standing between you
and lunch I believe so we'll try and
make this either an opportunity for you
to get those juices going to to look
forward to lunch or and certainly try
and try and make this an interesting
session for you as mentioned I'm the
director of engineering for a pro video
applications that includes final cut
Final Cut Express final cut pro pro
video codecs and some other other
projects within our pro applications
division I'm going to be talking today
about the xsan product and how it fits
into the video workflow and although i
will be talking it from the perspective
of final cut pro it actually is very
applicable to anyone in the applications
domain who's dealing with midi media in
a Nexian environment it's also I think
you'll find it appropriate for anyone
who's dealing with the kind of the
creative professional marketplace in
general there's a number of different
tools that fall into this category and
xn is an excellent tool in this space
but it provides some interesting
challenges as you as you're dealing with
media and we'll talk about a lot about
that today certainly folks are dealing
with site administration I think we'll
find this interesting as well because
setting up this environment is actually
the biggest challenge that will be will
be talking about so as we get into the
talk first I will talk a little bit of
that kind of set the context around how
video editing you know kind of the view
of video editing from the editors view
from the kind of creative professionals
view and what their expectations are and
then we're going to get into how this
applies to X an environment and how
shared storage and workgroup computing
kind of changes that model and impacts
that model in a pretty significant way
and then we'll finish up with a little
bit of thinking about where we go from
there so first to get into video editing
today I think one thing to understand in
the marketplace right now is that the
focus is around as much performance and
real-time capability is you can possibly
get out of the hardware but certainly
what drives the market in a big way
they're looking for the fastest hardware
they can get the fastest storage they
can get the looking for as many streams
of video as many effects as much
real-time capabilities that can possibly
get out of the system and that's driven
around a number of different kind of
workflows both from the kind of media
that people are dealing with and the
kind of people that are working with the
media so that's everything from the you
know individual independent you know
production house people or video in
event videographers who are producing
corporate videos to the independent film
makers to small work groups of people
who may be working on short form
commercials or movie trailers or doing
music videos all the way up to broadcast
television you know how two shows on you
know HDTV and you know movies that you
go to see at your local movie theater
all of that is expected out of the
today's existing tools and hardware and
it's a huge range of expectations and it
drives you know in a large number of
different kinds of media that people
have to work from if you're dealing with
film and you probably you're obviously
not working with film directly in the
computer but taking it into this on top
of offline format but many of you know
and for news gathering and for other
things they're dealing with db25 your
standard you know camcorders if you will
all the way up to your news production
high-definition uncompressed standard
definition you know for TV on converse
high definition as well as other types
of media like audio and stills one thing
that's really interesting to consider as
you're looking at the performance issues
around dealing with its media is the
kind of you know data rates that you're
dealing with for this kind of stuff for
your conventional db25 or your you know
standard mini DV camcorders you know
it's in the range of 2.8 to 3.6
megabytes per second megabits per second
of information and as you go up the
you know the through the different types
of media from dbg / 50 and dvcpro HD the
data rates of course go up appreciably
when you get to uncompress things get
really complicated because you're
dealing with it's just monstrous amounts
of information the good news is compared
to things like DV the various TV formats
is these are uncompressed so you you
know the kind of the trade-off there is
computational complexity if you will you
know having to compress and rican for
decompress and recompress that
information is you're dealing with it as
opposed to moving just large quantities
of bits for all this in a data and I
want to remind you again that you know
people today are looking to get as many
streams of videos and effects as they
possibly can through the pipeline and
today you know with final cut pro for
example we deal with a number of streams
of video of various types you know at a
real-time level without any additional
hardware and a dual processor g5 so
people expect that if they're going
through the creative process that
they're going to be dealing with lots of
streams of video and they're going to be
doing lots of things no expect you'll
time Behavior all of the video of course
I think it's worth mentioning and kind
of in contrast that there's also a lot
of streams of audio to be dealt with but
it's really interesting as you look at
the you know in upwards of 100 megabytes
per second or megabytes per second for
you know uncompressed HD you're only
talking 192 kilobits per second for you
know for audio but usually people are
dealing with many many streams of audio
at a time but there's a very large range
of expected data rates for the kind of
media that you're dealing with in a
typical project and this is a project in
Final Cut that and I think it again is
worth noting that typically there's a
number of streams of video that
particular dealing with if you look at
something like a commercial today a
commercial you know actually has lots of
streams of video going on at once
there's usually mom you know most
various motion graphics pieces there's
multiple camera you know attacks that
are coming in and their criminal cross
fading or moving across the screen
there's a lot of stuff people are doing
and you may not typically think of you
know in the normal process how many
streams of media that people are dealing
with but it's quite a lot one thing
that's very very common in the case of
audio is that people are dealing with
usually you know upwards of twenty four
tracks of audio certainly when it all
gets done it all comes down to one
stream of video and maybe a you know a
stereo pair of sound but when they're
going through the creative process
there's usually lots of different layers
of the stuff that is getting composited
together at various levels and as
they're you know working with it they
expect to see it listen to it in real
time so it all comes down to really the
way today's you know model of creating
video is it's very much a single system
workflow that you know people have
really optimized their process around
very high performance single system
solutions and while they may be in an
environment with many different machines
you know today most of the media is
tends to be moved around you know via
the network or you know by taking that
little firewire drive and going from one
system and doing the sneakernet thing
and the downside to that and that model
is that you end up getting a huge amount
of media that's duplicated all around
the enterprise it's not very efficiently
used but a lot very effectively use the
upside is you've got a huge amount of
power that you get out of that single
system because you can make a lot of
assumptions about being able to use
every bit of that band with the other
downside to having all that media
duplicated around the environment is
that you know dealing with backups
because it's all distributed around your
your your enterprise or infrastructure
and dealing with media security which
tends to be a really big issues in you
know for example and for folks who are
doing maybe trailers or you know trying
to do you know unreleased movies they're
trying to keep you know pretty tight
control over that media before it gets
out and gets leaked out and put on the
on the internet so doing that when all
that media is being sneaker netted
around anyway and a firewire drives
makes it even easier to
you know how thats get sneaker netted
right outside the front door and there's
certainly a number of other challenges
that the single system workflow you know
has it just really makes the process as
you're working as a group very very
difficult and that's where x gain comes
in xsan really provides an opportunity
for multiple clients to work together to
share data over what's the really most
important part of this thing is a very
high speed interconnect the difference
today when you're dealing with you know
a network file system solution and video
is it just isn't working all we talked
about the number of streams the video
that you know people expect to work with
you know over a network infrastructure
you know typically you just can't get
that type of flow the sand environment
really you know provides an architecture
in the design that I'm going to makes it
possible for you know media rich
environments to be able to share
information because you get pretty much
the benefit of you know local speed
interconnect because of the fiber and
the way that the file system and sand is
designed relative to how something like
a network file system is designed
certainly you know as we talked about
them you know and the negatives there's
a lot of of a single system environment
as we move to Juneau shared storage is a
lot of benefit here being able to get to
the point where you can actually share
media assets you know within an
organization is a big deal if you're
dealing in broadcast for example you
know you may have a lot of stock
material that you deal with it gets
brother and that you really want to be
able to use over and over again if you
have to copy you know literally
terabytes of data around to each one of
your workstations in order to be able to
make that stuff useful it makes it you
know very ineffective and a horrid use
of your storage so the other advantage
of xane sword shared storage environment
is the ability to consolidate that
storage typically you have a lot of
material that you're you're sharing
amongst you know troops with people not
just what you're working on right now by
the
large amount of you know stock material
or material that's potentially coming in
from various sources and again avoiding
having to have the large capacity
storage on each of those workstations is
a really big plus a considerable amount
of money the other big big benefit is as
you look at the economics of you know a
video environment that one of the
biggest you know most expensive pieces
these days is getting to be the
acquisition devices themselves the decks
particularly easy as you start talking
about high definition or or film you
know those you know pieces of equipment
there somewhere on the order of 50
200,000 to potentially up to a million
dollars in order to be able to acquire
that media into digital form it's not
the kind of thing you're going to stick
on every one of the workstations within
your environment so being able to get to
the point where you get stored shares
shared storage pardon me you get the
also the benefit of being able to you
know reduce the number of equipment
costs that you have for acquisition
because you can centralize that get the
material into a place where people can
share it and avoid having to I have lots
of redundant redundancy as far as that
kind of equipment is concerned obviously
the benefits you know being able to now
have that information there means that
you can actually start putting together
workflows you know within the workgroup
means you can get materials it can go
from editor editor or the creative
professional for the Deaf different
parts of the creative process because
they have the ability now to be able to
all get to the same media as I talked
about a little bit earlier the issues of
single system workflows is that the
security is a real challenge that
sneakernet that you know approach of
moving media around is you know a big
problem you know in in areas that are
dealing with potentially unreleased
material movies or other things and
being able to get to the point where
because you now have central control of
your media and being able to start
applying the kinds of permission
capabilities that that a UNIX system and
a shared storage system and you know
enable means that you also can start to
control the security of your information
being able to make sure that you can say
who it is that can actually see the
material who is it that's allowed to
change it and you know basically get to
the point where you have a lot more
manageability of your assets and of
course probably the most important part
of this whole thing is finally being
able to get point word because all your
materials together it becomes actually
feasible to back it all up however with
all those good things there's a number
of things to consider when you're
dealing with putting together a fan
environment one is performance remember
we talked about single system
performance or what people expect today
in the creative process they're getting
every single ounce of power that they
possibly can get out of that as soon as
you trying to distribute that now one of
the challenges is trying to rein in and
control the ability to continue to get
that kind of performance the other issue
is dealing with redundancy shared
storage is great because you can all
share it but you now have the
opportunity for single point of failure
and in that in that situation you need
to be thinking about what do you do to
make sure that you don't put your
enterprise at risk because you now have
a single point of failure those
individual workstations are great if one
went down you still have the other five
of them that you head up and running but
again you know this is one of these how
do you manage the trade-offs of being
able to get to share but being able to
get that sharing and make sure that you
don't put yourself a you know enterprise
at risk in the process security while is
a really important thing to get in there
it's a really hard problem to solve and
we're going to talk a little bit about
what some of the issues are there but
you know I want to prepare you that
getting and putting together in a kind
of the security infrastructure that you
need in terms of managing multiple users
is very difficult and there's so I think
a lot of challenges today in terms of
dealing with collaboration it's a real
plus but you got to think it through
ahead of times and can understand what
it is you want to try and accomplish
here so let's start with performance
first
the goal in the San environment really
is to get back trying to make sure that
you can get to that single system
performance that's what your editors are
going to expect let's require your
creative professionals and your
enterprise is going to expect because
you want to be able to take advantage of
the creativity that being able to do all
these things in real time enable the
challenge is you've got to be thinking
about now that you are spreading
potentially the performance load or the
expectations with a large number of
clients within your environment let's
say you're putting together a you know
five system fan and you're all going to
you know some sort of central storage
you know you map that against five
individual systems that had their own
local storage remember there's a one
common element in that which is the
storage which is now getting hit by five
times as many people as it would be if
you have them all completely isolated
and so you need to make sure you
understand what each of the clients
within an environment are actually doing
what kind of media are they actually
working with and how many streams the
video or media you know are they working
are they typically dealing with in the
projects that they're doing like I said
you know if you're talking about
something like you know commercials
commercials are good tend to be a good
example where lots of streams of video
is you know see these things on TV TV
it's all about you know transitioning
between lots of different shots there's
a lot of motion going a lot of activity
if you look at something like you know
more of a conventional film you tend not
to get quite a large number of streams
that you deal with because you're
dealing with a lot more long-form
material but you have to look at the
kinds of things that you're doing and
factor it in that into the process and
as you go you need to kind of go back to
that graph that I showed earlier in the
slide and and no map the kinds of things
that you're doing against the size of
the streams of the number of streams to
understand what exactly you're doing in
terms of the perform the data flow now
one of the things that you should also
consider if you're dealing with single
trying to get the single system
performance is that in your fiber
interest
sure for something like ass and your
switch is going to be your friend or it
needs to be your friend it's really that
one of the most important components you
know in the infrastructure and the
reason is its job is to help isolate
traffic from each of those individual
systems in the sand and the individual
components of your storage and the more
parallel ISM and the more isolation that
you get between those components the
more throughput you'll end up being able
to get through the process through
through the whole network basically the
more things bottleneck the more things
end up going through they want through
one pipe the left you know total
throughput and throw to you know total
capability that you get in your
environment the ability for your switch
to isolate is really really important so
as you're looking at that you know part
of your expenditure one you'll find out
quite shockingly that the switch is
probably going to be one of the most
expensive pieces of equipment you buy
it's going to be probably more expensive
than your storage it's going to be more
expensive than any one of your computers
it's certainly going to more be more
expensive than the software and you know
it is going to be something that you you
know should be prepared that it's going
to be a very costly piece and you need
to think ahead about how many clients
I'm am I going to have in the
environment how am I going to put my
storage together and how many different
strands of fiber am I going to be
running ultimately in order to be able
to get the maximum amount of data
through this to the system you've got to
do the math meaning if I talked about
just a moment ago you've got to look
back on what are the clients in your
environment doing what kind of media
they working with and how many streams
of media are they dealing with you gotta
count audio you've got to count you know
the video and you got to kind of there
and do all that calculation and
unfortunately it's one of those things
you've got to do really upfront figure
out multiply out you know total data
flow within your organization and figure
out how much data is going to be driven
through you know the strains of fiber
you know back to the storage and where
the bottlenecks are going to be
always remember that and this is one of
the the typical things that gets
forgotten is that the editors are always
going to be where we're dealing with
multiple streams and don't assume one
workstation one stream of data the
reality is that they're dealing with
many streams because that's part of the
creative process another thing to factor
into an environment like this is
typically in video you're dealing with
pretty large files you know they're you
know certainly many many megabytes if
not gigabytes in size you know typically
depending on them in the media that
you're dealing with and from a
perspective of impact you're dealing
with a reasonably small number of files
but a large amount of data the data is
kind of coming in you know if you can
think of a fire hose and I just a huge
amount of informations coming in but
you're not you're not dealing with a
large number files you're just dealing
with a large amount of data what you
have to be careful of in the
environments like there is that typical
computer usage workflow let's say email
or web files or you know you know you
know Microsoft Word documents these are
very small files and usually if you look
on your your hard drive there's lots of
them and the impact between those two
different usage models is pretty
significant that you know the fan is
actually extremely well tuned to be able
to deal with large file access lots of
data coming over the fiber but not lots
of little bits bits of data coming over
the fiber that's not a very effective
use of that particular resource and it
gets worse when you may start mixing
that data and so when you factor in your
planning and you're looking at your
infrastructure you know one of the
things you might you know think of doing
is you put in the same environment is
cool share everything well that may not
be the wisest thing for you to do
because as you start looking at all
those little file things that are going
on with your network you could end up
actually completely taking all the
performance out of
you're out of your infrastructure and
doing that and if it's appropriate for
you to do that or let's say the small
files you're dealing with their photos
because those are certainly reasonably
small files and you can have certainly a
lot of those what you mean start
thinking about doing is trying to
isolate that kind of material so that
those large files and small files are
actually kept in separate places so you
can get again more parallel ISM you know
through the process if you move on to
kind of the next step of you know first
understanding just basic capacities you
one of us start thinking about how you
actually can you know what things you
can do to enhance your storage
capability within the stand so certainly
a one opportunity is being able to add
more storage so more storage does not
necessarily mean more disk space though
that's certainly one of the things you
can get out of more storage but it's
really adding more storage in terms of
getting more parallel ways to be able to
get to data so it's basically more heads
or you know more rotating platters if
you will that can all be accessed in
parallel so in the case of what I'm
referring to your in terms of adding
storage the goal here are an outing
storage is adding more parallel ISM more
ways to be able to get to that data
simultaneously rather than adding more
bikes that you can possibly put on the
disk that's enough happens to do benefit
one of the other aspects of that is how
you actually orient your storage one of
the opportunities is looking at
spreading the load across multiple
basically multiple rotating services by
striping your data so this is a standard
raid behavior but you want to think
about again if you take all the you know
the capacity that you're looking for how
can you possibly get as much of it in
parallel across multiple devices
simultaneously and you should look at
striping the information across these
multiple devices is one way of being
able to get there another way of
approaching that is actually looking at
segregating your data so if you have
lots of different media within your
environment that you
dealing with there's a couple of you
know capabilities that exist within X
and to really help you there in terms of
of being able to isolate that media in
different ways one is the use of
affinities affinities are a technique
for being able to effectively take a
portion of the file system and tell it
to put it on specific storage or look at
and I look at the opportunity of
actually using multiple volumes as a way
and what kind of a more coarse grained
way of being able to you know force
material into different storage pools
and the kinds of things you want to be
thinking about in terms of segregation
is again keeping kind of the big stuff
separate from the little stuff is it
kind of an easy way to look at it
because if you can take something like
video which is again usually a less
number of files but usually a large
amount of data to get streamed over time
and separate that from potentially
smaller files that are going to be
accessed more frequently one example of
that is certainly audio and video they
have different substantially different
data rates much smaller much lower data
rates as compared to the much larger and
much much bigger amounts of data
certainly still is another example if
you're doing an environment where you're
creating you know video that has a lot
of skill material look at keeping the
fill material separate from your video
that can help quite a lot another thing
to consider is not only the kinds of
material but how the material is used so
if you have material that is like you're
you know stock photos or stock video
that you want to have on hand that can
people can use for creative purposes but
they're not used very frequently
particularly you may have a huge amount
of it and use people tend to use just
little bits of the time look at keeping
your stock material isolated away from
your current active projects current
active projects tend to get a lot of
change a lot of activity of you know
against them versus potentially stock
material that may you know have a much
lower rate of change and being able to
isolate that particular and slower media
or
better yet away from the material that's
getting accessed at high rate can have a
big impact another thing to look at is
actually separating your projects
themselves the actual you know different
projects that are going on within your
enterprise from each other you may have
you know potentially fiber you know six
major projects that go on at a time and
one of the things you may want to
consider is organizing your storage and
organizing your infrastructure so that
each of those projects are isolated from
each other not so much from a security
perspective but again from where it sits
on the storage so you can get more
parallel ISM you can keep it all on one
volume and use affinities as a way to be
able to do this kind of thing basically
allowing people to be able to share
material and be able to see the material
as appropriate but being able to use the
infrastructure of the sand to be able to
tell it to where to put the material
actually on your physical storage one I
certainly recommend is you know thinking
about you know where you're doing ingest
within your facility as opposed to you
know the kind of again the less likely
to change material static material or
potentially rendered material ingest is
one of those cases where it's really
important that you don't drop you know
frames or other things you know through
the process you're trying to get this
what's coming in probably from that
$50,000 deck or you know a
million-dollar tell us any device and
it's pretty critical that you you know
want to shorten that process for being
able to get that material you know into
digital form as quickly as possible
again looking at a fitting these is a
way to isolate that so you can manage
potentially bandwidth and to be able to
again make sure that you effectively get
isolation through the fiber from much of
the other activity that's going in in
the infrastructure you actually want to
take a moment and kind of draw out the
map of your of your fiber the users and
your storage and think about in the
different usage cases which lines of
fiber actually
get lit up if you will during the
process and you'll find that you know as
you go through that process that you're
going to find places that are you know
our bottlenecks that yet lit up every
time you do things and those are places
that are opportunities you know to look
for more parallel ism where you can
potentially move either workflows people
and data around to be able to get more
again more parallel ism through through
your data flow one of the thing to kind
of focus on your specifically people who
and are looking at developing software
is that you know within that context it
actually is really important to make
sure you enable your users to be able to
have fine-grained control over the
different types of data that may be
dealing with and where that it gets put
it's the flip side of being able to
manage the storage it's being able to
allow the softer to be able to segregate
the data so that you can manage your
storage and I'll show you an example of
that in a little bit with final cut but
you know for those of you you know who
are doing media applications who are
starting to think about this you know
you know look at ways to provide this
kind of fine-grained control for your
users more opportunities for enhancing
storage capability are thinking about
now where your be kind of the other side
of the data flow I'm assuming most of
you have been to some of the other
exeunt sessions and in are aware that
the you know infrastructure for dealing
with a fan is both fibre channel for the
data access and an ethernet path for the
metadata the metadata is the basically
file system operations aspect of of the
of the file system that that
communication path occurs over an
Ethernet channel as opposed to the fibre
channel fibre channel is really designed
for again that big packet large data
when you're doing lots of file creation
and file deletion that's one of the
cases where there's a large amount of
traffic that tends to go back and forth
between the client and the metadata
control of them basically the server
aspect
in the X an environment that basically
is the arbiter of how the information is
ultimately laid out in the disk when
you're dealing with video again big
files but little numbers of them you
don't tend to do lots of metadata access
usually it's a I need to find where the
file is there's a real quick message to
the metadata controller metadata
controller says it's in this blocks on
the on the storage and then from that
point forward the data flow typically
goes through the fibre channel when
you're dealing with small files and
you're doing lots of creations and lots
of deletions there's a huge amount of
traffic that goes back and forth between
the client and the metadata controller
and if you're doing a lot of that
particularly over a large number of
clients this is another opportunity for
a bottleneck so as we've talked about
for fiber you also need to be thinking
about switches for your Ethernet you
need to be looking at isolating that
particular path of the Ethernet traffic
so that the metadata you know traffic is
not you know interfering or getting
overrun by potentially you know browsing
the web or getting email off of your
mail server or you know connecting
through a network attached storage
device with some variety so it becomes
really important to think not only of
your fiber infrastructure and where the
data flow is but potentially what your
Ethernet traffic is within the
environment excuse me the other thing to
look at on your metadata controller is
you're doing more of these types of
things is ram helps a lot metadata
controller tend to the cast this kind of
information it is something that an Ram
will help live and multiple metadata
controllers to can help spread some of
the load one thing to think of at large
is look at keeping much of your caching
activity on your love on the local
system on the local client cash a cash
for is typically a files that are
created by an application purely for the
purpose of speeding things up its job is
to basically the pre-render material or
two
do something so that it doesn't have to
redo that operation many many times it's
one of those pieces of information that
typically can be thrown away because the
computer well ultimately will will
recreate them so if by keeping them on
the local system you're not losing any
important information in your workflow
and you're keeping the amount of traffic
between the local machine and the rest
of the infrastructure down to a minimum
the cash is really intended to be kept
fairly local so if you have potentially
render files or various cache files the
logs those are the kinds of things that
really are encouraged if possible income
cases you can't but if possible to try
and keep these on your local system that
can only take a load off of the rescue
infrastructure the other thing you
should consider particularly in the case
and I can speak for final cut is that
you should really consider keeping the
project files themselves locally that's
not the media the media ends up getting
can have kept on the shared storage but
the projects that you're actually
working on is kept local it's one of
those things that tends to get accessed
a lot it's an example of lots of small
file reads and writes and it's one of
those things that again isolation you
know can help a lot with really the
whole goal when you're looking at
enhancing the storage capability within
your environment is finding a way to
spread the load of across as much of
your infrastructure as you can and avoid
single choke points and that's where
drawing it out as tedious as that maybe
it can be really really helpful in
figuring out where those single points
of acts are access are we doing it
within your fiber infrastructure or
within your network infrastructure and I
want to take a little commercial break
here and talk about redundancy it's one
of those things you want to think about
here you know when you're dealing with
the sand that you've now gone from lots
and lots of individual seats that are
fully isolated to you know the ability
now to share lots of material it also
means you have an opportunity for single
points of failure so you should be
thinking about you know that from the
get-go you probably don't want to take
the chance of taking down your whole
interest
sure because a component fails as
something happens and there's some ways
to protect yourselves from that
certainly reported raid in a raid
configuration doing mirroring is one of
the you know the things that you should
look at there's you know looking at
failover method metadata controllers
within the the fan environment you can
even use and particularly in small work
herbs it might make sense to actually
use even one of the clients as a
potential failover in case it's one of
those things that you hope will never
happen but if it does at least the
system will fail gracefully and you
won't take your whole infrastructure
down another thing to look at is multi
passing multipathing is basically using
those two strands of fiber that are
coming out the back of the computer you
know as potentially multiple paths to be
able to get to your data in case one of
them fails and I believe one of the
other sessions talked a little bit about
how you know you can use multi passing
with within the the axion environment
certainly one of the most important
things in any situation is back up and
so back up is you know something you
should factor in when you're doing your
site planning and the other data
consider of course is is power
protection this is a great place to be
putting battery backups or other other
things so again just one of the things
you want to think about is you're going
through the process of setting up the
environment that you consider now that
you have another opportunity for a
single point of failure and look at all
the places that you can protect yourself
moving on to security and collaboration
dealing with multiple users in a media
environment is a real plus but there's
some real challenges here you know it is
a lot lot lot harder than dealing with a
basic file system server setup I think
you know from the get yo should
understand that as you're starting to
think about how it works here you've got
to think things through a little bit
more and the reason is it's more
difficult is because in a typical file
server setup your you talking about you
know someone being able to log in to a
file server to get to their files and
just you know save and retrieve their
files as you start dealing in
in a multi-user media environment you're
not actually having to think about the
collaboration aspects of that and how
you actually can share material across
multiple people if you're dealing with
that in a fire in a network file system
environment then you're probably dealing
with many of the same issues but you
typically don't tend to deal with that
in that environment to deal with a
shared media environment certainly with
Tiger being it now so the supporting you
know acl's excess control is and
actually that will be an incredible
benefit unfortunately we don't have that
right this moment so until then one of
the real things that you have to kind of
factor into the whole process is
planning make sure before you even put
the first piece of fiber to a computer
that you've thought this out because
it's one of those things that six months
into it you're going to go oh my gosh i
completely forgot i didn't think about
this and what you almost are stuck doing
is carrying it all apart and starting
over again so it's really important to
actually take the time upfront to
actually do the planning there's a lot
of parts to deal with here when you're
dealing with shared media and we'll talk
about what some of those things are and
we talked a bit about performance well
one of the they you know the big
challenges is that you know security and
performance don't always mix real well
and as you'll see we'll get into this
they tend to collide in a couple of
important ways that you can plan for and
work around but that's because it if you
do it up front with planning and the
other thing to think about is when
you're dealing with multiple user
projects where you're going to be
sharing material that again you need to
think it out a little bit ahead of time
so as you're planning for multiple users
one of the things that you know I
consider to be an absolute must is you
need to use a centralized directory
server of some type I'm not going to say
which one you've got to use but the fact
is you need to make sure that you
centrally manage your users in this
environment because as you're now
dealing with all the way UNIX
permissions in this missus in this
system files are going to have you
IDs applied to them and if you don't
think about what the things are that are
going to be assigned to those different
you know files what user ideas and what
group IDs and as soon as you start to
deal with sharing those files you know
six months down the road after you've
you know finally realized gee maybe I
need to worry about this you could find
yourself in a lot of trouble because
there wasn't something that you thought
up front and then you don't have a
centralized way of managing so it's
really critical that you manage the
users within some environment and some
centralized way and you need to just get
over the fact that you are going to
become an expert at UNIX permissions
this is one of those things that you
know in this particular situation it's
it's something you can't avoid it's a
going to be very critical in terms again
how you lay out your file system how you
deal with the multiple users and you're
going to have to really understand you
know at a time how you want to you know
put things together you might want to
look at putting together you know a
fight point in kind of checklist if you
will and look at each of the
workstations within your environment and
look at for example whether you're you
know how your media workstations are
going to be configured in the
environment how your normal office
workstations are going to be configured
so in the North kind of normal office
workstation environment I wouldn't
expect any big surprises this was should
work as it normally does using home
directories or whatever on your network
attached storage or whatever solution
you have those particular things will
probably work pretty much as you're used
to for media work stations on the other
hand deciding whether or not there's
going to be a dedicated workstation per
user or whether you're going to allow
them to move around within the
environment if something got to think
about because in a media environment
there's a lot more than just a few
preferences that go from place to place
in these setups particularly today and I
certainly think over time this will get
easier but at the moment anyway it's a
really complicated problem because as
you move if you're going to move users
from workstation to work station usually
the project its performance
characteristics it's expected
asians in terms of that real-time
behavior what kind of devices graphics
etc you know can be very much tied to
the workstation that that that project
is being done on and if you move around
you may find yourself having to redo an
awful lot the other thing is is there's
a lot of other material that tends to go
with the projects scratch files and
render files and things like that and as
you move around from station to station
you'll find you won't will end up having
to effectively recreate that you know
through the process and that can have
effectively slow down you know the kind
of usage model that people may have you
want to think about where where the home
directory is going to be for a given
user in that environment because the
home directory in a media environment is
going to get hit a lot that's where
those scratch files tend to be put
that's usually where a lot of other
cached information is kept within the
within the system unless you manually go
and override and set that up for each
workstation and user they're going to
end up having to you know one it's going
to end up putting a lot of load back on
your your your infrastructure and two
you're going to end up finding out that
your performance is just not what you
expect it to be things are not working
the way you expect one of the thing to
think about to at least right at the
moment is that the home directories tend
to be when you're dealing with an
environment where you've set up a
centralized no file server for for
dealing with multiple users the home
directories are actually connected over
the network they're not connected over
the fiber and this can be very confusing
and again I think this is one of these
things that over time it will change but
you'll get Network beer get performance
characteristics of home directory access
that's network based as opposed to
fiber-based again placement of where all
of the material that has to go with an
individual user can really have an
impact on performance so you want to
think about
you know how you're going to be doing
that you also want to be thinking about
how media plays into this environment if
you have globally shared like stock
material permission set up in that
particular situation is reasonably
straightforward usually can keep all
that stuff together and usually stock
material is something that can be shared
again you know by all the users within
within an environment but when you get
to per project media this is where your
permissions issues tend to get more
challenging and you want to think about
when you're setting up projects within
your fan environment you know how you
set up your directory tree how you set
up the permissions within that directory
tree usually when you create a new
project for example you want to make
sure you create a UNIX group for that
project so that you have the ability of
being able to manage which users
actually have access to the material
within that project and you need to do
that from the get-go you also want to
make sure that the directory tree within
the project is set up appropriately with
the permissions not only for the whoever
the owner of that particular project is
but that the group permissions are
correctly and properly set and that the
media within that project within that
directory tree is mapped as appropriate
based on the performance affinities that
we talked about earlier and this is
where it gets really complicated because
you basically have this one tree of
information that you're dealing with in
terms of how your projects organize and
a totally different structure that
represents how your media is actually
stored across your storage and these are
you know multiple dimensions that you
have to factor in at once because it's
so complicated he really encouraged you
to actually draw a map of the projects
and how they're organized over your
storage again not only in terms of their
organized how the user sees it but more
importantly how they're organized in
terms of the storage so that you get the
performance expectations that you have
when you get to this level this is again
where our dick tends to get kind of hard
merging the needs of sharing material
and the needs of being able to get
performance you know can be you know
rather complicated again you're dealing
with multiple dimensions of of hierarchy
here that need to be fettered in and
it's one of the things that can really
help you over time is making sure that
from the get-go as you're planning that
you can think about putting together a
consistent structure for all your
projects and I'll show you in a moment
you know a very simplified example of
this but you really want to look at
organizing your projects by performance
types by the kind of material you have
so they have the ability later to go
back in and apply the affinities that
you may want to apply so you can make a
performance adjustment you may find on
getting more and more users the
performance was going down I need to add
storage but if you didn't organize the
material well to begin with you're going
to find you have to back it all off put
it back on and restructure and that will
take you down potentially for a few days
so one thing you might want to think
about is you're dealing with storage
again is kind of this two dimensions to
this is how your material is structured
where you put your you know shared
material at your stock media and within
the various projects how you structure
it so for example within each project I
would you know encourage you to isolate
your media from the metadata basically
the project stuff the little file so
you've got all your media isolated you
know from all the little things that
tend to go with your project and thats
things maybe like scripts or notes that
you have or other small files project
files for example that may make up your
project that you're doing those are the
small things and keep the media separate
from that and even within the media
looking at separating the media out by
type certainly looking at you know
breaking out audio and video separately
the advantage of that is is that you'll
find again the requirements for audio
and the requirements for video in terms
of speed in terms of the amount of data
that's moving are pretty different and
you may be able to organize your storage
appropriately to be able to give you
much better performance the good news is
that if you look at organizing your
projects this way from the get-go you
can reorganize your storage separately
from this because the users don't need
to know how your storage is organized
and they just care that if I go down and
look in these particular folders I'm
going to find the stuff that I need to
find so think about it in advance so
that gives you the flexibility of being
able to get your storage organized in an
effective way I've been talking a lot
about scratch data through a number of
these things so this is an example from
Final Cut Final Cut allows you for
example to segregate a bunch of the
ancillary files that you know tend to be
involved in a project you may typically
think all i'm doing a video project i've
got certainly my final cut pro document
and then i've got a bunch of video in a
bunch of audio well the reality is
there's usually and this is not you know
just you know for final cut but I think
for many you know sophisticated media
applications a lot of other stuff that
is generated part of as part of the
process in order to get performance and
in order to get the kind of you know you
visual expectations that people have for
final cut you have the ability of
segregating for example where your you
know your audio from your video and
where it's placed as we talked about
from you know a project organization
standpoint that can be helpful for
affinities you can in this case also
manage where some of the render files
the files that are used to you know
maybe you know render out you know
complicated transitions or effects those
are scratch files that are not
necessarily critically important you
know in terms of day-to-day but it's
important more from the you know getting
the kind of performance that you expect
and there's places where it puts those
and puts those things on the system and
final cut allows you to kind us again
you can segregate where that stuff is
kept you want to keep that stuff locally
you may want to put it on the separate
volume or within the sand and there's
other caches that in the case of Final
Cut are used where various you know
thumbnails for pictures and other things
that use little things that that are
used just to make the UI and other
things
you know perform a little bitter many
times most of the stuff can begin except
on the local local system you you know
don't lose anything by losing this
material other than kind of the UI
responsiveness typically in a given
session it's the kind of thing you
probably don't want to put on the sand
because it just generates a lot of
unnecessary traffic as you're thinking
about multiple person projects
independently of the security issues and
other things we've talked about you want
to look at other things to consider as
well to make it possible for
collaboration given that I think in the
case of media this is a pretty new world
in terms of able to get root you know we
kind of you know if you will you know
completely seamless that we know flow of
collaboration between multiple users
that actually needs to be a little
thought right now you know from the
standpoint of how you lay out your
projects and work in order to make it
you know work more effectively as far as
so for final cut for example you may
want to look if you've got a really big
project that you're dealing with
multiple people that you actually find
ways to break it into sub projects in
the film industry they usually do
20-minute real for example you know you
don't do the whole you know multi-node
movie it's actually done in 20 minutes
it's client in a little segment you can
actually break up your project into this
and it makes it a lot easier to work
with and in the case of Final Cut it's
actually pretty easy to create a master
project that brings all the pieces
together you want to think about and
make you know watch out for that you
know multiple uses shouldn't be
modifying the actual thing in the same
files from a developer perspective this
is something that you serve you know
certainly we're used to in terms of you
know CBS and you know other kinds of
tools and facilities there's not a lot
of that kind of you know facilities
right now in the media space there are
some media management tools that are out
there but most tools that exist today
you know don't handle concurrent
modification merging and other kinds of
things that you might expect from you
know source code or you know software
development and you know multiple person
projects the multiple personal project
model has been
very hard to to get to without something
like this and where you know this logged
media is actually can be shared in a
real way so this is an area where
there's weakness you know in this
particular an area what I think there's
also a lot of opportunity but you want
to think about you know how the
structure and organize things to avoid
these kinds of collision usually for
media files this isn't a problem
something like Final Cut Pro or logic or
you know the various you know dawes and
other you know digital audio
workstations other things most of them
are in a non-destructive editors the
media itself tends to get modified
rarely if ever it's usually what gets
modified is the project files and you
know corresponding render files and
things that are done so the good news is
you probably don't have to worry too
much about me people modifying the same
media files usually but you need to
think about it I mentioned early on it
it's really important to me to consider
making sure that you use the central
directory service in terms of dealing
with multiple users I want to reiterate
that again it's one of those things that
if you don't think about that up front
it will bite you I you'll come back
later and you go yep you're right you
need to think about that the other is
making sure that you can you know do
that organization up front to enable the
storage tuning later down the road two
really important things that require you
know some forethought so now we're
wrapping up here the looking you know
kind of forward from this point one
thing that's really interesting is that
with the introduction of X and and with
the just the dynamics of the market
certainly I think Apple and final cut
and xn are itself are really changing
the economics here this is something
that used to take hundreds of thousands
of dollars of infrastructure and cost in
terms of the workstations other things
that evolved to be able to put together
a shared media storage environment but
something like a fan environment
together and the economics at this point
are changing in a very dramatic way and
we've seen this you know certainly
apples been through this
kind of economic transition in a number
of different areas before you know
desktop publishing certainly video
production you know in general and now
as we looked in terms of the shared
media space I think we're going to see
the similar you know kinds of change the
economics are really going to change the
market we're going to see a lot more of
this going on there's some really
interesting interesting technologies
coming in tiger that actually they're
really going to help here I think
spotlight for example is going to be a
major major major plus here as you now
look at now having storage media
environments for metadata can be
attached to on the media and it can be
searched and found very quickly and
effectively means that sharing masters
of information actually becomes very
feasible and you know very practical and
it's been 11 of their other real
challenges looking at things such as you
know karate and core video just so many
implications of their the kinds of
things will see in the production space
are going to have another big impact as
well because they're going to be more
and more people involved in the creative
processes these tools become more and
more approachable you don't have to have
expensive hardware to be able to do a
lot of these things the challenge I
think as we move forward is how to get
media management you know more as part
of this integral process I think there's
a real developer opportunity there and I
think you know something that is an
obviously big hole that needs to be
filled in terms of now how do you manage
the media through the workflow and there
are some good solutions out there we're
bringing down to the scale at this level
and to be more approachable I think it's
going to be one of the interesting
opportunities I think the other thing to
factor here is that is just the actual
workflow process itself I think is going
to be an interesting part that this
enables that you know now having vast
amounts of information available to many
people in effectively real time at the
desktops means that you know managing
how the material gets through the
creative process through multiple people
and gets trucked through there
which is beyond just media management
but actually workflow process management
is going to be another big developer
opportunity another opportunity within
the space hopefully there are people out
here who are thinking about those kinds
of problems and certainly love to hear
about the cool things that your guys are
doing