WWDC2001 Session 103

Transcript

Kind: captions
Language: en
it's been it's been a great year for
Darwin and opensource starting with Mac
os10 shipping on March 24th and we have
great contributions and signed up for
Darwin open source and we'll be sharing
a lot of accolades in the numbers with
you as well throughout the week so
without further ado I'd like to kick it
off by introducing the director of
Korres engineering Brett Haley so good
afternoon the first session after lunch
hopefully you guys won't fall asleep on
me off we have a lot of technical
material in this session so I will try
and get over it
the focus of this session is to you know
basically give an overview of what
Darwin is all about and I'm sure you've
seen this picture a numerous times
that's just this morning and our focus
again really is on that bottom layer of
the system and there's a huge amount of
technology that exists there as we talk
about it today I think it's important to
note that you know we will use some
terms quite interchangeably
Darwin is the core OS of Mac OS 10 we
talk a lot about Darwin we talk a lot
about Pakistan they are the same thing
where they are not two different piles
of Technology they are one in the same
and from that perspective it's important
to note that the Darwin is also open
source so as we talk about the open
source aspects of Darwin and the
technology it's the same pile of code
the same technology that ships as part
of Mac os10
again they are one in the same so how in
the world do you describe an arm you
know the core OS or the operating system
of Mac OS 10 interestingly enough it's
technology that tends not to be very
visible if we're really doing our job
well you don't really see us so you
might consider that you know we're
certainly kind of the foundation of the
system
we're you know kind of deal with the
plumbing and the piping and the you know
wiring if you will that holds Mac os10
together again if we're doing our job
well you don't see the core OS you as
the developers will interact with
very you know at a very high degree of
interactivity but the developers should
rarely if ever be aware that the core OS
is there the architecture of darwin is a
very unusual in that we've you know
taken a basically best of class
technologies and pop-rock them together
to bring in the base of mac OS ten
together and there's a number of
different technologies that we'll talk
about in this session this technology
comes from a number of different areas
academia the open source community and
certainly some significant history at
Apple and next and there's probably
thousands of person years worth of
development and testing and usage in
much of this codebase but we've taken
all that technology and we've again kind
of brought it together in a very unusual
combination but tuned it toward the
customer base and developer base that's
unique to mac OS 10 so you've certainly
heard a lot of focus around things like
UNIX or certainly the heritage Mac OS
we've actually brought aspects of this
this technologies together to support
both of these environments and to really
focus on our customer base directly and
from that perspective there's an awful
lot of untapped potential that exists in
the system we'll talk a lot about more
that what more of that during the week
of course but in terms of providing that
modern operating system base this is it
this is the basis that you know really
enables all those capabilities and
because of the power that exists there
I think we're enabling you to create
applications that you couldn't do on
previous versions of Mac OS from a
feature perspective you could consider
that Darwin or the core OS is
responsible for the preemption aspects
of the system the you know memory
protection the basically it's
responsible for managing and
coordinating the multiple address spaces
that represent the system each of the
applications in the system exists in a
separate address space it supports the
application environments provides all
that plumbing and wiring and stuff that
allows the application environments to
exist on the system and to be able to do
there
export their particular brand if you
will of memory management and such up to
the application the core OS is
inherently device independent and what I
mean by that I mean that seems kind of
an obvious statement but that we don't
actually have any particular preference
to particularly any I don't know an i/o
or device architecture we realize that
you know technology in the i/o space is
evolving at a fairly furious rate and
we've tried to make sure that we have a
design that will enable us to move
forward and be able to take advantage of
the fact that we make the whole widget
and to be able to you know enable new
hardware and new technology together new
software technology it's a very flexible
base if you as we talk about the pieces
of the core OS you'll see that they were
designed to support a number of
different kinds of categories of
application spaces and Hardware spaces
and and such and we've really kind of
tried to leverage that flexibility of
the system to give us a base which we
expect to last for the next 10 20 years
the course is also designed to be
scalable and you can interpret this in a
couple of key ways it's scalable from
the hardware's perspective in that we
you know support certainly various
classes of you know processor
architectures g3 g4 but also be able to
support multiple processors in a system
so support for symmetric multiprocessing
from that level of scalability as well
as scalability from the i/o perspective
being able to support you know the
storage network devices and large you
know very large capacity devices on the
system or high bandwidth networking with
Gigabit Ethernet for example but it's
also scalable in the kind of customer
set that we've tried to support the same
base Darwyn core OS is used for our
client operating system Mac OS 10 and
Mac OS 10 server 10 point oh that was
announced this morning so the same
operating system core is used to drive
both the server and the client
environment and that the system itself
basically is tuned
you know by those environments to kind
of provide the appropriate level of
service and capability and being an OS
and things like networking and all that
there's more buds worse than you can
imagine that's part of the system so
we're gonna spend a little bit in
actually do a tour of the chorus and the
various components and that are part of
it and you know first of all this kind
of gives you a bit of an exploded view
of that Darwin layer of the system and
there's a number of pieces that make it
up we're gonna start first by spending
some time and talking about mark mark is
kind of the actual foundation layer of
the system it's designed in fact as an
OS foundation and abstraction layer if
you will and it's responsible for
managing the processor and memory
resources of the system it's basically
the piece of the system that actually
schedules the processor and give an
application or thread in the system for
being able to run and it's important to
note that at this very very basic level
of the system the threading is that
threads are considered to be you know an
essential building block of Mac OS 10
that scheduling and the various services
that go with it or actually supported
all the way down to the very base of the
system it's not a tacked on the top
concept mark is also responsible for
handling the memory protection the
memory management basically if you will
they both the processor and memory
resources of the system all the VM
capabilities and such because mark keeps
everything abstract it as far as how
memory is concerned and the various
address spaces that exist in the system
it also has to provide some very key and
basic services so that there's a way for
the user level applications and the the
kernel to be able to communicate back
and forth together and that's done
through something called IPC
inter-process communication RPC remote
procedure calls that is a fundamental
again key building block of MOC and it's
the way MOC uses to be able to
communicate with any other portion of
the system
Marc is inherently policy neutral it's
again it's designed to be a very much a
an abstraction layer to the processor
and memory architecture of the system
and it doesn't actually have any concept
of things like file systems or i/o or
networking or anything security policies
things like that
that's not mocks job it really is
designed to be a very low level
abstraction layer the mock we use in Mac
OS 10 is based on for those of you that
follow this technology is based on a
mock 300 plus a bunch of stuff that has
been added to it by the various players
that have been participating in its
development it's derived from mock that
was developed at CMU avi tavini and of
course was one of the original team
members of the of the mock team at CMU
but it was taken on by the OSF reach
Research Institute a number of years and
in fact Apple funded a lot of that
development work at the Research
Institute and and when we took it in and
incorporated as part of Mac OS 10 of
course we've done considerable amount of
work to enhance the the 300 base beyond
even those starting points it as fare
very much from its original design point
for MOC was to actually be able to
support things like SMP and some of the
work that was done in 3 oh and later
also has architectural support for being
able to deal with various mechanisms and
in facilities for supporting real-time
and we'll talk about that a little bit
more and certainly more in the kernel
session later this week
the wall is talking to us the markers
also has built into it because it's
responsible for managing the processor
resources the capability of supporting
multiple scheduling policies for example
built into to the market ships with 10
is the ability to schedule certainly
either kind of a real-time scheduling or
fixed priority based scheduling or time
sharing based policy scheduling and
there's actually the ability to support
other types of scheduling policies
within the system Marc also tries to
abstract the mechanisms for being able
to deal with the other half of the
virtual memory system so VM certainly if
you look at the management of memory
there's the managing the actual you know
parceling out of the address space
that's available for a given application
but there's also the other aspect of it
which is what to do when you try and
access more memory than you actually
have physical RAM installed in your
system for and MOC actually interacts
with a number of external interfaces to
be able to support being able to store
to be able to do handle the backing
store for the OS either on the file
system or and certainly it's capable of
other kinds of backage towards backing
store solutions Marc is also again it's
kind of repeating here but it's very
much designed to be very modular MOC was
intended to have OS personalities put on
top of it again it's policy neutral so
it's you know it's interface layer it's
abstraction layers of stuff that it
assumes that there's some other OS and
other RS technologies that sit on it and
that's very much a key part of its
design space now this is all fine and
interesting this is all down at the very
base of the system but if you're writing
an application for Mac OS 10
what does MOC mean to you well it's
important to note that all process
models of the system in other words the
things that actually kind of handle the
scheduling and the kind of an
environment which your application
rests within are all fundamentally built
on top of Mark Priven primitives those
mark primitives are tasks and threads
and we're whether you're using carbon or
whether you
classic or you're using Coco they all of
those application environments are
fundamentally built on these particular
abstractions all memory management is
also built on top of mock primitives so
at the very lowest level again there's
management of Hewell VM objects and the
handling of virtual memory and the
paging services and at your application
you're probably dealing with things more
like malloc or you know other kinds of
services get pointer or new pointer
whether depending on the type of
application that you're using but
fundamentally when he gets right down to
actually managing the memory at the OS
level Mach is responsible and again as I
mentioned before with Mach built on this
concept of inter process communication
that all kernel to user and user to
kernel and user to user communication
and then what I mean by user to user is
user space from a more process
architecture processor architecture
perspective well all that communication
that exists in the system between
applications and between applications in
the OS are done built on top of these
mock primitives basically inter-process
communication RPC or what you'll end up
hearing a lot in mention and number of
the documentation is mock ports now to
get into this a little bit more and give
you a sense of exactly how the system
actually uses some of these facilities I
want to invite Joe Sokol up on stage and
what we're going to do is show you that
as right you're running the applications
on your system that there's actually the
ability to look at the system much more
from a if you will a systems perspective
to see the kind of the impact of your
application that at the OS level okay so
many of you are probably familiar with
that's not the one with the applicant
with top running on UNIX which is
capable of showing you a lot of stuff
going on in the system what we're going
to do here today is just kind of take a
look at a simple little carbon app
because that's kind of the runtime
environment that isn't quite so
used to running on top of Unix and to
show you that in fact just because it's
carbon right it actually is making use
of a lot of these mock services so
you're going to see the fact that
they're threads
they're ports being allocated on your
behalf and we're going to take a little
look at some characteristics of
performance or lack of performance that
would like you to avoid okay so you've
got W so the little app is called this
white next event loop and the first
thing we're going to take a look at is
what it means to have an app that is
basically not only compute BAM but
basically running through it take a
slightly different view here a lot of
system calls ok so the weight next event
in the carbon runtime is actually
implemented with a you know a few mock
calls and a few BSD calls but when you
amplify that if you're just sitting in
the loop doing a wait next event with a
zero timeout basically you get really
really busy on the system now this has
basically three bad aspects to it the
first one is kind of obvious in a
time-sharing system our little test app
here his priority is diving way down and
it's going to stay down because he's
never blocking so you might not like the
responses responsiveness you get in this
app over time secondly one that you
might not think about so much as you're
keeping a number of pages hot all the
time because you're constantly executing
them so those are pages that the VM
system might have been able to you know
steal and actually give to a foreground
application that really needs them so
it's continuing to put pressure and
competing for for memory and then the
third thing is all of our power
management is actually driven
and triggered off of our Idol loop right
which is an actual mock thread that goes
that gets run when there's nothing else
to run so that is where we make the
decisions to put the system into nap or
doze mode you know to reduce the power
consumption not the deep sleep but these
other modes that allow us to cut back on
on power consumption and if the
application itself is basically
implementing an idle loop you know that
does not allow us to enter into those
modes another thing that we can look at
with top is some of the other resource
utilization so again you can see that
this is a very simple dumb you know
hello world style Carbon application and
he's managed you know we've managed to
allocate him two threads and 61 ports so
these are you know things that are
happening as part of the carbon runtime
model so another thing we can show is
let's go and cut down on the amount of
when we flip back here real quick sorry
this little bit easier to see in terms
of the CPU utilization and the number of
system calls we've now simply made the
wait next event wait for one tick so
instead of it just being strictly pulled
even just going to one tick wait really
reduces the load on the system if we go
to something that's a little more
approaching infinity then of course it
goes real prescient so even if you can't
construct the app to be purely an
event-driven model if you have an event
loop if you can at least make the times
that you're blocking for long that
really helps in terms of relieving
pressure in the system alright so now we
want to show the a memory leak so excuse
me
so again top can be used to see things
like memory leaks going on in the system
you'll noticed under under the art
private which is the resident private
column that all of a sudden we have a
little plus sign sticking out there on
the right and what that's indicating is
that that actually is a growing size
there so you can see the size going up
slowly this is basically a fork a leak
once a second now if we were to be
leaking every time around that event
loop we don't really we don't really
enforce any kind of real upper limits on
the amount of memory that an applicant
obtained so it is probably a good idea
on your part to watch your app and make
sure that it doesn't have any leaks
because over time you know maybe the app
stays up for a long period of time of
course that would just keep going a
larger you know going longer and lot
growing larger and larger over that
period of time to the point where you
might run out of paging space on the
system so an app can approach three
gigabytes and size thank you
back to slides here so I think it's
important to realize that even though
you may be thinking of your application
in terms of the carbon API space or if
you're writing a new app in Cocoa that
there actually is a lot going in on in
the system under the covers and that
there are actually tools that you can
use to be able to tell what the actual
resource limit usage in the in the
affects if you will of your application
are on the rest of the system if you're
interested in learning more about the
Darwin kernel there's a session later
this week on Friday that I encourage you
to go to over in the civic center and
this is this particular session will not
only cover the mock aspect of of the
system but will also talk a bit quite a
bit about the bsd kernel as well moving
on in our tour now into the i/o kit
space io kit is an object-oriented
framework that's basically designed to
try and ease the process for developing
implementing drivers on Mac OS 10 it has
a lot of kind of native or inherent
capabilities that are part of that
framework support for things like plug
and play and and dynamic device
management and of course again those
things sound like you know mom and apple
pie kinds of things but it's very
important to note that again from the
very basis of the system that the ir
system is intended to be able to support
devices that can come come and go as far
as the system's current concern that the
i/o system itself has no inherent policy
in terms of the you know kind of the
arrival or that or the departure of
given devices on the system that policy
is actually managed up how much higher
you know in the system space for example
whether the the file system has a
dependency on a you know volume that
happens to exist on a particular disk
but IO kid itself supports the ability
for devices to hit to come and go in the
system and is a very inherent capability
another important aspect of i/o kit
that's that's really there from the core
is support for power management is that
expected that all devices in parts of
the our system participate and are
involved in the process of managing
power on this
and we'll talk a little bit about more
of that in a minute in a minute and as
well I okay it is intended to be very
modular and extensible we don't know
what kind of devices may appear five 10
years from now and we've tried to make
sure that the basic architecture for our
kit can be very flexible as new i/o and
our device technology come on on the
scene in addition to that basic
capability of course il kit does support
and provide the abstractions for all the
common you know class of devices that
exist in the system things like scuzzy
and 18 USB and stuff like that and those
are all done through another aspect of
the framework called families and we'll
talk about that in a minute as well
again iokit is a framework it's actually
an object-oriented framework c++ based
and it provides basically the common
kernel services and facilities for being
able to support IO in the system and it
also has the facilities to be able to
enable the communication from the kernel
to the user space so that the actual IO
can get up to where the applications can
take advantage of it in many cases
that's done through all the layers of
abstraction but it the inherent
capability for being able to do that is
built into I okay I okay if in effect
models the physical world if you will is
you'll you know think about your
computer there's a motherboard and
there's chips and there's potentially a
PCI bridge and potentially PCI cards in
a graphics card or a scuzzy card that
that's in there and I okay it actually
is responsible for being able to develop
that model if you will of all those
devices in the system and then once it
has that model being able to actually
connect up the appropriate driver to
that given Hardware so that the rest of
the system can actually talk to it the
protocol specifics for any particular
class of devices and something we call
families a family basically is in a if
you will a domain appropriate
abstraction for a particular device a
good example of this is if you would
compare something like scuzzy and audio
they're pretty very you know very
separate very very radically different
approach you know
use if you will on i/o you're probably
not as interested in setting the volume
of your scuzzy discs you know for
example and so the families actually
provide the appropriate API set and
capabilities for that particular kind of
device space the actual device specifics
themselves exist in drivers so that's
the the part of the system that directly
talks to the hardware and they in and of
themselves then communicate with the
families and then the families with the
rest of the system as I mentioned power
management is a very key part of macros
Tanner you guys were at the keynote this
morning and saw obvious demo and it's
it's really quite amazing to see
something like the titanium you know
effectively be a week you know before
you can open it I've heard a number of
you know refrigerator door kinds of
stories of you know trying to make sure
is it is it really off or or not but
it's because of the the i/o the
architecture for power management that
that actually becomes possible it's
integral to the i/o kid architecture and
it's based on this concept of power
planes much like I mentioned that i/o
kit tries to model the physical world
from the power management perspective
there's another model that models kind
of the power distribution in the system
and they may not correspond in a
one-to-one basis depending on how the
hardware is designed the
interdependencies at a power level
actually may be different but Iook it's
responsible for trying to keep that
abstraction in that that set of
relationships together it's important to
note that for Mac OS 10 the way we were
able to get the speed and performance at
of wake up is that because of the full
architecture of both yoke it as well as
the rest of the system that this is the
kernel itself is multi-threaded the
system itself is multi-threaded and that
the wake up process of waking up the
system is actually done as much in
parallel as possible and as a result
you're able to actually enable the parts
of the system that are necessary to get
that application up and going as quickly
as possible as long as it doesn't have
any inherent dependency on you know the
disk or a networking device which may
take a little bit longer to come up to
speed you're able to get up and use and
interact with your system in a UI level
very ver
quickly if you're familiar certainly
with mac os9 in waking up sometimes it
can take a pretty substantially long
period of time and part of the reason
for that is the fact that the system is
very seriously dependant on making sure
that all those components actually are
fully awake before you can access any
part of it
in addition things like Apple talk for
example have to go through the process
of renegotiating if you happen to have
Apple talk enabled on nine have to go
through the process for any
renegotiating node numbers and such like
that which end up having to be done
before the system can fully wake up on
ten again because these things can be
done in parallel for those parts of the
system that aren't dependent on on
having to use Apple talk for example if
you have that enabled you don't have to
wait for that particular renegotiation
to occur before you can use the system
from the application perspective you
probably won't deal with Ione kit much
directly most of the devices are
actually handled through other layers of
abstraction certainly if you're
interested in discs chances are the way
you're accessing those discs or through
file systems or networking through
either you know your ot interfaces from
carbon or form from sockets or other
network services that may exist or if
you're interested in input events most
likely you're dealing with the actual
event system so again for most
application developers you probably
don't ever directly and get involved
with the specific devices however we do
have situations where there are specific
user clients that that may exist when
there is an application need to be able
to access a particular kind of device
USB and the human input device manager
is a good example there are a lot of
sessions for i/o kit this week so if
you're interested in various aspects of
it be it firewire or USB or the storage
drivers or the basic architecture I
encourage you to go to that in fact
immediately after this session and if
there's another overview because I
couldn't possibly get into enough detail
on i/o kit here but I encourage you to
get to go to that session if you're
interested in you know working either at
the device level or are interested in
devices from an application perspective
moving on now in our tour to the bsd
portion of the system
and this is where that power of eunuchs
comes into play so there's two aspects
that we look at for four bsd and it
helps us kind of manage the different
kind of views and responsibilities that
exist the first is the bsd from the
kernel perspective and the bsd in the
kernel is one it's based on bsd for four
with a lot of intense integration that
exists between it and mark and it an i/o
kit so if you look at the kernel
environment for mac OS 10 it's basically
the bsd kernel io kit and mock are the
three key pieces that make up the kernel
environment the bsd kernel is what is
actually provides the OS personality if
you will
api's and services for the system much
like mock is completely agnostic as far
as the policy is concerned vsts
responsibility is actually to manage the
policy of the OS so it actually provides
the application process model they
support for things like signals and
tracking of file descriptors and things
like that when your application you know
dies or gets torn down its BS Dee's job
to make sure that all those rate
resources are all reclaimed and ran and
handled it also is responsible for the
basic security policy of the system so
if you will that's the concept of
multiple users on the system or even
when you take it down to the fastest the
mauville being able to manage access to
individual files or or ask pieces of
data that basic security policy is a
very inherent part of the kind of the
bsd environment and is an enforced again
down at the kernel level it's also
important to note that both the file
systems architecture and the networking
architecture for Mac os10
basically fit inside the BSD in kernel
space they are both based on the bsd
implementations and we'll talk more
about both file systems and networking
in a few minutes the system framework is
basically how we try to describe or
abstract the set of interfaces that bsd
provides and it's basically the system
calls and if you will POSIX level
interfaces and bsd system api is if you
will for the system if you're accessing
things are like porting a bsd
application this is actually the api is
that you're mostly what most likely will
write to the application environments
themselves are tend to be written on top
of this api set the system framework is
also responsible for other kinds of
services as well things like pthreads
the math library the basic c libraries
that basically the very low level set of
api's and services that you expect to be
able to support an OS environment the
user environment is kind of another if
you will almost a fourth application
environment that exists on the system
you've probably seen all the other
architectural diagrams and they refer to
classic and in carbon and cocoa or in
potentially even java it's important to
note that in terms of the actual system
architecture there's actually one more
and it's the bsd user environment it's
where things like the shells and command
line scripts and things like that
actually run it's where a lot of the
network administration client tools
actually execute the things that are
kind of common command-line tools and
facilities that come from the bsd
community it's also a huge number of
file you know various file tools for
creating directories and getting copying
files and that kind of thing as well as
tools for being able to kill other
processes or top like i show you we've
showed you a few minutes ago all these
things are available and exist within
the bsd user environment it's also where
some very key and important kind of
system services or network
administrative services exist in the
system
things like net info there are dns
services bind network time protocol
server the system log and things like
apache and it may be important to note
here that if you for example are running
you know the client environment and you
happen to turn on personal web sharing
that it's actually apache that ends up
getting launched and run on the system
really the single most used web server
on the planet
but all that stuff exists kind of behind
the scenes and it helps support the
system from a developer's perspective
it's important to note that you really
want to be kind of aware of how BSD
influences your life certainly again it
contributes to the process model the
system the support for things like
signals and management of file
descriptors and other important OS
resources are all managed by the BSD
Process Model but there's also things
like environment variables and things
like that that are also available the
kind of the standard things you would
expect in it from a UNIX process it's
also responsible for the security policy
again and remember those will impact the
way your application behaves and from
that perspective you should be aware
that things that you maybe have been
used to doing under Mac OS 9 you know
that the security policy of the system
actually plays a significant role here
in immaculate in whether or not you can
access certain files and places that you
can write into the system and things
like that it's also important to step
back a bit from that from the
application perspective and look at BSD
from the developer perspective there are
a ton of tools that exist in this
environment now certainly we'd try and
discourage the BSD kind of UNIX user
experience for our typical end users but
from a developer perspective this is an
incredibly powerful part of the system
as you saw with top you know there's
ways to actually look at the system kind
of from behind the scenes this
particular setup we had was actually two
systems we telnet it into the system
that was running the application to run
top so we could observe it without
impacting the UI in any particular way
kind of being able to you know put
instrument probes if you will on the
system but there's a huge number of
other tools that are available there
that might help building your
applications or you know just managing
your development resources and in the
environment that you work in importing
UNIX applications to Mac OS 10 it's a
you know important to note that there's
a lot of options that are available here
later this week there'll be a session
that'll actually show you a couple of
ways that you can approach this if you
happen to have your favorite Phoenix
favorite UNIX tools or other
that you you might want to use on Mac
os10 porting them over to 10 is actually
fairly straightforward but and in fact
you can actually use a kind of a
stepping stone approach to getting your
applications on a Mac OS 10 if you
happen to have one that even has a GUI
there I believe there's a third-party X
Windows server for example that's
available so if you want to pour it over
an X base application and get that up
and running very quickly and then spend
the time to actually wrap it in
something like cocoa there to replace it
with a more aqua friendly you know more
user friendly experience that ends up
being a very easy thing to do under Mac
OS 10 it's also interesting to know that
it's also very possible in something we
actually do a lot ourselves being able
to use UNIX tools from within a GUI
application many of our control panels
for example or actually you know execute
command line tools in the background
they actually you know are able to
launch and interact with a UNIX command
line environment again that's completely
hidden from the user Apache as a pro you
know it's an excellent example of that
if you go out and get an Apache manual
I've had to do full of system
administration there's a pretty thick
books out there for administrating the
command the the various configuration
files and stuff your user should never
have to deal with that and it's possible
to actually wrap that kind of power in
something that's very easy for them to
use there's a number of sessions also
that are related to the BSD aspect of
the system leveraging bsd services
latter part of the week will actually go
through in demo porting some UNIX tools
and wrapping them in a in a cocoa
application but there's also a couple of
other important sessions as well
threading on Mac OS 10 and I will talk
about the P threads and some of the
other threading options that you have
available in the system and as well
support for directory services and such
on the on Mac OS 10 moving on now into
file systems oops
sorry about that
filesystems of course is a lot more
complex than just this little blob on
the diagram is actually a lot a very
very broad range of file systems that
are supported within system and this is
all based on basically the the BSD file
system architecture it's important to
note the file systems implementation is
part of the bsd kernel environment and
it's based on the VFS design there we go
out and get your favorite unix bsd unix
tex well actually as a great reference
for being able to learn about the VFS
architecture but it's basically a
stackable virtual file system model
which we've extended both from the basic
POSIX interfaces beyond that to be able
to support more rich file systems such
as HTTP plus it's apart of file system
type in fact on Mac OS 10 we support a
huge number of file system types today
and there are third parties that are
also been making file system services
available recently I read that the
Andrew file system for example has been
ported now to Mac OS 10 we've also
extended those file system interfaces to
be able to take advantage of Unicode so
you'll find that there are ut8 utf-8
interfaces available so that you can be
able to take advantage of localized file
names and such but it's also really
important to note that from a way we've
designed Mac OS 10 much like how we've
approached the i/o system that we have
no particular file system affinity at
the OS level in that we don't
particularly care whether or not a Fitch
HFS+ whether it's ufs we are
intentionally trying to design the
architecture of the system so that it
can be flexible across a number of
different file system architectures as
you move up the application chain more
of the those high-level services and
capabilities are exported through your
applications the file system is
responsible for enforcing the file
system security policy of the system
again unlike Mac OS 9 in this respect
there's a concept of security does
existence
your ability to write given files
renamed even execute privileges all
exist in are enforced by the file system
there's a concept of users and groups
that is throughout the system and the
access control is all managed by this
aspect of the system the application
environments themselves tend to provide
the abstracted level of interfaces that
you'll tend to use if you're writing a
carbon app you're using the file manager
and things like resource Forks and
things like that are you know fairly
fluidly part of that particular
environment if you're writing in cocoa
you're going to be using something like
NS file and where the use of resource
Forks is not the the typical way of
writing an application in this space or
if you're porting a bsd application
you'll end up using the POSIX + + api's
and what I mean by deposit + + is this
the POSIX + the extensions that we've
provided for Mac OS 10 now to talk a
little bit more give you a better idea
what the implications are file systems
in the system it's Joe to come back up
again and again much like Mark where
there's a lot going on behind the scenes
on your application the same is true for
file systems
okay so we're gonna talk a little bit
about a tool called FS usage let me get
the top out of the way here that
basically is something that you're
probably not familiar with I don't
believe it shows up on other UNIX system
it's something that we developed here
but it will give you a comprehensive
list of all the files and directories
being touched and the size of the iOS
that are going on the amount of time
that you're waiting for that IO to
complete there's also on the far left
there you can see a current clock with
millisecond precision and then the name
of the the task that's actually causing
the iOS to occur so it's both
comprehensive both in kind of a global
manner if you're looking at the
basically allowing to look at all of the
tasks that are running in the system or
you can focus it in on a particular task
but I find it more interesting to look
at it with more the global method just
because there are lots of things that
are happening on the system that you
trigger when the application does
something that you might not be aware of
because a lot of the services are tied
in indirectly to some of these
higher-level calls it's useful for
exposing redundancies of access pretty
obvious and as we as I alluded to
earlier it can show you where your app
is blocking or waiting for i/o so if the
app feels sluggish you you can use this
tool to determine whether or not it's
it's due to lots of wait time for iOS
and I'll even show you where you're
waiting for synchronous page ins to
complete so where the apps trying to
page fault some of its code and/or its
data and then one other interesting
aspect which you can I would suggest
reading or reading the man page so you
can see how to turn this on but there's
a little backdoor trap that basically
enables the display of the higher-level
carbon calls so for you Carbon
developers out there who are curious as
to what system calls what UNIX system
calls we turn a lot of these higher
Carbon file manager calls into FS usage
will show you that it will actually show
you the encompassing or the calls being
encompassed by these various higher
level calls so we do something like open
the menu and then maybe bring that up
you get an idea of all the stuff that's
going on but this is a real useful tool
for you know kind of getting a good idea
of what's actually happening in the
system you know from some of those
higher level calls that you're making it
may turn into just a whole lot of other
stuff going on so it's very educational
thank you so again it's worth noting
here that for much of the things that
are going on in your applications that
down deep underneath that application
there's a lot of OS you know facilities
in play to make those things possible
and there are a lot of very very
powerful and useful tools on Mac OS 10
that you can take advantage of to be
able to see the system in this kind of
from this kind of perspective if you're
interested in learning more about the
file system there's a whole session on
that as well on Wednesday in the morning
encourage you to go to that Clark is
always quite entertaining moving on in
our tour now to networking for
networking onto Mac OS 10 again
networking is part of the BSD kernel
environment and it's based on the as you
put this morning there and a network
with a reference stack if you will for
for IP and in the internet world we're
based on the four-four tcp/ip stack
which we've actually synced up with
FreeBSD 3-2 and this is a sockets API
based environment it comes with a quite
a lot of powerful capabilities built
into the networking stack certainly
support for things like multihoming and
routing and firewall and network address
translation or nap mechanisms these are
all pretty basic services that exist in
our networking stack
another very key one however is this
concept of Auto configuration and again
I'll be touched on that the keynote this
morning but it's a very important part
of our architecture and a key part of
foundation that we're working on as well
as we move forward with Mac OS 10 for
the networking environment the way that
you actually go about extending that is
through something we call n keys or
network kernel extensions um there's a
basically a model that we've developed
for being able to extend or enhance the
the capabilities of our networking stack
much like you can create drivers a
forgiving piece of hardware or you can
write a fire file system kernel
extension for being able to support new
in exotic file systems there's also ways
for you to be able to extend the
networking aspect of Mac OS 10 one
important part of the architecture for
our networking is actually how we
support classic and we very part key
part of that architecture of the system
is to be able to support so that the
virtual environments above us for
example like classic or maybe something
like a virtual PC like environment can
actually share the the core networking
services in configuration and setup so
it's not necessary to have multiple IP
addresses for each virtual environment
that exists on the system we also
support access of the networking through
an open transport layer that exists for
porting carbon applications things like
Internet Explorer and things like that
all use the open transport services that
we provide as carbon as a way to get
their app over quickly onto Mac OS 10 we
also support PPP and PPP over ethernet
as key services that exist in Mac OS 10
and pppoe is something that's like new
for for Mac OS the ability to be able to
inherently be able to connect to your
DSL or you know cable modem provider
also built into Mac OS 10 is the support
for DHCP both the client and server so
this is actually used in different
situations depending on how Mac OS 10 is
set up certainly as a full server
environment you can administrate and
manage DHCP for things like netboot
and stuff but it's also used for being
able to do ad hoc networking ad hoc
networking is the ability for you know a
small group of Macs to be able to just
to set up either either through a
crossover cable or over airport and
without having to have all the
infrastructure full internet environment
but so that you can transfer files back
and forth for example and those actually
take advantage of things like DHCP in
order to make that work mobility is a
very key part of the direction of our
networking architecture networking under
Mac Westin is already incredibly rich
and powerful through the multihoming and
other aspects when one area that has not
been historically been well known in
UNIX networking is the ability to deal
with mobility and that's something that
we consider to be very critical so we've
actually spent quite a bit of time
developing a very flexible architecture
for being able to handle mobility be
able to support things like automatic
network configuration being able to deal
both configuration and reconfiguration
based on the you know link level
detection that exists so if you have a
portable and you unplug that cable out
of the back um it'll automatically
notice the fact that you're even removed
the cable and if you have an airport um
it'll automatically yes and silently
convert over completely to be able to
use the airport if you're using DHCP
configuration and all of this support is
very dynamic in addition we also support
multiple configurations and this is
stuff that Ivey demoed this morning so I
won't go into a lot of detail around
this but they certainly the ability to
be able to support multiple kinds of you
know interface and configuration sets
but another key aspect of it is to be
able the ability to support application
level notification our mail program is
into the first example of this use where
an application can actually be aware of
the ability of a network a network
connectivity and accessibility to the
network so if you actually have had a
chance to play with mail if you you know
remove the cable out of the back of your
system or lose connectivity to the
Internet mail will automatically go
offline it detects the fact that you
know
or have a connection and does the right
thing and when you plug it back in again
it'll automatically be involved in the
process of reconfiguring itself and
reconnecting it to the appropriate
servers and this is the kind of behavior
we'd like to see as we move on and move
forward with Mac OS 10 there are a
number of sessions for networking this
week networking overview exists tomorrow
and a number of other networking
sessions tomorrow both the kernel and
extensions for networking as well as the
configuration and mobility session as
well tomorrow and if then there's a
feedback form later this week so take a
little side trip here and talk about how
the core OS and g4 architecture actually
played together this is certainly kind
of outside of the architectural space of
thinking of the core us but it's an
important part in an impact sexually or
designed space for Mac OS 10 in terms of
how we get some incredible performance
on on the system first and foremost of
course is support for the velocity
engine for all the ultimate unit of the
processor and there's a number of
libraries that are available on Mac OS
10 there's a V V DSP library which is
responsible for basically signal
processing and fast Fourier transforms
and convolutions and things like that
and that library as it was existed on
Mac OS 9 1 but is also available on Mac
OS 10 new for Mac OS 10 however as a new
library called V Blas the vector basic
linear algebra subroutines which
basically are handled support for things
like a large matrix manipulation
basically this is this kind of thing is
used for things like mp3 coding and
decoding MPEG speech recognition and
image processing those are common places
where this kind of library would be used
and this particular service is only
available in Mac OS 10 it's important to
note that from this library on a 500
megahertz g4 this is where we get some
of the really amazing performance
numbers that are possible because of the
g4 is 2.2 gigaflops
performance again this kind of stuff is
used in in things like iTunes and stuff
like that
there's also a number of other vector
libraries that are available math the
math lib which is basically basic math
library that's been altivec tuned vector
operations and basic ops which is kind
of an extension to the basic instruction
set if you will for the velocity engine
and bignum which is a sports so
basically large you know 1024 bit
multiplies that kind of thing the other
aspect of the g4 worth noting and how it
plays with with Mac OS 10 and the core
OS is in multi processors that Mac OS 10
inherently supports and some true SMP
and this is really differentiated from
Mac OS 9 whereas if you have an
application and you happen to have a
name you know MP box that unless you
have specifically written your
application to use the MP API you won't
actually benefit
Blainey from any significant an MP
system on Mac OS 10 that's very
different in that the system itself
takes advantage of the processors that
are available and scales automatically
each thread in the system is capable of
running on a separate processor again
mock at the very lowest levels of the
system is where the part of the system
responsible for handling the scheduling
and processor management and will
automatically make sure that each thread
on the system runs on whatever processor
resources are available it means if you
have a single application with multiple
threads each of those threads might
actually run on a different processor
and that's really important to note
because that introduced really tends to
be a great way of finding situations
where you may not necessarily be doing
your multi-threading in the best
possible way and doing the locking and
things that need to occur
I encourage if you have a multi-threaded
application that you find a way to test
and run it on an MP system if you're
using
tend to want to be able to take
advantage of that there's a number of
ways through the different application
environments to take advantage of it
certainly the MP API is four-carbon and
interesting enough classic as well as
anise thread for cocoa and the P threads
package for BSD applications so there's
a number of abstractions that you can
use to be able to
take advantage of multiple threads it's
important also to note kind of how the
system itself deals with MP on the
system again mark is you know design
very much from the beginning to be take
advantage of the multiple processors and
to be able to deal with things like
threads UNIX is was is not quite as
evolved in the same way it's from its
heritage is actually designed to to run
more on uniprocessor based systems we've
done a bit quite a bit of work to be
able to take advantage of MP on Mac OS
10 and we'll be doing a lot more as time
progresses but it's important that if
you're actually doing work on Mac OS 10
that you know that the BSD environment
and how it manages itself across
multiple processors is a key part of how
the system is designed I encourage you
to go to the session later this week
where there's more discussion on this
particular issue particularly if you're
doing file systems and networking you'll
find that there's this concept of
funnels that are involved for managing
what parts of the system actually are
involved and run on which processors
since we're talking about things in the
kernel there's this concept that I've
used number time now of kernel
extensions and basically these are
modules of code if you will code
fragments that are allow if you will a
plug-in model for the kernel allows
things like drivers and network kernel
extensions and in file system plugins to
actually be load and and managed as part
of the as part of Mac os10 really
important thing to note that a Mac os10
kernel extension is not the same as a
mac os9 extension we have very explicit
usage models for these particular kinds
of extensions if you're doing a driver
or you need to be able to create a new
file system this is the place that would
you use a Mac os10 extension many things
that were used that were are extensions
in a Mac OS 9 are not appropriate to be
done as a kernel extension you really
should only do them if you have to it's
basically is the plug-in model for the
kernel and we try and keep the structure
of these things much like an application
bundle in the sense that we have bundles
that have property lists and the
binaries for the actual thing that gets
loaded and there's tools and stuff for
loading and unloading these things in
the system
as well as some common api's and things
that are supported for this the other
API is however beyond the basic loading
and initialize and finalize capabilities
are all actual api's that are very
domain-specific that's why I said this
is really not a general mechanism for
extending Mac OS 10 it's really only
used for things like drivers and faucets
and plugins and networking services when
you're developing inside the kernel
should you be one of these people
writing external a kernel extension it's
really important to note that the rules
change a bit one you tend to have to
deal with things through a to system
debugging environment to machine
debugging that the language features
that you have available are much much
more limited we don't support Objective
C in the kernel that C++ that exists
there is actually a very strict subset
of the full C++ standard so they're
things like multiple inheritance and
exceptions and things like that are not
the kind of thing that we really
encourage or in some cases even support
obviously and when you're down at the
kernel level there is no direct user
interaction some cases you're at
interrupt level you certainly aren't
going to be able to talk to the user at
that point you don't have access to
things like aqua there are interesting
logging tools and other things that you
can use so that you can get information
to you while you're writing your
applications but it's definitely not a
user interaction solution resources in
the kernel are considerably more costly
here when you're dealing with an
application if it allocates memory even
if you run out of the available physical
memory the OS itself will look you know
through VM and through backing store
will do its best to try and support your
application if you're in the kernel and
you use lots of memory you're stealing
away actual physical RAM from the rest
of the system and it's very expensive so
if you're writing in this particular
space you should always use the least
amount that you possibly can
failures in the kernel are fatal
obviously one of the reasons we want to
discourage development of things like
kernel extensions is this is one area
where you can introduce instability in
the system if you have a driver that
goes wrong while you can just as easily
crash the system that way as you can
under Mac OS 9 so that's why there are
very explicit cases where we support an
extension of the kernel are we in
discourage all other
of extensions from this level moving on
to the open-source aspect of darwin i
think it's important to note again that
virtually all of the components that are
available in the core OS are open-source
a couple of very rare extension
exceptions things like some some of the
drivers that are licensed from third
parties things like that and that we
actually very actively are involved in
actually feeding code back to our
upstream providers so it's not a we've
just taken code from the community and
have integrated in and they're just are
running with it we actually work for
things like BSD and Apache and other
things that if we find bug fixes or the
things that we actually feed those fixes
back to the BSD community their apache
community or other providers we are
actually very actively trying to make
sure that there's a community
involvement here an open-source excuse
me and that it's important that this
community is very cooperative for
example things like security fixes that
come in some of the patches that were
part of some recent software updates
were actually fixes that came in from
the external community that we rolled
into the software updates or even
platform fixes we ended up getting a
from one of the bug reports that was
submitted to our support group actually
included a bug fix right right in as
part of the submission so it's really
important to note from a perspective of
open source this is something we take
very seriously it's not an experiment
it's very much a part of how we do
business for in the core OS and that our
developers are involved in the mailing
list in the community to to actively
make this a two-way two-way process and
that you as developers can actively
participate in this process it's
important to note again that our
repositories and things like that
actually many of them are outside
apple's firewall that we actually have
people outside of Apple that do not are
not under a payroll don't work for us to
actually have commit access to these
repositories that people can actually
are involved in the process of
developing and involving the darwin and
darwin codebase
excuse me there's a couple of sessions
later this week that I encourage you to
participate in one is the open source at
Apple on Wednesday talk a bit more about
Darwin from an open source perspective
and other open source projects as well
as a birds of a feather session tomorrow
evening and finally I'd like to give you
a handful of things to think about as
you kind of go from this point hopefully
off to all these other sessions one is
that you know it's our intention even
though we have all this power on the
under the hood that the user experience
of Unix is something that we really want
to hide the goal is not to make Emacs
the the ideal user interface right from
our perspective this is really an
optional environment for power users and
developers this should not be the
typical behavior of Mac OS 10 but this
is something that you as developers
should take advantage of but always keep
in mind you know give you some examples
of how we've actually evolved the system
in this way is that the concept of root
under Mac OS 10 is something that we've
really tried to move away from it's
possible to actually set up your system
so that you have a root user but you
have to go to some work and you have to
really know what you're doing in advance
but this is part of the model that we've
tried to apply as Mac OS 10 as to how to
provide as much power as a UNIX based
environment can possibly have without
having to be create you end users that
have to be you know knowledgeable units
administrators be careful that you don't
repeat bad habits under Mac OS 10
there's a lot of different ways to
approach things and you can probably
solve virtually any problem you have
with the mechanisms that are there but
think about them before you do it don't
just jump into the only way you can
solve a problem is by using shared
memory because you end up recreating the
same set of problems of stability that
have existed in the past there are other
ways to be able to communicate between
multiple applications than running on
the system shared memory for example
should be avoided and it's really
important to look at the system you know
to have a very system's view when you're
writing your apps tools like what we
showed with top and FSU's and stuff
allows you to get a sense of how your
application actually impacts the rest of
the system there's a lot of other
threads and
processes and things that are running on
the computer and in order to be able to
make sure that the system behaves well
with lots of different things occurring
you need to take a few minutes and
actually looking to see how your
application affects the system always
use the highest level of abstraction
possible again they usually are very
appropriate services to use either the
carbon level or the Coco level or
whatever and that you should only get
involved in using bsd services or mark
level api services if that's the only
way to solve a given problem don't just
jump right in and use vm allocate for
example for doing memory management
there usually are a lot of other things
in the abstraction layers that exist
that are helping to manage those
resources and make sure that they you
know are handled appropriately and
remember that open source is an
incredibly powerful tool this is
something that we're doing which we
think is pretty novel in terms of you
know a adopting open source in a very
big way as part of the major you know
aspect of the product that we sell again
this is not a little interesting side
project you know Steve put it very
clearly we're betting Apple very much on
the success of Mac OS 10 and open-source
is a very key part of the lower portion
of that system get involved in it help
participate you know there's a lot of
email lists and ability to actually
contribute code and other things help
make the system better and also remember
that there's an incredible amount of
power that's exists here one of the big
changes between Mac OS 9 and 10 is this
introduction of a you know modern
operating system well this is that
modern operating system the ability to
do threading and you know being able to
do lots of things in power and parallel
and things in the background important
leverage UNIX applications and multiple
application environments and so this is
all incredibly powerful aspect of Mac OS
10 and something that you want to think
about because I think it will enable you
to take that next big step and lastly I
want to remind people that there is a
feedback form we'd love to hear your
comments on Darwin and things the ways
that you think we can make it better so
encourage you to come to the if you're
here at the last part of the session
five o'clock on Friday we'll be here
you