WWDC2000 Session 106

Transcript

Kind: captions
Language: en
please welcome Jim McGee
wait I have to the official cracking the
water bottle there all right I want to
thank you all for coming this early in
the morning to talk about something so
wildly exciting against the colonel yep
okay
what are we going to talk about in this
session well mostly we're going to talk
about the mock Colonel what the basics
of the Mont Colonel are what the basic
abstractions are when you want a program
to those when you don't but most
importantly how the mach kernel and its
environment affects the rest of your
development for the majority of you you
may be bringing over a carbon
application and for a lot of those
applications you fix a couple of things
that the carbon dater tells you are
wrong with your application and you're
done
but you're not because things don't
behave exactly the same on 10 as they do
1 9 and a lot of those differences are
based on what happens in the mach kernel
and so we want to give you a little
background about what's happening under
the covers so you can understand your
app ok you're probably kind of tired of
seeing this picture but it's basically
the layout of all applications in in Mac
OS 10 your application is going to fall
into one of the basic categories of a
classic app a carbon app or a cocoa app
at the bottom of that is the Darwin
layer and inside the Darwin layer is
where the mach kernel resides the mach
kernel is the foundation of the OS right
it provides the basic abstractions for
getting the system running things that
your tasks in your threads the CPU and
memory abstractions of the system
macht is intended to provide a rich set
of semantics for those kinds of services
one of the things you'll find is that
Mach was intended to develop lots of
different operating systems on top it
was originally designed as a foundation
for UNIX operating systems but then
other operating systems came along
people provided dos boxes running on top
of mock provided specialized operating
systems or embedded environments on top
of mock and over the years the mock
semantics have become richer and richer
and a little bit more formal each time
and that ends up giving you a set of
semantics that that let us do some
things in Mac OS 10 that you wouldn't
otherwise see Mach is intends to be
policy neutral it leaves the policy
decisions to higher layers of software
in our situation a lot of those policy
decisions end up in being implemented in
the BSD layer some bubble up in the
higher layer so classic has some of its
own policy decisions they use mock
services but the mock services try to be
policy neutral they give you a set of
mechanisms but not the policy so what
that's what mock is what mock isn't is
an operating system right mock is just
the foundation for building operating
systems on top mock does not provide I
out
we have i/o kit for that and it provides
those set of abstractions mock does not
provide any networking although people
have taken mock and networked some of
the mock services mock itself does not
provide any network services mock
doesn't do file systems as you saw
yesterday if you were in the file system
session the file systems are implemented
inside the BSD layer mock doesn't do any
security policies either again mock
tries to provide mechanisms it provides
lots and lots of security mechanisms
and some really fundamental ones but it
makes no decisions about how they get
applied in our situation a lot of those
policy decisions again fall to the BSD
layer so you're going to see that we use
the BSD style of user ID and permissions
BSD then restricts your access to
certain mock resources based on those
attributes but the mock services really
are all based on whether you have access
to those services and don't care who
what your user ID is or what permissions
you have or who you logged in as we
basically have a set of services a set
of access points and if you're granted
access to those services by the policy
maker the decision maker in this case
BSD and you get to have access otherwise
not mock doesn't really provide
application api's either
while you may dip down sometimes into
the mock api's in some of your
applications you're not going to be
programming to to mock itself mock is
intended to be the layer that the
operating system pieces converse with
all right and all of those abstractions
that we just talked about are actually
provided in higher levels of Mac OS 10
or Darwin okay so why are we doing this
mock thing it's the number one question
people ask when in the Darwin mailing
lists or otherwise why mock well we do
have a complex system in Mac OS 10 we
have lots of environments to program too
we have the classic environment the
carbon environment the cocoa environment
we have BSD we have AI okay we have Java
others are coming that that people may
be working on in here people are doing
dos emulators or doing other
environments for this system each of
these has a set of semantics and
policies that they implemented and they
control and they cannot change its
legacy right macht provides a set of
abstractions that sits below a common
layer the
it's below those that allows you to
implement all of those it's been tuned
over the years to give a fairly good
match to each of those environments
there are some environments which people
have tried to put under Mach or on top
of Mach and haven't had such a good
match but for all of the ones that that
traditional operating systems and the
new Java VM and those things there's
been a lot of work to tune those layers
between Mach and those environments to
provide a good match a good impedance
match between them but Mach does more
than that it makes sure that those
services are consistent across each
other so that a carbon application using
memory in a certain way doesn't doesn't
destroy a classic application doing the
same thing or a bsd application doing
the same thing
another reason for Mach again 15 years
now Mach is nothing new to the research
in the research world it's new to Mac OS
10 but it's to the Mac OS environment
but it's been around for 15 years now
lots and lots and lots and lots and lots
of research obviously the original
research done at Carnegie Mellon was
intended to prove that you could build
an operating system by defining a layer
underneath
that provided the basic services and
layering the operating system on top
more work was done at the University of
Utah kind of jumping forward in time
they took the mock work that had been
done before them then tried to make it
more modular and embed it and add a
bunch of real-time services to it and
then a lot of research was done at the
open group / open Software Foundation
depending upon your time frame their
research institutes both in Cambridge
and in Grenoble France lots and lots of
real-time work done for government
contracts and otherwise and
as some of you may know osf focused a
lot of their research on cluster work
building up complex systems out of
modular boxes a single system image if
you will across many many boxes and
there was a lot more research in MOC MOC
was one of the first publicly available
source codes for something other than
you unix most people had been using DSD
in the research community forever but if
you've tried to do research on BSD you
realize that there's a lot of nuts and a
lot of things tied together in DSD that
makes it difficult to pick a piece out
and do your research and replace it with
a different piece MOC is really good at
allowing pieces of the system
higher-level systems to be picked apart
and replaced so a lot of that research
occurred on MOC and most of that
research is available to us and to
others and we've been picking pieces
select pieces of it to include in Mac
os10 okay
and another obvious reason if for some
of you that don't know avi - Vania dr.
avi - Vania and our senior VP of
software was one of the original authors
of MOC and obviously he has a fondness
for it but that's not a misplaced
fondness in my opinion okay so which MOC
if we're going to choose MOC because it
allows us to build complex systems multi
environmental systems on a single box
that maintain consistency and yet
provide good performance to each of the
environments there are several mocks to
choose from lots of research produces
lots of mocks okay the original MOC MOC
2x most of you probably know the 2.5
version of MOC it's the one that
originally got out from CMU and found
its way around it was that the original
work at CMU to decide to prove that you
could build a layered operating system
that there we're common services
underneath
those that a traditional operating
system provides and that you could
formalize those but mock to five was
monolithic in nature right it was a
layer underneath a BSD inside of BSD if
you will
they picked apart some of the pieces but
built it as a single unit
areas where that kind of showed through
the most is the virtual memory system
they provided an abstraction for virtual
memory inside of mock but the lower
levels of that virtual memory system had
hard-coded dependencies on what BSD did
for file systems and the like it also
had hard-coded dependencies to a lot of
the other parts of BSD for signal
handling and the like that was kind of
intertwined with mock and i/o it had
direct dependencies on the SDIO style
that kernel was also not really
available in a form useful for SMP and
when they did the work their main goal
was to see if we could take a UNIX
operating system with UNIX current real
time constraints and see if we can
implement it in a layered fashion that
did not try to extend it in any
real-time fashion next in the horizon
was Mach 3 oh and again that was started
at CMU to say can we take this a little
bit further can we formalize the rest of
the interfaces so that we have no
dependence between the Mach portion of
the kernel and the rest of the kernel so
they formalized all those interfaces
which had been into five hard-coded
dependencies so they ended up with a
more modular architecture as a result
and there was a lot of work that
occurred once this basic work happened
there was a lot of research that came
along in the areas of real time and in
the areas of MP and most of that work
got folded back in some of it didn't
there was a bunch of work it at CMU on
real time if any of you familiar with
dr. to Cudas work on real time a lot of
that
did not get directly into the standard
mach3 Oh base but found its way in
overtime through various other projects
and Mach 3 Oh was where MP became a very
formal part of the system and of course
that's quite interesting to us yeah Mac
os10 the server product and the Darwin 0
dot variants of the operating system
were based on that original Mach 2 5
work it included if you any of you have
looked at the darwin source a lot of the
features the niceties at the user level
from the mach 3 o work a lot of
democracy interfaces and the like had
been brought forward by apple and others
into that kernel mostly by next but we
picked a lot of that up at Apple and
that was in the mock the server kernel
form up for Mac OS 10 and you got excuse
me I'm a lot of times I'll say Mac OS 10
a bias for Mac OS 10 and Darwin 1 oh it
was all rebase tree hosted on a Mach 3 o
variant right and one of the things that
you'll find is since there was so much
work research work done in the world of
mock around the 300 base is that there
are a lot of Mach 3 O's I've never seen
a Mach 3 anything other than oh but huge
numbers of those in different variants
so people tended to take a Mach 3 Oh
like the one done at open group or open
Software Foundation and they've tagged
another acronym to it so arm op 3 o was
based on work done at OSF the 7.3 MK 7.3
base for those of you who have any
history with MUC that base was the base
for a lot of their distributed work
their cluster work they're at a d1
system in OSF one systems if anyone's
familiar with those basically a lot of
distribution was
they were going after but we had there
was a bunch of work going on at OSF also
to fold back in some of the research
that have been happening at other places
after seeing you stopped working on Mach
3 Oh a lot of the Mach work drifted over
to University of Utah and they were
working on their Mach 4 Oh base which
actually wasn't a lot of people think
for oh maybe more advanced than three
dot but it's not it was just had
different goals and they chose a new
name because 300 had been used so much
so a lot of that work from from that
environment had been brought back by OSF
and integrated into their Colonels but
not the ones that they made publicly
available the ones that were used
internally and given to their partners
well when we started our work we had
that code brought back and merged with
the work we had done with our McLin ik's
mach 3 o colonel
and that's became our base so in essence
we actually pulled a lot of that private
code that had been done at OSF and only
available to us at partners and through
our licensing of that have made it
available in to the public source again
where it had not been available before
and besides the work that we had done on
my clinics at Apple we have done lots
and lots and lots of things to it since
then and you'll see some of that work in
the further slides ok so a lot of people
say but that Mach 3o thing it's
inefficient I've heard it
everyone tells me it's inefficient
because the general goal of Mach 300 was
in order to prove that you could have
the mach kernel providing a rich set of
abstractions we wanted to be able to
prove that you could run the mach kernel
separate from the other environments in
the system so BSD ran as a user level
task in that environment that was a
really good exercise it proved that the
abstractions that were being created
were a the right abstractions and B
could be represented efficiently in a
nice segregated manner between the two
components
but the main goal of that work was to
was to show that you could have a bsd
environment that that could fail and the
mock environment would continue going
now in some real-time environments
that's a really neat thing but in the
Mac OS 10 and Darwin environment BSD is
critical to everything we do every
application is a BSD application you
can't keep the system running with just
the mach kernel so having them separated
is a nice exercise in research but it's
not useful to us and it does introduce
some penalties some people say a huge
penalty some people say it's all it's
only a minor penalty but it doesn't
matter it's a penalty to have them
separate so we've combined them together
again but we haven't done it the way
Mach 2/5 did it we don't take those
services and just wedge them together we
leverage those nice clean mach3
abstractions and we call those same
abstractions now from a BSD layer inside
the current over an i/o kit layer inside
the kernel right rather than i/o kit or
BSD running at user space but we use
those same abstractions so we haven't
broken the modularity gains by
eliminating the IPC and the
communication the cross task
communication okay so we access them
directly and not through IPC okay so how
do you guys access all these powerful
abstractions that allow all of these
funny things all these wonderful things
to happen in Mac OS 10 well for most
part you don't most of the Mac
abstractions are used directly by lower
layers of software than the application
for so for the most part you should be
programming two-carbon - classic - cocoa
and sticking it those layers even bsd or
java and using those if you need to get
a little bit lower in the system a
little more detail and a little more
control you're typically going to drop
one layer
from that and maybe you're going to
access some corefoundation things
directly some bundle stuff or access
some of the CF run loop stuff in the
system right but you're going to
possibly stop at the CF level as well
the core foundation level or you're
going to use i/o kit to do things or
you're going to use the BSD file system
extensions or you're going to use the
BSD networking extensions but typically
you don't you don't end up having to
program to mock directly okay if you do
program to mock directly one thing you
got to realize is it's not portable
if you drop if you have a carbon
application that tries to reach under
the covers and access mock that
obviously that carbon application is not
going to go directly back to 9 there are
tricks you can play with Gestalt sand
trying to load extensions and load
bundles to do some of that work and if
you really need to you can do it but
again it won't be portable and you've
got to go out of your way some of these
interfaces are not final at the Mach
level as we build up the Mac OS 10
environment with all the stacks coming
up now - they're pretty much their final
form right we're taking a look at the
abstractions that moc provides and we're
trying to find more efficient paths and
and better matches to the semantics of
all those environments so we're tweaking
them a bit as we go along you're going
to see some of that work happen between
now and and when the products final and
so some of these interfaces are going to
change on you but again the majority of
them are correct and and final or
reasonably final there aren't going to
be that many changes okay so then why
should I care about mock right you care
about mock because you need to
understand how your application works
you need to understand how the
frameworks are built mostly so that when
you go to debug your application and you
you see that all of your threads are
blocked in a certain location what does
that mean why would that happen right
and you need to understand how your
application in those four
worx might interact and a lot of the
things that you will see end up being
mock things okay so what are these
wonderful abstractions that lets us
build all these different environments
tasks and threads are fairly common
fairly well understood abstractions in
mock a task owns all resources for an
application so all resources are tasks
wide every thread inside a task then you
have threads threads are the unit of
execution the things that the operating
system schedules and virtual memory
provides protection for loads and stores
to your application then once we build
up all of those environments - that
gives you all the isolation that Mac
os10 is supposed to give you - a carbon
application let's say you've now got a
nice isolated application it's got its
own private virtual address space it's
got its own set of resources they're
protected in to the task itself and
you've got another carbon application on
the other side now for the first time
you've got to have formal ways of
communicating between them right you
used to be able to cheat and poke this
location in memory and check that when
you're coming around your next loop and
the carbon application if you need to -
to communicate right but now they're
isolated so you need some formal way for
them to communicate and so there's a
formal task to task communication
mechanism in mock okay so at a mock task
as I said before owns all resources it
sets up an environment for threads to
run right all your threads have equal
access to all of the test resources
there are other resources available so
virtual memory and a port namespace and
we'll get into what the port namespaces
and a little bit virtual memory I think
most of you understand it's a protected
address space for each application each
task but there are other resources
associated with a task we won't get in
them too much but there's task wide
exception handlers and a bootstrap
server reference for a task that's
basically how a task understands what
kind of task
it is when Mach gets a task created all
my tasks kind of look alike so how do
you distinguish a job a task from a
carbon task well a lot of that happens
through the bootstrap server the task
very first thing it does when it
executes is go converse with its
bootstrap server and that gives it some
of its original primordial references to
things and then it bootstraps itself up
from that and all of a sudden becomes
one of the environments or the other so
when you look in Mac OS 10 you're going
to have your application maybe it's a
carbon application well each carbon
application is actually a BSD
application it's a BSD process and a BSD
process has two basic sets of resources
with it the mock tasks part and file
descriptors there's other bsd resources
shared memory id space and the like but
most of the things in bsd processes are
file descriptors or tasks you end up
with that environment but inside a mock
task you have multiple threads executing
in a single task sharing a virtual
memory space and a port namespace the
thread the threads the unit of execution
in mock they're preemptively scheduled
against each other this is how Mac OS 10
gives you pre-emptive scheduling mock
does all of the scheduling work for
those threads Carbon and classic don't
have to do their own scheduling work
again they have equal access to the
system resources an important thing to
note at the mock level a thread is
nothing more than a register state and
some scheduling attributes it there's no
such thing as a stack to a to a thread
at the mock level we don't care whether
there's a stack there or not you give us
a register state to start the thing we
started running will preemptively
schedule it take it off the processor
restarted that state something else is
responsible for doing those other layers
right so this is what a thread ends up
looking like each application in mock is
going to have threads multiple threads
each in Mac OS 10
each of those threads actually is a
POSIX thread and the POSIX thread is the
part that's responsible and does the
work to allocate the stacks and set up
things that you would expect from a
higher level concept of a thread and
even inside of that in a carbon
application when you have preemptively
scheduled threads or sorry cooperatively
scheduled threads in mock in carbon
those are P threads which are mock
threads and Carbon does some work to
make sure that mock schedules them in a
way that they look like they're
cooperative rather than pre-emptive one
of the things you also have to be
careful when you're developing your
applications you may create a simple
carbon application today single threaded
carbon application and you go and you
look in the debugger and lo and behold
there are two or three threads in there
where'd those other threads come from
well they're actually created by the
frameworks the frameworks sometimes have
to create threads to bridge the gap
between the semantics of the services
below them and what the carbon
application may expect up above so
you're going to end up with some threads
that you're not aware of and you're
gonna have to worry about those when
you're doing your debugging so beware
they come a lot of the work we're doing
for semantic matching is trying to
eliminate some of those worker threads
and all of our tools are threat aware so
you're gonna see that our debuggers will
show you dumps and trace backs of all
the threads our sampler tool will show
you what your thread is doing but it'll
also at the same time show you what the
worker threads they're provided by some
of the frameworks they're doing and so
you end up with a picture like this that
you have mock threads mock threads are
wrapped at user level by P threads to
provide stacks and the like inside of a
carbon application you may have those
thread manager threads like I talked
about that are cooperatively scheduled
against each other through the
assistance of carbon but you then have
MP threads which are free to run
and execute and mock is the only one who
schedules those so obviously if you're
worried about getting true MP
performance and through true parallelism
the carbon guys are trying to push
people a little bit towards the MP
threads if you're doing a cocoa
application all of the cocoa threads are
POSIX threads there's no cooperatively
scheduled notion in those so you don't
have to worry about the cooperatively
scheduled part what's new for threads in
mock in Mac OS 10 versus server and
Darwin 1o versus previous Darwin's well
P threads replaces see threads see
threads was something that was
explicitly for mock from way back when P
threads are a more standard version of a
lot of the same services so we've
switched to P threads we also have
scheduling frameworks in the mock 300 or
in the Darwin 100 and mock 3 o code that
we've got scheduling frameworks allow
for different scheduling policies for
threads rather than just the standard
timeshare scheduling policy that's
trying to getting fair access to
everything there's also some fixed
priority real time scheduling policies
in there and you can select those from
user state and one of the big things
obviously on the PowerPC processor is g4
support for velocity engine is now in
what's coming soon for threads a rich
additional real-time work additional
policies isochronous policies that some
people are looking for are being
investigated no promises but we're
looking into those there's been a lot of
research in mock again with being able
to leverage work that other people have
done real-time priority inversion
avoidance a lot of the work that came
from the University of Utah was about
trying to avoid those situations for
real-time applications where you're
dependent upon a resource and that
resource is currently consumed by some
other thread in the system one that you
don't really care about but you've gotta
wait for it and that thread may be so
low a priority that it's not actually
making any progress to get rid of its
hold on that resource and so the
system-wide approach of promoting those
threads
in order so that they could let go of
their resources and then let the high
priority thread continue is something
that's being extended it's already in
the codebase that you're looking at now
in Darwin one oh but it's being extended
and deployed system-wide in kernel
preemption it's something that was built
in to the mach 3o code that we've got
what's currently disabled but it's there
and and enabled and it will be or enable
and it will be turned on in the near
future okay the other service another
service in in mock is virtual memory
again it provides a protected address
space for each application it provides a
flexible way to construct those address
spaces and it allows for controlled
sharing don't go hog wild because mock
lets you do sharing jumping down there
and doing wild sharing we do sharing in
a controlled way in Mac OS 10 one of the
key features of virtual memory in mock
is copy-on-write behavior if you look at
each carbon application you can almost
consider a carbon application as a
unique instance of the Mac OS 10
environment right a lot of the
frameworks that used to get loaded once
in Mac OS 10 or sorry in Mac OS 9 right
get loaded into each application as it
starts right if each application had to
have a unique copy of those frameworks
it would take forever to load it would
be huge amounts of memory that would be
required so mock provides a way to
flexibly share those spaces and to
flexibly provide copy-on-write semantics
to those so there unless you modify a
page you get to share a common page but
as soon as you modify it only that one
page gets to be a unique copy for you
and that's done lazily as you touch it
rather than upfront so when you look at
the virtual memory system each task ends
up having its own virtual memory space
and inside that virtual memory space is
little regions of mapped memory objects
well what do you map
well for most of you you make calls to
frameworks and they do this magically
for you but let's take an example where
you map a file descriptor under a BSD M
map call right that file gets converted
to a memory object an abstract memory
object inside the kernel won't go too
much into that but there's a general
concept called an abstract memory object
when you map that into an address space
the kernel creates an object called a
virtual memory object it's the manager
of the cache of that data associated
with that file so we don't bring the
whole file in we don't even know what
the file is but we know that there's
some object that we have to cache data
for and so we create a virtual memory
object for it and as you touch pages or
access locations in your address space
we bring in some of those pages and
cache them and then make them available
to your application so here's an example
of that now I've come along and touched
in that second page I've touched a
location and when I do it that causes a
fault in the operating system all right
at the lowest level that gets translated
into we look up the address space we
look up the mapping the region and
describe figure out which object and
which range of that object and with
which permissions you have access and we
go and check the virtual memory object
Paige we just map it in your address
space and everything's fine there's only
one version of that data in the cache if
you're in the file system session
yesterday you heard some of the ways
that we make sure that not only are we
maintaining at the VM level but we also
maintain consistency at the file level
but sharing isn't the only thing we do
as I said before you've got a situation
where you don't want to have to bring in
whole copies of things that you want
unique copies of so the data section for
each file or framework that you have
loaded we don't want to bring that all
in at that process initialization or a
task initialization time so we want to
do copy-on-write
right we want to access the original
object as much as possible and only copy
bring in unique copies of pages for
those that we modify so here's the
situation of how that works when you map
something copy-on-write rather than
having a direct reference to the object
there's a new object that gets created
and stuck in the middle all right and
when you take a write fault on one of
the pages right we go to that virtual
copy object it instead of going directly
to its abstract memory manager it first
checks the thing that it's a virtual
copy of and says do you have that page
and in this case it's really that second
page that we just talked about and of
course he does so he sends us a copy of
that page back and we cache only that
one copy page and we make that available
so as you run a bunch of the tools in
the system you're going to see things
like private pages alias pages virtual
pages right the private pages are those
things which we have pulled our unique
copies of at the lower level below that
abstract memory manager what happens
there well as I said before typically
each one of those things is a file that
got mapped and so those go through the
Vinodh pager and they access the bsd
filesystem directly for each file type
but there's also those
they're kinds of objects which aren't
files those thing when you allocate
memory or you just did one of those copy
operations we end up creating some
magical data for you a new object that
you didn't know about nobody really has
an explicit handle to write all of those
objects are actually managed by a pager
in the system called the default pager
then what he does is takes all of those
objects and the since they are very
sparse objects they have a one-page
touched maybe at the beginning of the
object and another page touched way down
inside the object you know it at offset
10,000 we don't want to create unique
files for each one of those objects
you've created there are thousands of
these in a running system so he creates
the default pager creates a set of swap
files right and maps those data that
data from those sparse objects down into
a swap file and writes that through the
file system to disk or across the
network or wherever your file systems
happen to be and there's a demon in the
system that gets queries from you know
notifications from the default pager if
one swap file is filling up we send a
notification up to that demon and he
allocates another swap file and and
creates one and we start swapping into
that one really neat feature that's new
to Mac OS 10 for those who are used to
Mac OS 10 server is now swap files can
go away and they do automatically so if
you burst up in your system and access
huge amounts of written data anonymous
data and then all of a sudden deallocate
at all the as the data gets freed space
gets freed up in the swap files and then
the swap files are free to coalesce and
you can actually take a swap file
offline so if you want to put a swap
file on one volume you can even put it
on removable media start swapping to it
then when you want to unmount that media
you can say ok unmount that swap file
and the default page removes all the
data from that swap file into the other
swap files maybe making space available
somewhere else for that data to move
and then freeze up that volume that's a
neat unique feature that we've gotten in
the current kernel okay how does it show
all this funny stuff to you well what
you're going to see when you look at
your applications is this long range of
memory regions in each virtual memory
space of an application you just see a
little bit that get created here for
mapping this framework in a little bit
that was here for this framework right
there going to be sparse they're
typically going to be copy-on-write for
things like frameworks most of them are
not you never touch them like all your
text is mapped to copy-on-write well I
don't write text why should I map text
copy-on-write
well debuggers do when you run a
debugger on your application you would
hate to sticking a breakpoint in one
application at a particular location in
a framework would cause every
application to take that breakpoint
all right so by doing everything
copy-on-write
you have the ability to come along and
modify just one application you're going
to see lots of tools hopefully some of
you will be able to go to the tool
session later today or the debugging
session first thing tomorrow morning
you're gonna see a lot of tools around
managing your virtual memory space of
your application malloc debug is
probably going to be the one that that
you're going to go to first
well maybe second you're probably going
to go to top first top is going to tell
you how much memory you've got how many
regions you've got in your application
and if you see that starting to leak
you're gonna turn around and go to
malloc debug and malloc debug is gonna
help you walk through each of your
applications in these areas there's
another neat program that's not it's a
it's a demo program a helper program
right now it's not finalized but I
really like it not necessarily as a
debugging tool but as a tool to
understand how an address space or an
application is put together and that's
the the page watched app page watches in
debug developer tools I think or debug
developer applications sorry local
developer applications on your CD and
when you bring it up it gives you a V
image of what your address space looks
like a little square for each page in
the system in your application and it
tells you whether that page is present
or not and you can set it to scan at
certain intervals right and you can see
pages faulting in and out you can also
click on another view now that's the VM
summary review there's another view that
lets you see each of the individual
things that got mapped into your
application each library and which pages
are being used and how many pages are
being used out of each one of those it's
really neat just to get a sense of how
an applications put together
okay what's new in Darwin 1l we've got
the external memory manager interface
that was something that wasn't in Mac OS
10 server now we don't currently have
any need for user level pagers right
this is actually an exportable interface
between the kernels so we can call out
to a pager at user space and a lot of
people will go oh really neat I can go
ahead and build myself my own private
pager and I can do really neat things
with that well in most cases you don't
want to do that in most cases what you
would rather do is build a file system
layer right because you don't want
something that just behaves that way
let's say you wanted to do an encryption
pager oh I could do encryption on my
data well but you want reads and writes
to see that data encrypted or decrypted
appropriately as well right if you do it
as a filesystem extension the Vinodh
pager I showed you earlier allows you to
go ahead and access those things for
free it becomes a pager for free right
if you do it just as a pager well you've
got no way to access it the other way so
there's very limited use for pagers
through that formal interface but it
gives a nice separation between those
things even inside the kernel right
another thing that's in Darwin 100 is
the ability rather than just having an
address base be a linear list of memory
regions you may have
a bunch of things in common between a
bunch of applications and so rather than
mapping each one into each application
you may want to just create one memory
address space that has all of those
files mapped into it right and then
rather than mapping again each
individual file into each address space
you can take that one address space and
recursively map it into each of the
other address spaces so as data appears
in that single address space as things
get mapped in and out they magically
appear in the other address spaces as
well it's a really neat feature most
users won't have direct need for it but
coming soon will actually if you look at
the bottom one or the the shared memory
part should region part we're going to
use that to employ the system frameworks
throughout the system they're all going
to be mapped into this common thing and
then that common thing is mapped into
each address space it provides a shared
level of efficiency that we don't have
today other things coming in virtual
memory if you were in the file system
session yesterday they talked about how
the file system part of the system is
already enabled for 64-bit files but
that the VM subsystem keeps them from
being able to be mapped and one of the
things the first things we're doing is
adding the 64 bit object size support
and if people are familiar with the
PowerPC roadmap there's 64-bit PowerPC
processors coming right down the pike
and we're looking at ways to to add
64-bit address support now this is a
little bit different than traditional
64-bit systems because in the PowerPC
roadmap you can mix 64-bit applications
and 32-bit applications on a single
running system so we need to be able to
flexibly support both and that kind of
work is ongoing and there's also some
changes in that EMM I layer to add some
efficiencies there okay so we we now
have a task and it's isolated and has
one or more threads in it and it has its
virtual space and we may have two of
those tasks
all right but how do they talk to each
other now that they're isolated we got
to let them talk and the way they talk
is through IPC IPC in Mach has three
basic abstractions
that's ports which are the endpoints of
communication between tasks they're
highly restricted each you cannot send a
message to a port unless somebody hands
you a right to send a message to that
port and through that restriction we
actually get the basics of how the
security model in in the system works
right that you're only granted access to
those things which the policymakers
decide that you're allowed have access
to and by not having access to the task
port for some other tasks some other
process like a system process you can't
do things to that process a port set is
a collection of ports so you can have
both sins and receive rights to ports
you can implement something so someone
can send you a message right and reports
that allows you to collect those into a
single spot so you can have a single
thread wait and receive on any one of
those ports it's kind of like selects
ads or FD sets or file descriptors
except that they exist forever the
kernel maintains them and you can add
things to them so you don't have to
construct them on each call and where
are all these ports maintained there's a
namespace for each task that maintains
the task the port space so here at the
bottom you have this port space right
and you may have a port which gives you
the permission for one task to send a
message to another right and that task
may own a port of its own right that it
owns the receive right for and so it can
construct a message not only that sends
data but you can also send rights to
ports to that other task and it can also
send memory copy-on-write memory to that
other task so all of these systems
inside amok are kind of Rickard
recursively defined they each depend
upon each other and they each take
advantage of each other services so VM
uses IPC to implement itself
in certain areas and IPC uses VM to send
data to other tasks so when you send
that message over to the other side and
the other guy receives that virtual new
virtual memory appears in that other
task and now he sees a virtual copy of
what was sent by the other process right
and you also received that port so you
can now send a message back the other
way and through this mechanism and
building up of these little individual
ports and sending writes and sending
messages all of the interfaces to the
kernel use that mechanism to the mock
part of the kernel to implement their
services so when you say task or thread
create write the first parameter to that
is the task reference well that's really
a port write that gives you the right to
send a message to a task port right that
sends a message to the kernel the kernel
processes that request if you didn't
have a right to that task port you
couldn't create a thread in that task
port in that task but by having a right
to send to that port you have the right
to do that operation that goes into the
kernel he does the work and as a result
he sends you back a right to manipulate
that thread that's how the kernel works
this is how the window server works so
when you converse back and forth between
the window server and applications the
window server gives you rights to access
certain core graphic fundamental pieces
of the system if you don't have the
right to access that you can't so you
can't trance um buddy else's window as I
said by using the protection of mock
ports right you're gonna see that each
application ends up having a pile of
ports typical application will have 50
or so ports to it right
these are reference handles to objects
implemented somewhere else either in the
kernel or inside a Windows server but
they can also be things that you
implement and reference and give
references to other people exception
handlers are one of the common examples
from the mock perspective
when an exception happens in a task
we'll send a message to whoever the
exception handler is for that that task
whoever registered a port with us to
send a message and we'll send a message
that says this exception happened and
that maybe your own task or that maybe
gdb you're going to see that each
application almost every time you look
at your stacks or your trace backs
inside of sampler app you're going to
see that they're all blocked waiting on
IPC operations right at the very highest
level that's because almost everything
in the system is at its fundamental
basis an IPC operation right so when
you're waiting for a semaphore to go off
or your your in your event loop right
for carbon or whatever you're gonna that
thread is gonna see at the very lowest
level you're gonna see that it's in mock
message overwrite trap today you're
gonna see that over and over and over
again and you got to realize that okay
well that's because everything G
generates down to a message eventually
and you're gonna see those things
through top and you're gonna see them
through se usage SE usage is really nice
to see what you're doing in the system
and that should be coming up in the tool
session later okay so how does this show
to me like I just said each of your
applications is going to have a set of
ports for accessing other things but
each of your applications is actually
going to have a set of ports for
delivering events to it right and at the
lowest level they are a collection of
ports they're wrapped up in Port sets so
that a single thread in your application
can receive them typically your
applications going to be using something
like a carbon event queue the Carbon
event queue actually internally has run
loop associated with it and that run
loop has a port set and that thread
eventually when it's waiting on that
event queue you're going to see at the
lowest level that's waiting for a
message to come in on that port set all
right so all of these things that
deliver events to you are typically
created as ports for those that aren't
there are certain things that are in CF
run loop that aren't delivered that way
see a front loop actually creates a port
so that and possibly a thread sitting in
that other box to receive that event and
then pass it on as a message a simple
message says this event happened to one
of those other port sets because that's
where your applications thread is
waiting in that one spot and it just
tells him to come over and look at the
others right and the same thing at the
Carbon event loop model or the carbon
event queue mo things will happen that
aren't delivered by ports but because
your thread is waiting for a message a
port is created to send that the
notification that happened what happened
in Darwin 100 or Mac os10
well there were some IPC changes one of
them allowed that discussion we just
talked about where you may have a CF run
loop in he may have more than one mode
or a event queue so you're in a certain
mode you're in a modal environment where
you only have a few things you want to
wait on before lots of activity had to
happen in order to change the set of
things that you were waiting on we added
a facility that lets you wait lets you
put those events into multiple ports
that simultaneously and then you just
have to switch your set that you're
waiting on that all happens under the
covers but you'll see that if you look
at the code coming soon as you saw that
there are certain things that cannot be
delivered efficiently through the port
mechanism and since that's where your
application is waiting we had to come up
with some other mechanism DUP ping pong
it over there well we can do that more
efficiently and so we're creating
additional channel types other than a
message queue so that we can deliver
those events directly to that waiting
thread rather than having to ping pong
it and there's their new IP CI
new IPC API is coming with those and
related nig enhancements to come with
those well you saw earlier that I'm a
part-time mig pilot
what is mig mig is a tool that allows
you not to have to worry about the
details of how this IPC works it's used
for system services you typically won't
have to deal with it
a lot of the interfaces you may want to
call may be defined court graphic
services use it the low-level mock
services use it and it's just an ideal
that creates message formats for you and
packages them up so that you can just do
a remote procedure called model again
you'll typically use higher-level things
you'll use Apple script or Apple events
in your carbon application where you may
get down to in the lower level and use
CF messages but that's typically where
you'd stop if you want to go below that
and use mock what we suggest you don't
use mock messaging directly but try and
stick with the interface generator
there's other things that mock provides
won't have time really to get into those
but it provides all of the hosts
resources as well the system-wide
resources security management control
rebooting all that stuff is is a mock
function and processor control and
statistics coming soon to a processor
near you is power management and thermal
management a lot of that work is is
pretty much ready to go obviously all of
this stuff is not only as the core of
Mac OS 10 but it's also Darwin alright
it's open sourced this is keep in
keeping with Lamacq legacy mac has
always been or pretty much always been
an open source thing there was that
brief period where OSF did a lot of
research and didn't make it available to
anyone except their paying partners but
again we've we've taken that legacy and
now we're extending it by adding all of
our changes to mock into the open source
environment obviously when all else
fails having that source is a wonderful
form of documentation what is going on
why is my application not making any
progress well if it really comes down to
that you can dive down and look at the
source because you know that things
blocked in a fubar you know Mach message
overwrite trap well what am i what am i
waiting on right and you can actually
use in really really really bad
tuitions because we have open source you
can actually to be debugging the kernel
at the same time you're debugging your
application so you can kind of watch it
go through the kernel that requires two
machines but again if you're really
stuck it's an interesting way to look at
things the road map and while sorry most
of the things that talked about a lot of
this stuff has already been completed a
lot of the BSD and file system stuff but
there's some interesting sessions left
there's the performance tools later
today you definitely want to look at
that because when you're tuning your
application you're going to be looking
at things that show all of these mock
pieces
there's the feedback form tomorrow
actually tomorrow morning at 9 o'clock
is the debugging your your Mac OS 10
application session at 9 o'clock in the
big hall I didn't add this to that slide
but it's probably a really good session
to go to and they will dip into a minor
amount of these kind of things how as
you're debugging you're gonna see the
mock pieces showing through and then
there's a session also later tomorrow on
BSD and how the vyas the environment is
made available yeah where else can I
find out about this stuff if I care on
your CD there's a kernel environment
book and you can look at that it
actually discusses in more detail a lot
of these these pieces obviously the
Darwin open source you can go get that
and the Mac OS 10 homepage talks quite a
bit about what mock provides it's also a
book programming under mock it's getting
a little bit dated you may have trouble
finding it in some of your bookstores
but you can go to libraries and pick it
up there but it gives you a good overall
concept of how a mock is put together
and who to contact john signa if you
have business questions but you also
have the darwin mailing list to contact
us on technical issues most of us are on
the darwin mailing list all the time
watching what's going on and with that
I'd like to invite the rest of the team
up and we can have some Q&A
you