WWDC2013 Session 707

Transcript

X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[ Silence ]
>> Jerry Cottingham:
Good Morning.
I'm Jerry Cottingham, I'm an
engineer on the Core OS IO team
and today we're going to talk
about what's new in
Kext Development.
So in my part of the
session, I'm going to talk
about Kext Development
overview, give you a little bit
of overview of the
Kext development.
The Developer ID program,
why that's important for you
and your attention please, I'm
going to call out some things
that are in OS X 10.9 that
you need to be aware about.
We're assuming that
the target audience
for this is somebody that's
already developing a kext,
you think you're about
to develop a kext.
And if you're here, you know the
top three reasons why you need
to avoid writing a kext
if at all possible.
So if you're thinking about
it then here's a couple things
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that you need to think
about before you get
into writing a kext.
Kexts require wired
memory, so all the memory
that your kext consumes
is not available
for the rest of the system.
Mistakes are often fatal,
so if you run it writing an
application, you make a mistake,
you crash the application.
You're writing a kext, you
make a mistake and you crash,
you're crashing the whole
system so you affect everybody.
Debugging is harder.
So today, if you're developing
an application with Xcode,
debugging is very
simple, very, very simple.
If you're debugging a kext,
you need two machines,
you need to setup the machine,
you need to configure
it correctly.
The problems that you're dealing
with are often much harder
to figure out, so
debugging is much harder.
So, if you're still not deterred
and you still really think
that you need to develop a kext,
then if you must there's
two things that you need
to read before you even start
writing your first line of code.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The Kernel Extension
Programing Topics
and the Kernel Programing
Guide, so make sure you read
through those, and if you
still need to develop a kext,
make sure you talk to us to
see if there's any other way.
And, if you're going
to write a kext,
you're mantra should always be,
"With great power comes
great responsibility."
Now let's quickly go
over a little overview
of what kext development
is all about.
So a kext is just a bundle
that extends the kernel.
Kext stands for:
Kernel Extension,
and it's only available
for OS X.
So if you're trying
to write a kext
for iOS, we don't allow that.
And a kext is structured
just like any other CFBundle.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Do you see the binary
down in the OS folder?
So here the Apple sample PCI
kext, you're going to see this
through all three of our talks
today about kext development.
You'll see that the executable
is down in the OS folder.
The Info.plist describes all
the properties of your kext
and I'm going to call out
two here that we're going
to discuss later today.
The CFBundleIdentifier, so
that identifies your kext
in the whole kext name space,
so that's a unique identifier
across the whole system.
CFBundleVersion, so that allows
the kext management system
to figure out which
of your kexts,
if it has the same
bundle ID to load.
Now this becomes very
important and today when we talk
about some of the other
things that we're changing,
that you are bumping that
version number as you go along.
Okay, so you're part of the
kernel and part of that means
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that you have to buy
into extra protection.
So, we require specific
File and Folder permissions.
Kexts must be owned by root
and the group must be wheel.
Must only be writable by
root, and permissions are deep
so this little folder here on
the right shows you if you did
like an 'ls -alR' all
permissions all the way
through would look like that.
If you get any of
those incorrect,
your kext will not load.
And you can use kextutil
-tn which is test no load
to verify your kext as
you're developing it.
Okay new in OS X 10.9, we
are now code signing kexts
and checking the
signature of your kext.
In OS 10.9 all kext
signatures are verified.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Unsigned or invalid
signatures are not fatal
with just one exception
and I'll talk
about that exception
in just a little bit.
So if you sign a kext in
10.9 and try to run it
on earlier versions, newer
than 10.8, it will not load,
there was some code signing
machinery changes in there
that will not allow those to
load, so just keep that in mind.
Valid code signatures will
eventually be mandatory
on all kexts.
So now in 10.9, there's
only one place its mandatory
and I'll talk about that, but
be prepared, be proactive.
Let's get your kexts signed and
you'll be ready for the future.
And again, kextutil -tn is good
for testing your
code signatures.
Now I have to mention in the
seed that you have today,
I didn't quite get around
to getting that working,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
so it's fixed in the next
seed they will actually test,
so you can just try a kext
load to verify your signature.
Okay so now I want to talk
a little about kext loading.
So there's three types of loads
here I want to talk about.
Autoload. So autoload is
basically your IOKit KEXT
when it matches hardware that's
attached to your machine,
the kext machinery goes
through, matches the best driver
for that, and loads it for you.
Now, where we find
those kexts now is
in /System/Library/Extensions.
So on-demand loading
of kexts, so you can do
that by an explicit path,
so here we're using kextload
with our Apple sample PCI kext.
So I just give it
a specific path.
You can also use your
CFBundleIdentifier.
So here is kextload -b with a
big, long bundle ID that we have
for our sample kext, and again,
if you supply nothing else
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to kextload, we'll go
search for that bundle ID
in System/Library/Extensions.
And the other place that
your kext gets loaded is
from the kernel cache.
So the kernel cache is built to
speed up the boot of the system
and in there is anything
that is required for booting,
for rooting, anything that needs
to be loaded early can be
put in the kernel cache.
And again, when we
build that, by default,
we'll look for those kexts out
of System/Library/Extensions.
So you see a pattern there
that /System/Library/Extensions
is very important
for us finding kexts
automatically.
Okay now in 10.9, autoload,
on-demand load by bundle ID
and kernel cache builds
will look not only
in /System/Library/Extensions,
but also /Library/Extensions.
So, and again, if you touch,
the /System/Library/Extensions
we will automatically rebuild
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
the kernel cache.
/Library/Extensions is the
same thing except, again,
the seed that you have,
that will not work
until the next seed.
We have a fix that
I just put in.
Okay, and you must sign
your kext if you're going
to deploy them in
/Library/Extensions.
So this is the exception I
was talking about earlier.
So, if it's not signed, we
will not load it from there.
Okay. So we've done
some research to find
out where your kext is
already installed recently.
Of course,
/System/Library/Extensions
and those are needed for
autoload or autosearch.
/Library/Application
Support, that's a fine place,
and that's where you guys are.
/System/Library/Extensions/File
Systems,
that's where mainly our
file system kexts are.
/Library/File Systems are
your File System kexts.
Library startup items, in
the application folder,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
inside your app bundles,
that's a perfectly good place
for your kexts and
/Library/Extensions.
Again, it did not support
autosearch or autoload.
In OS 10.9, same setup, these
are all perfectly fine places
and now we have the
autoload and autosearch
in /Library/Extensions and,
again, the caveat there is
that they must be signed.
Okay where we want
your kexts installed.
So the autoload kext that
required rooting, booting,
early, you know, early
load from the kernel cache
and autosearching of
kexts, again, we want those
in /Library/Extensions
and they must be signed.
/System/Library/Extensions
for compatibility.
So if you have shipping kexts
out there, which you already do,
it's fine to leave those in
/System/Library/Extensions.
As you revise and start
signing your kexts,
move them to
/Library/Extensions,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and anything on 10.8 and
earlier, the unsigned kexts,
leave them in
/System/Library/Extensions,
that's fine, all other
kexts though, any other kext
that doesn't need
to be autoloaded
or automatically
found by the system.
Signed kexts can go into
/Library/Extensions,
so just because we autosearch
and autoload from there,
if you sign your kext, it's
still fine to put it in there.
Even if you don't
need to be autoloaded.
Do not install anywhere
in /System.
So in the future, we are
going to tighten down access
to the system hierarchy,
the whole hierarchy
down from /System and
everything in there.
So, be proactive and stay out of
there because changes are coming
that will require
that you not be there.
So all the other common
locations that I talked about,
applications in your application
bundle, all those are fine.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So, you may need to have
two kexts for compatibility
and in a little bit we'll
show you actually how
to build those kexts, sign them,
and deploy them across
multiple OS's.
Okay. So, we're talking
about signing your kexts,
so how do you go about that?
The Developer ID
Program is your mechanism
to get your kext signed.
So what is this?
Its part of the Mac
Development Program,
which you're probably
already a member of.
It allows distribution
of applications outside
of the Mac App Store.
So a kext is not
going to be delivered
through the Mac App Store,
so this is actually a
great place for you guys.
And then you get a special
Developer ID Certificate
for signing your applications
and now in OS 10.9 you're going
to get a new certificate for
signing applications and kexts.
So how do you go about getting
this Developer ID Certificate?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Very simple.
So you go up to
developer.apple.com,
Resources/Developer ID,
you click on the link
to request a Developer
ID Certificate,
you fill out the form, and then
once the form is approved you'll
go back to the Member Center
and in
Certificate/Identifier/Profiles
you will see this.
This should be active
right now, by the way,
so you should be able to go up
and get your certificate
right now,
so click on the Developer
ID Application
and kernel extensions,
and you're ready to go.
So for more information
you can go back
up to developer.apple.com
and then, also,
Session 702 from last
year's WWDC, Perry talked
about Gatekeeper and
Developer ID, so he talked
about the Developer
ID Program last year
in context for applications.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
All that stuff he talked about
in that session, he gives a lot
of information about
code signing,
is applicable to you guys too.
So go back and look at
that, there's a great deal
of information there for you.
Okay. So in the seed that
you have now, in 10.9,
I want to call out some
things that are going
to affect you and our customers.
The first thing, as I said
before, /Library/Extensions,
kexts in /Library/Extensions
will not load
if they're unsigned or if
the signature's invalid.
So what this means is that
the user will see this no load
alert dialogue.
Basically, it says the kext is
not going to load and the part
that affects you is that
very last line in there
that says contact the Developer
to get updated software.
There's a second alert
that will come out.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Anything that is a kext outside
of /Library/Extensions will load
if it's unsigned or if it has
a signature validation problem,
but the user will
see another alert.
In that particular case, it
will say that yes we're going
to load it, but please
contact the vendor
to get an updated
version on this.
Also, what you guys will see as
you develop your kext and some
of your customers might
contact you and say, "Hey,
I saw this thing in the System
Log," so there's some warnings
and error messages, so
the warnings would be
if it's actually going to load
the kext, but it has some kind
of problem in verification,
and the error messages are
when you get that alert
that says I'm not going
to load it at all.
So the most common code
signature error codes
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that you're going to
see is the -67030,
which means that something in
the bundle has been modified.
Something that, the kext
signature is no longer valid,
so if somebody went in and
messed with your Info.plist,
then you're going
to get this error.
The other one is -67062, which
is the kext is not code signed,
which means there's no
signature at all on this kext.
So there's all kinds of error
codes in the security framework
and you can go to the
cs.h and see all those.
So here's an example of when I
took the Apple sample PCI.kext,
built it, had it signed, then
I went into the Info.plist file
and changed just
a line in there,
and then I tried
to load the kext.
And so this is exactly
what you're going to see
in that particular case.
So this is an example of
a modified Info.plist,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
invalidates the signature,
and this is the message
that you'll get.
Okay, in summary, sign your
kext with your Developer ID,
it's very simple and easy,
get out of
/System/Library/Extensions
as quickly as possible, in fact,
all of /System, and if you think
that you need to be in /System
for some reason anywhere down in
that hierarchy anywhere, please
talk to us and let us know
so we can provide
alternatives to that.
And get your kexts into
/Library/Extensions.
Okay, now Dean Reece
is going to come up
and actually show you how to
build your kext, get it signed,
and deployed across
many versions of the OS.
[ Applause ]
>> Dean Reece:Hi.
Thanks for joining us today.
My name is Dean Reece and
I manage the IOKit Team
in Core OS, and I'm
just going to run
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
through a couple examples
real quick.Let's get started
with the most basic
project here just to kind
of set a base to start from.
We're going to build a kext that
follows the new signing rules
and installation
paths and only loads
on OS X 10.9.So you're going to
need to use at least OS X 10.8.3
as your minimum build
for the build machine
and this is necessary to
get correct code signing.
It was a change to the security
framework that's available
in that build and
later OS releases.
You're also going to need
to use Xcode 4.6 or later,
and for these examples
I'm showing you Xcode 5
which is preferred.
And as Jerry described,
you're going to need
that Developer ID
Certificate that allows you
to sign kernel extensions,
and you're going to install
that on your keychain so
Xcode can get to it.Okay,
so you create a new Xcode
project and there's two flavors
of kernel extensions: The
"Generic Kernel Extension",
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
which is typically used for File
Systems and network plug-ins.
An "IOKit Driver",
which is typically used
for autoload hardware
drivers.And you'll go
through the normal creation
process so we'll give it a name,
a unique bundle identifier,
and you have a project ready
to start putting code into.So,
we're going to set a
few properties and most
of these are going to default
this way, but a couple won't
so we'll walk through those
very quickly.Make sure
"Architectures" is set to
"Standard Architectures".
You only need to building for
64-bit Intel on OS X 10.9.
And also make sure
the "Base SDK" is set
to "Latest OS X (OS X
10.9)".Under Code Signing,
this is new for kernel
extensions, so you're going
to want to change the
"Code Signing Identity"
to your Developer ID Certificate
that you've installed
on your keychain.
You can set it to this setting
which would match
any Developer ID.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
If you have multiple
Developer ID Certificates,
then you might have to pick the
specific one that is authorized
for signing kernel extensions.In
the deployment section,
another change here,
you're going
to have change the
installation directory.
It will default to
/System/Library/Extensions,
and of course as we said,
that is now /Library/Extensions
for OS X 10.9.
And the deployment target should
match.We're not going to go
into the actual coding
of the kernel extension,
that's outside the
scope of this talk,
but there's a few quick
reminders that I wanted
to give you.When you're
developing a kernel extension,
you may only use
headers that you find
in the Kernel.framework.
The Kernel.framework is intended
for use by kernel extensions
and conversely, it cannot
be used by applications.
So all the other frameworks
are for applications,
Kernel.framework is for kexts.If
you're writing an autoload kext
or IOKit kext and you
happen to subclass OSObject
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
or any class that's
derived from it,
which all the IOKit classes are,
then you must use the OSDeclare
and OSDefine macros
that you'll find
in the OSMetaClass.h header.
This is important for
our binary compatibility.
This includes additional symbols
that our run-time system picks
up and uses so that we
can patch up your kext
and make it binary compatible
with whatever OS
you're loading it on.
So, for example, if you build a
kernel extension using the OS X
10.8 SDK and you fail
to use these macros,
that kext will only ever load
on OS X 10.8; it won't load
on OS X 10.9, you'll get symbol
errors.And the last thing,
is you need to not strip those
binary compatibility symbols
out of the binary.
You can use "strip -S -x".
That's safe for stripping
a kernel extension binary,
but anything more restrictive
might rip out those symbols
that we need.The project we
just described builds one kext.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
It will load on OS X 10.9
with no errors or warnings
and you're going to
debug it using lldb.
In a few minutes Brian's going
to join us and describe how
to do that, but you'll need
to be running OS X 10.9
on both the target system
and your host system
for that lldb connection.And
of course after you package it,
part of your Q.A. process
should be to verify
that it is installing
in Library/Extensions.
I think that might
have come up once
or twice.Now let's expand
this project a little bit
and we're going to add
support for OS X 10.8.
To do that, we're going
to create a new target,
and we'll have visit those
same settings that we talked
about before.But
before we do that,
we're going to check
our version numbers.
Now, this is very
important; this calls back
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to what Jerry told you earlier.
If you have multiple copies
of your kexts installed
on the system, and the system
encounters both of them,
it's going to pick the one with
the highest version number.
Because of the way that
this project is going
to install its kexts,
we're going to say
that the target intended for
Mavericks, for OS X 10.9,
will have a higher
version number.
And that the one intended
for OS X 10.8 will have
the lower version number.
The reason this works is
because the newer kext is going
to be installed in
/Library/Extensions,
and older OS's don't look there
so they won't see it by default,
they won't know about
the newer version
and they'll just ignore it.
Whereas Mavericks will see
the newer one and load it
by preference.Okay, so let's go
and visit those same
build settings we talked
about last time through.
Your new OS X 10.8
target is going
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to have the same
architecture setting,
again you only need
64-bit Intel there,
but your "Base SDK" will be
"OS X 10.8" in this case.And
of course "Code Signing
Identity" will default
to "Don't Code Sign", but you
should verify that is the case,
because you don't
want it signed.And the
"Installation Directory"
in this case will need
to be
/System/Library/Extensions,
and the "OS X Deployment
Target" needs
to match the SDK.This
project now builds two
kernel extensions.
The signed one is
exactly the same
as the one we built
in the first example.
And the unsigned kext
that is being built
by the new target will load
on OS X 10.8.Now you're going
to use gdb to debug that one.
That's the divide
between 10.8 and 10.9:
10.8 and earlier is gdb,
10.9 and later is lldb.And
you're going to verify
that that did, in fact,
go
to
/System/Library/Extensions.Since
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
you'll be installing
two kernel extensions,
it might be a good idea if you
were to look at the target OS
in your install scripts, see
what OS you're installing into
and avoid installing the legacy
kext if you're installing
onto a Maverick system, because
it will never need it.And
another warning I'll throw
out here is in the future
as we start to lock
down the /System folder,
you might actually
get write errors.
So when you try to
install a kernel extension
into the /System folder,
the write itself may fail.
Your installer script needs
to be smart enough to know
if it tries to write something
there, as long as the one
that went
into /Library/Extensions
made it okay,
that's good enough.Alright.So
let's expand one more OS
release back.
In this case we're going
to modify the project we
just created for OS X 10.7,
but one difference is we're
going to use Xcode 4.6.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
You can't use Xcode 5 for this
because Xcode 5 doesn't
have the 10.7 SDK.As before,
you're still going to
need that certificate
for signing the Maverick's
kext.We're going to start
with the same project as
last time, but we're going
to show the screenshots
in the older Xcode.
So the second target is going
to have slightly different
settings, particularly
to target OS X 10.7, we have
to add the 32-bit Intel
architecture for that,
for full compatibility.
And the resulting kext
from that target will load
on both OS X 10.7 and
10.8.As before, you're going
to make sure that the
module identifier,
the CFBundleIdentifier
that Jerry talked about,
is the same for both targets.
And you're going to make
sure that the legacy kext,
the one that's installing
in the old path,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
/System/Library/Extensions,
has the older version
number.Let's go and look
at these build settings.
Again, "Architectures",
you need to make sure
that its standard for this SDK.
So the "Base SDK" is
going to be "OS X 10.7"
and the "Architecture"
is going to be both 32
and 64-bit Intel.And as
before, this target needs
to not sign the kext.In
deployment, again,
/System/Library/Extensions
is where it has to go
for OS X 10.7 and
10.8 to see it.
And the "OS X Deployment
Target" needs to match the SDK
as before.This project
will also build two kexts.
The signed kext is
unchanged, it's exactly
as the previous two examples.
But the unsigned kext should
actually load on both 10.7,
all flavors of 10.7 and all
flavors of 10.8, and you will,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
again, use gdb to debug that.
Oh yeah, and as before,
you're going to want
to install only the
kexts that are necessary.
If you're installing to
a 10.7 or 10.8 system,
you'll install both kexts,
but if you're installing
into an OS X 10.9
system, you only need
to install the signed
kext.What happens if you want
to target releases even
further back than 10.7?
It is possible.
You can't do it from a
single Xcode project,
because there is no single Xcode
that spans that many releases.
So probably the best
approach for you is
to take your existing
kernel extension project
that already runs on OS's
up through 10.8 and just use
that kext as the unsigned
kext that you install
in /System/Library/Extensions.
And then use the
simple OS X 10.9 project
that I described first, and
just import the source code
and build a second kext that
way and that will install
in /Library/Extensions.So
effectively you're kind
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of building a line in the
sand between Mavericks
and older operating systems
that didn't use signed kernel
extensions, and so it's kind
of a clean break in the
new kext project and focus
on Mavericks and the future.
So, when you're building for
older OS's, I couldn't go
into all the different
permutations, it really depends
on what OS's you're targeting,
what families you are using
within IOKit and so on as
to exactly what compatibility
you would have.
But you need to follow
the compatibility guidance
that we've released for
previous operating systems
when targeting those, and
again, you're kind of separating
out the 10.9 kext anyway.
So, the compatibility
guidance we've given you
in the past still applies
for the legacy kext.And
always install only the kexts
that are particularly
necessary for the target system
and also make sure that you're
robust against write failures
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to the /System folder
for future.So this is your handy
pocket guide for compatibility.
The first column here
says, OS X 10.7 and beyond.
Those are all the various
settings you're going to need
and each column further
to the right takes it
further 1 OS release.
This is mostly for the at-home
audience that can freeze frame
on this and very quickly
get all those build settings
that they need.All
right, and with that,
I would like to turn the stage
over to Brian Bechtel
who's going to talk to you
about lldb.Thanks
for your attention.
[ Applause ]
>> Brian Bechtel:
Hello everybody.
I'm Brian Bechtel.
I work in Core OS
doing Panic Triage.
I look at panics
all day and we have,
I know it's a wonderful
job [laughter].
We have a saying in our group.
[Laughter] And it really is true
that after about 10,000 panics,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
there pretty straightforward.
In Mac OS X 10.9 we have
a new debugger, lldb.
It is a very powerful, very
fast, very nice debugger.
It is slightly different.
I'll be talking about
some of those differences.
The big things we get out of
lldb are tighter integration
with llvm and the rest of
the development tool chain.
We have a much, much
faster debugger.
I will be demoing that later.
And we're supporting
it very actively
and we're very much listening
to developer reports of issues
and fixing them, which
is a very good thing.
lldb includes Python as
its scripting language.
That means you can do almost
anything that you can do
in Python inside lldb, it
gives you way more power
than the old macro
language in gdb.
There is a gdb to lldb command
map up on the llvm.org website,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
which helps you make
this transition,
but much of the commands
are exactly the same
for the things you typically do.
lldb is the debugger for
Mac OS X 10.9 and beyond.
gdb is dead.
[Laughter] gdb is
gone in Xcode 5.0.
There is no gdb binary there.
There is no kgmacros file in the
kernel debug kit for Mavericks.
We have no plans to fix
any remaining issues.
It is still the debugger
of choice for Snow Leopard,
the cats basically, if you
have a cat, gdb is your friend.
If you have a giant wave or
some other California feature,
lldb is your friend.
This is a demo of how fast gdb
is, versus how fast lldb is.
I'm just going to
issue one command
in the debugger while connected
via gdb to a 10.8 system
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and lldb to a 10.9 system.
I've set up two terminal
windows here,
the light brown one
is running gdb
and the white one
is running lldb.
I'm even going to let gdb
start first so you're going
to see a cursor there
that touches that window,
I hit Return, then go down
and hit the other window.
And what you see
is lldb is much,
much faster at just
printing the zone list.
Every other command in lldb has
this kind of speed improvement.
It's really quite, quite
noticeable, and wonderful
if you're doing 10,000 panics.
[Laughter] All of the standard
kgmacros are still there,
I said there's no kgmacros file,
but the kgmacros themselves,
reimplemented in Python, are
in the mach kernel.dSYM file
in a directory, which
I'll discuss later.
These are all the
standard macros that we use
when we're debugging to just get
the generic information we need
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
for a panic.
They're still there, there
are some minor changes
in the output format, but
everything works great.
For adding kext symbols,
on startup,
lldb will automatically
scan for kexts installed
in System/Library/Extensions,
the new Library/Extensions,
and in the directory
/Volumes/Kernel/DebugKit
because that's the kext
symbols that we give you in all
of the symbols for IOKit kexts.
You can specify additional
locations in the .lldbinit file
in your home directory.
There's the command line for it.
And of course while
you're debugging,
you can add kext symbols
using the addkext macro.
I'll have some examples
of that later.
Every time you change something
you'll have little gotchas,
these are some of
the ones that hit me
that I thought you guys
would like to know.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
lldb likes to pretend it
doesn't know how to do addition.
Backtick is your friend.
Backtick, if you get a, if
you try to display something
and it says, "I don't know
what that is that you're trying
to display," put the
expression in backticks.
It sometimes fixes that.
gdb was very, very sloppy
about describing a structure
or a pointer to a structure
or fields in a structure.
lldb is very pedantic about that
because it gets very pedantic
information from llvm.
So if you're saying this is
a pointer to a structure,
you have to say it's a
pointer to a structure,
it'll painfully warn you that
you probably meant to say that
and let you reenter
your command.
And you may need to cast a
variable very explicitly.
We've had issues where you
know that this class is called,
"MyClass," and you say, "Print
MyClass," and instead it says,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
"I don't know what MyClass is,"
but if you say, "Class/MyClass,"
or "Struct/MyStruct," it'll
say, "Yeah sure I can do that."
Anything you encounter like
that, we want to hear about it,
so please go to
bugreporter.apple.com,
file any issues you're
having, we really want
to hear those things, they're
going to the engineers,
they will repair them or
at least have a reason why.
So one of the great things
in lldb is the mixed
mode disassembly.
A lot of times when
you're trying to figure
out what happened, it's
difficult to translate
in your head what is
going on in the assembly
versus what the compiler did.
The disassemble -m option
gives you mixed mode
of whatever you're asking for.
You can say, for instance,
"disassemble -f -m,"
which will disassemble
the entire current frame
with both source and assembly,
intermixed so you can
see where everything is.
If you just want to see the
current line that you're stopped
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
on you can say, "disassemble
-l -m
or you can give it
a start and address.
And you can abbreviate that,
the minimum command abbreviation
for disassemble is di.
So you can say di -f -m.
Here's an example of
mixed mode disassembly.
The stupidest, simplest
example I could come up with.
There's a routine in the
kernel called, fseventsf-write,
which just returns
EIO, an error.
So this is the example of
that routine disassembled
with mixed disassembly.
Here you see the
prologue of the routine.
EIO turns out to be the value
of 5, so here it's moving 5
into the return register.
And here you see the epilogue.
So in a much more
complex example,
this is really, really helpful.
Here, you could probably figure
that out in a day or two.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[Laughter]
[ Silence ]
The kgmacros now take
an additional argument,
or series of arguments.
You can now redirect output
from a macro directly
to a file with -o.
If you get a macro that
has a lot of output
and you don't want
all that output,
you can grep inside
the output with -s
and you can add additional
verbosity to the macros
to see additional details
about the macro with -v and it,
it is cumulative so you can
say -v -v -v, you can be anal
and see how far out you can go.
So here's an example,
if you want to find
the load information
for just our sample PCI kext,
you can say, 'showallkmods,"
which shows all of
the kext information,
and then say -s AppleSamplePCI
to just get the information
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
on one line rather than have to
scroll through a lot of lines.
For security purposes, we do
not automatically load the lldb
macros in lldb by default.
The reason we do this is
if you're doing application
debugging
or you have a malicious kext,
and they've put some bad lldb
macros in a hidden symbol file
or something, bad
things could happen.
You know? It's the
internet nowadays.
So to overcome this because
you trust your development
environment and your setup,
you can add that
particular setting line
to your .lldbinit file and then
it will automatically load the
script from the symbol files.
You can also create custom
Python and put it in the,
in the dSYM file
in the Contents/Resources/Python
directory,
and then that will be
automatically loaded assuming
that you have your
settings set correctly.
For faster loading, a lot
of times you only want
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to see your particular
kexts, you don't care
about Apple's kexts,
you don't care
about all those other things.
Add this line before you
do the connection or put
that in your .lldbinit file
and it won't load
all the other things
and you just come right
back up to Ready To Go.
And then at that point if you
want to automatically load,
or manually load symbols for
your kext, you can use any
of these four examples
of the addkext command.
You can either pass it
the UUID, pass it the name
and if the name is found
in one of the places
that lldb searches,
it will load it.
You can give it an
explicit path, an address,
or you can say, "Hey, I changed
my mind I want them all loaded,"
that will take a lot
longer than it would
if you'd just let lldb
do it at the beginning.
nvram is needed, I'm
going to now be talking
about two machine debugging.
We use nvram to enable
debugging at all.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
If you have no debug
boot argument set,
your system will
reboot on panic,
that's a new behavior
starting in 10.8.
If you set boot-args using the
nvram command line tool as root
and reboot to let those settings
take effect, you can change
that behavior so
you'll halt at panic
and let you connect
from another machine.
So in the old days, back when
we had just iMac's or something,
we had built-in Ethernet
and built-in FireWire.
Nowadays, probably most of you
out there, have a MacBook Air
or something that no longer has
built-in Ethernet nor built-in
FireWire, so we've added
new ways to connect.
You can now connect for two
machine debugging using a
Thunderbolt Ethernet
Adaptor, using a Thunderbolt
to FireWire Adaptor, using
an Apple Thunderbolt display
in either Ethernet or FireWire
on the back of that monitor.
We do not support wireless and
we do not support USB Ethernet,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
so you're going to have to get
one of those other connections.
When you connect, two-machine
debugging assumes a network port
of en0, that's the old
behavior from the original iMac,
MacBook Pro, Mac Pro, whatever.
If your network port isn't
that, you have to add,
as part of your boot arguments,
kdp-match-name=,
and then some value.
That some value will
depend on your connection.
For a network connection,
ifconfig, will tell you
which port you should use.
So if you have a Thunderbolt
to Ethernet connector
you would use ifconfig
to find the connection and
then kdp-match-name=en whatever
that number is.
Here's an example of that.
This is the output from the
ifconfig command just cut
down to a single port.
I'm looking for a port with an
inet value that makes sense.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So, for instance, here
at the conference,
we're running off the
10.0 net or 10.1 net,
I'm not sure which one.
I look for a connection
that's 10.
whatever, the en1
tells me that the port
that I will be using
in kdp-match-name=.
So for built-in debugging on
a machine like a MacBook Pro
with built-in Ethernet, you
would just set your boot args
as this: debug=0x144, the
100-bit means don't put
up the graphic panic
screen, just halt.
The 4-bit means let me
connect to the machine,
and the 40-bit means
let me connect
over a different
subnet if necessary.
So you can simplify this to
104 instead if you wanted.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
For Thunderbolt to
Ethernet, you'll notice
that same command line
has added kdp-match-name=
and then an en value.
And for Thunderbolt Display,
you would do the same thing.
I'm giving examples
here with en2 and en4,
but you actually need the
output from ifconfig to figure
out what the proper en value
is, and you would put that in.
Many times this value may
coincidentally be there,
and it may be the right value,
but you have to use ifconfig,
you can't just trust a
slide at a conference.
[Laughing] If you're
using native FireWire,
you set kdp-match-name
to FireWire.
If you're using Thunderbolt
FireWire Connector
or Apple Thunderbolt
Display to FireWire,
you have to add an additional
command, which is fwkdp=8000hex.
That tells the Thunderbolt
System not to turn off power
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to the FireWire chip, which
would be rather inconvenient
when you're trying to connect.
And as always, you
need to reboot
to let the settings take effect.
So for FireWire debugging
it's very simple,
you open up two terminal
windows.
In one terminal window,
you enter fwkdp,
in the other terminal
window you invoke lldb,
and then connect
to the local host.
The fwkdp is a redirecting
program that makes it look
as if your local
machine is panicked,
but it actually redirects the
commands out the FireWire port
to the panicked machine.
It's important to note
I used xcrun there.
xcrun guarantees
that you're going
to be running the lldb
that's in your current Xcode.
You want that because who knows
what horrible thing you've got
installed in /usr/bin
or /usr/local/bin
that might accidentally
be grabbed.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
If you use xcrun lldb it will
always find the proper lldb
for your Xcode.
And I'm giving a couple examples
of how you connect to a machine.
If you're connecting live,
you can either give a
name or an IP address.
You can use a Bonjour name
although there is a warning
that with Bonjour you may have
a reassignment of an address
as you're going along,
so that's a little shaky.
I tend to either use a
fully-qualified domain name
or a dotted IP address.
And if you're reading a coredump
file, the command is "file --
c" and then a path to the file.
In fact, talking about
coredumps, I'm going to talk
about several things that
we've changed recently
that are helpful for you.
The first thing that you'll
notice is we've changed how you
do a NMI in Mac OS X 10.9.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
With boot-args set
so that the 0x4 bit
or the DB-NMI bit is
set, in 10.8 or earlier,
you'd just hit the Power key.
Anybody who's done
kernel debugging
and hit the Power key will
discover that the Power key
on a MacBook Air is directly
next to the Delete key.
And then you curse and
reboot your machine
because you didn't have
it setup for debugging.
So we changed that in Mavericks.
Now it is Left and Right
command keys and the Power key.
This will make your
day-to-day life a lot easier.
If you really like the old
behavior, you can revert to that
by OR'ing [phonetic]
in the hex 8000-bit,
so instead of debug=0x144,
you'd say debug=8144.
You can automatically
save coredumps.
This works far back in the
system, but I'm documenting it
to make sure everybody has it
because it's a pretty
easy process.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
As root on a host machine,
the machine where you're going
to save the coredump, you need
to create a directory
called PanicDumps,
change it to permissions, and
then launch the kdump.plist,
which will load a demon that
will sit there waiting to see
if anybody wants to send
it a coredump to save.
On the client machine, on the
machine that you want to panic
and save a core from, you
can set your boot-args
where debug is 0xd44 panicd ip=
and then the dotted IP
address of the host machine.
It has to be a dotted IP
address because at the time
of a panic we can't
do DNS resolution.
We've added some new
protection boot-args,
which will help you track
down buffer overruns,
use after free errors,
mismatched frees, you know,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
freeing from a zone that you
didn't actually allocate from,
and some timing-induced races.
-zc, the allocate code inside
the kernel for zone allocations,
checks the free-list
pointers of all the zones
for correctness every
time you do an allocation.
-zp, the free routine,
will overwrite freed memory
with deadbeefdeadbeef.
That will catch any attempt to
execute or access freed data.
And here's an example of how
you would set that: debug=144,
for the two-machine debugging,
-zp -zc, and of course reboot
to let it take effect.
We also have a guard mode
kernel zone allocator.
This will let you target
any zone with a certain size
or any zone above a certain
size, below a certain size,
or in a range of sizes.
You set, specify one of
gzalloc-size, gzalloc-min,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
gzalloc-max, or min and
max, and then gzalloc-wp.
And that will write-protect
the allocations rather
than unmapping them.
So the allocation
will still be around,
but it will be write-protected.
Zp and zc aren't very much
of an impact on performance,
I wouldn't do your
benchmarking with them set.
But they don't really
take up too much memory
and they don't really
take up too much time
in the allocation and free.
The gzalloc-wp, gzalloc-size
stuff, every allocation is going
to be put on its own 4k page
with an additional guard page,
measurably slower,
you'll notice it
when you're using the machine,
but it catches all kinds
of bugs, it's a very
valuable tool.
Another point I wanted
to make is that starting
in Mac OS X 10.8,
we changed graphics
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to support multiple buffers.
We draw the console output
tool buffer in instead
of directly to the screen.
This was an enhancement
needed in order
to support the retina
display Macs, but it applies
across all the machines.
Unfortunately, that means you
have unreliable console output
to the screen if you panic.
So if you have your boot-args
set to display text instead
of a panic screen, you
may not see that text,
it will appear to hang.
The only way to tell
is to attempt
to connect to the machine.
For more information
about all of this,
we have documentation
online, we have a contact here
from the evangelist group,
and the developer forums
are actually get pretty good
coverage from people
inside of Apple.
I recommend understanding
and debugging kernel panics.
The Kernel Core Dumps,
I specified all
of the stuff you need from
Kernel Core Dumps on one slide.
So, it's not very difficult.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And there's a new revision
to the generating a
non-maskable interrupt,
which is in the Mavericks
section
of the developer website,
the original version
of that is referenced here.
We have some related
sessions later this week,
on Friday in the Pacific
Heights Conference Room,
they'll be an Advanced Debugging
with LLDB, I recommend you go.
So in summary, sign
your kexts in 10.9.
We've had two people
tell you that,
it's the wave of the future.
lldb is actually pretty cool.
I like it.
I prefer it over gdb
now that I'm used to it.
Like any new technology,
it takes a bit to get used,
but it's very powerful
and very nice.
We have new ways of
debugging for machines
that do not have built-in
FireWire or built-in Ethernet,
and we have new boot-args
that help you track down bugs.
There you go.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[ Applause ]