Transcript
[ Music ]
[ Applause ]
>> My name is Joe.
And later, I'll be joined by my
colleagues Simon and Scott.
We're from the Core OS Group and
we'd like to tell you about some
new developments with Kernel
Extensions in macOS 10.15
Catalina.
Kernel Extensions or Kexts, are
a technology that's been part of
macOS from the very beginning.
Using Kexts, you can build
powerful and innovative apps
that extend the built-in
functionality of the operating
system.
This power to extend the system
is an important part of what
makes the Mac the Mac.
But there are some problems with
Kernel Extensions.
They can be difficult to develop
and to debug, they can be a risk
to security and privacy on the
machine, and they can be a risk
to the reliability of the
system.
It's time for an upgrade.
MacOS Catalina introduces two
new technologies called System
Extensions and DriverKit.
Using them, your apps can extend
the operating system in ways
that are more reliable, more
secure, and easier to develop
than ever before.
Here's what we'll talk about
today.
First, I'll introduce these new
technologies and show you how
they avoid the problems of
Kernel Extensions.
Then, Simon will come up and
show us how to build Driver
Extensions using DriverKit.
Scott will show us how to write
and debug a simple USB driver.
And then, I'll tell you how to
include System Extensions in
your apps.
Let's get started.
System Extensions are our first
new technology in macOS
Catalina.
A System Extension is part of
your app that extends the
functionality of the operating
system in ways similar to a
Kernel Extension but running in
user space outside the kernel.
There are three kinds of System
Extensions that you can build in
Catalina.
They are Network Extensions,
Driver Extensions, and Endpoint
Security Extensions.
Network Extensions are a
replacement for Network Kernel
Extensions.
They can filter and reroute
network traffic or connect to a
VPN.
For more information on Network
Extensions, there's a session
dedicated to them on Friday
morning.
Endpoint Security Extensions are
a replacement for Kexts that
intercept and monitor
security-related events with the
KAUTH interfaces.
Some of the apps you can build
this way are Endpoint Detection
and Response and Data Loss
Prevention apps.
If you're interested in Endpoint
Security Extensions, please,
come to the Security Labs.
There's one happening right now
and continuing after this talk
and one on Thursday afternoon.
The third type of extension is
Driver Extensions which are a
replacement for Device Driver
Kexts using IOKit.
In Catalina, you can control
USB, Serial, Network Interface,
and Human Interface devices.
Driver Extensions are built with
DriverKit which is our second
new technology in Catalina.
DriverKit is a new SDK with all
new frameworks based on IOKit
but updated and modernized,
designed for building Driver
Extensions in user space outside
the kernel.
Now, that we've met these new
technologies let's see how they
avoid the problems of Kernel
Extensions.
There's a phrase I used in both
of those definitions; in user
space, outside the kernel.
Why does this matter?
The kernel is a very unforgiving
and difficult environment to
program within.
The kernel is the conductor of
everything that happens on the
machine, so it must never stop
running, must never wait for
anything to happen, and must
never crash.
Code in the kernel has to be
fast, has to be predictable, has
to be frugal with its use of
resources like memory, and has
to be essentially bug-free.
It's very difficult to write
code that meets all of these
restrictions.
System Extensions run in user
space outside the kernel which
means they run in a comfortable
modern programming environment.
This makes them easier to
develop than Kexts where kernel
code has restrictions on when
and how it can allocate memory
or synchronize between threads.
This means it cannot use most
system frameworks such as
Foundation since they are not
designed to run in this
environment.
The only supported language for
Kext development is C and C++.
System Extensions, on the other
hand, have no such restrictions
which means they can be built
using any framework in the macOS
SDK
and can be written in any
language including Swift.
[ Applause ]
There is one exception to this.
Driver Extensions, because of
their close relationship with
hardware, still have some
restrictions.
They must use the DriverKit
frameworks and run in a tailored
runtime which isolates them from
the rest of the system.
Driver Extensions must be
written in C or C++.
However, the default is C++17.
[ Applause ]
System Extensions are also
easier to debug than Kernel
Extensions.
Attaching a debugger to the
kernel halts the kernel and the
entire machine including the
debugger.
This means you usually need a
second machine to debug and you
may need special cables or
network configuration to connect
the machines.
The cycle of building, testing,
and debugging a Kext can be slow
because any crash in the Kext
means the whole system has to
restart.
And the kernel debugger has
limited support.
It cannot do things like
evaluate expressions or print
the value of objects.
System Extensions, on the other
hand, can be debugged and the
kernel keeps running.
There's no need to restart if an
extension crashes.
You can build, test, and debug
all on one machine with full
debugger support.
But the biggest improvements of
System Extensions over Kernel
Extensions are in the areas of
security, privacy, and
reliability.
The kernel has many jobs, but
one of the most important is to
define and enforce the rules of
the system's Security Policy.
The kernel separates apps from
each other and from direct
access to hardware.
Then, it allows them to share
data and system services
following the rules of the
Security Policy.
When a Kernel Extension loads,
it becomes part of the kernel.
It has access to everything on
the machine.
This is where a Kext's power
comes from.
But it can also be a danger.
Because the Kernel Extension is
part of the kernel which makes
the security rules, it is above
the rules.
If a Kext has a bug that allows
it to be compromised, it can
take over the entire machine
doing things its developers
never intended and its users
don't want.
There are no security rules that
can restrain it.
This means that any bug in a
Kext can be a critical security
problem.
Any bug in a Kext can also be a,
can also be a, what's happening?
Where are my slides?
Any bug in a Kext can also be a
critical reliability problem.
[ Applause ]
Because the kernel does not just
crash, it panics, and the entire
machine has to restart.
If you're a Kext developer,
you've surely seen this dialog a
lot.
And unfortunately, so have too
many of our users.
Let's see how the picture
changes with System Extensions.
A System Extension runs in
userspace.
Like other apps, it has to
follow the rules of the System
Security Policy.
Unlike other apps, System
Extensions are granted special
privileges to do special jobs.
For example, they may have
direct access to their
associated hardware devices or
use special APIs to communicate
directly with kernel systems.
If a System Extension crashes,
the rest of the system and apps
are unaffected and keep running.
For all these reasons, we think
that System Extensions are a big
step forward for the Mac
platform.
In fact, we think they're such
an improvement that we recommend
you adopt them immediately.
As Sebastian said in the State
of the Union yesterday, "We are
beginning the process of
deprecating Kernel Extensions.
MacOS 10.15 Catalina will be the
last release to fully support
Kernel Extensions without
compromises."
[ Applause ]
Specifically, for the
capabilities supported by System
Extensions and the device
families supported by DriverKit,
using a Kernel Extension to do
that same job is now deprecated
and a future release of macOS
will not load Kernel Extensions
of these kinds.
In future releases, we will add
more kinds of System Extensions
and more device families to
DriverKit.
In turn, Kernel Extensions of
those kinds will also be
deprecated.
So, that's a brief introduction
to System Extensions.
They avoid the difficulties of
kernel programming by running in
user space which lets your apps
extend the system in ways that
are easier to develop and debug,
that protect the security and
reliability of our users' data.
And now, I'd like to turn it
over to Simon, who will show you
how to build Driver Extensions
using the new DriverKit Runtime.
[ Applause ]
Well, thanks, Joe.
As Joe just said, a Driver
Extension is a new type of
System Extension that controls
the hardware device and makes
its services available across
the whole OS.
We call these new Driver
Extensions a Dext.
And our goal is to make the
transition from our Kernel
Extension to a Driver Extension
as easy as possible.
To show you how Driver
Extensions work and how you can
build your own or transition
from a Kext, we're going to talk
about four things.
We're going to talk about their
lifecycle
and how they match and start.
And how they compete with Kexts,
and we're going to talk about
how to build them with the new
DriverKit SDK.
And we're going to go over some
security features such as
entitlements.
And finally, we'll talk about
some compatibility questions
about how you can deploy an app
to macOS Catalina and Mojave.
So, let's talk about the
lifecycle of a Kext.
Let's look at the process that
happens when a device appears
that has a Driver Extension.
We start with IOKit Matching
creating a kernel service to
represent your service.
This is written by Apple.
We then start a process hosting
your driver with your DriverKit
class instantiated.
And the process also as proxy
objects for any services it uses
such as its provider.
This device is using a USB
device or device.
This, this device is using a USB
device so it has a proxy object
to call that represents the
kernel device.
This means that DriverKit
drivers appears to Kernel
Extensions and can compete in
matching with kernel drivers.
You can see them in the registry
with tools like IOReg and you
can use the IOKit Framework APIs
with them.
Since DriverKit drivers are
separated from the kernel and
from each other, another device
will have its own process and
another instance of the driver.
In macOS Catalina, Apple has
started to move several of its
own drivers to Driver
Extensions.
Here, you can see a process that
is hosting a USB networking
device which is visible in the
registry as a normal ethernet
device to the rest of the OS.
And you can also see some other
processes in the registry
hosting several HID and serial
drivers.
So, now we're going to talk
about building your Driver
Extension with the DriverKit
SDK.
We wanted to make building
Driver Extensions an easy
transition for those of you who
currently build Kexts so we
started with the IOKit C++ APIs
that you are familiar with.
The DriverKit APIs are an
extension of the IOKit APIs to
userspace.
And we have collected them into
a new DriverKit SDK that is
separate to the macOS SDK.
This SDK has a limited API
surface for reliability and
security and there is no direct
access to the file system,
networking, or mock messaging.
This allows Apple to tailor the
userspace process to running
drivers and can give it elevated
priority and increased
capabilities.
So, let's talk about some of the
classes in the DriverKit SDK.
First, the IOService class
exists in DriverKit and is very
similar to the IOKit class.
There also IOMemoryDescriptor
and IOBufferMemoryDescriptor
classes available that are,
again, very similar to IOKit.
We also have replacements for
the IOWorkLoop and EventSource
classes in IOKit.
And finally, there's a new class
called OSAction that is required
to represent a C Function
Pointer.
So, let's take a look, closer
look at some of these classes.
The IOService class has the
lifecycle APIs from IOKit like
Start, Stop, and Terminate.
For synchronization, every
IOService has a default
DispatchQueue and all methods
are invoked on a queue including
interrupts, timers, and
completions.
IODispatchQueue is built on the
pre-run Grand Central
Dispatch code and is a special
version optimized for running in
the restricted DriverKit
environment.
Drivers do have control over
their queues and which methods
are invoked on which queues for
advanced use.
The event APIs are similar to
the IOWorkLoop model in IOKit
but are now based on the Grand
Central Dispatch APIs with
queues and dispatch sources for
interrupts and timers.
The block API is replaced by a
command gate and GCD provides
synchronization parameters that
are easy to use and very likely
familiar to you.
There is also an
IOSharedDataQueueDispatchSource
class that provides a shared
memory ring buffer for fast, for
low overhead message passing.
The last class we'll talk about
is OSAction which encapsulates
what would be a callback in the
IOKit APIs.
They are always asynchronous and
they hold the callback client's
state privately.
And they allow the callback to
be defined with arbitrary
arguments and with type checking
in compile and runtime.
So, there were some of the
classes in DriverKit.
Let's look at how we define
these classes which is a little
different from IOKit.
DriverKit interfaces are
described with a new file type
with the .iig extension and they
are processed by a tool also
called IIG.
The IIG file is a class
definition that is compiled by
Clang and can import C and
C++ headers for common types and
structures.
But it does have some extra
attributes to its class and
method definitions that allow it
to be used, to be used for
calling from separate address
spaces.
Here, you can see a basic class
definition that looks mostly
normal but it has some extra
attributes such as kernel on the
class, which means the class is
defined in the kernel.
And local on some method
declarations which means the
method is implemented in a user
driver.
Some of the families that are
available in the macOS Catalina
developer preview today are
NetworkingDriverKit for creating
network interfaces.
HIDDriverKit for creating HID
devices,
USBSerialDriverKit to make a USB
serial device available to the
OS, and USB DriverKit to make
use of USB device providers in
your drivers.
Shortly, Scott will be
demonstrating what a USB device
support looks like with
USBDriverKit.
So, now we're going to talk
about some security aspects of
developing Driver Extensions.
There are a few types of driver,
of, of entitlement that your
Driver Extension will need to
obtain.
There's one for all Driver
Extensions and there's one to
take control of, of a device
called the transport entitlement
which is specific to the kind of
device.
And there is also a family
entitlement that is required to
make available a service into
the OS.
And Joe will talk later in this
session about code signing and
the approval process for
obtaining these entitlements.
Now, we'll have a quick word on
shipping products that work on
all versions of macOS.
Shipping a product for both
macOS Catalina and earlier
releases will require you to
install a Kernel Extension for
older releases but use the
System Extensions Framework and
provide a Driver Extension on
macOS Catalina.
So, now Scott will be
demonstrating how to make use of
the new USBDriverKit Framework.
Thanks, Scott.
[ Applause ]
>> Thanks, Simon.
So, today I'm going to show you
how to build a simple USB driver
that reads data from an
interrupt endpoint using the new
USBDriverKit Framework.
We'll briefly go over how to
create a new project in Xcode
using the DriverKit Template.
After that, we'll take a look at
a kernel class versus a
DriverKit class.
Then, next, I'll go into the
details of the implementation in
DriverKit.
And finally, I'll give a short
demonstration of the driver in
action and how you can debug
your Dext live with LLDB.
Creating a new DriverKit project
with Xcode is as simple as
selecting the proper template
during Xcode's new Project
Workflow.
Once completed, Xcode will have
autogenerated a few files to
help you begin.
The generated project includes
the standard files needed to
successfully build.
As with the Kernel Extension,
the project includes the C++
implementation, entitlements,
and an info.pist.
In addition to those, Xcode has
also generated the IIG file that
Simon mentioned earlier.
This file contains the class
definition for your driver.
So, let's take a look at the
class definition for
MyUserUSBInterfaceDriver.
You can see how this looks very
similar to a kernel driver.
For example, the same public
IOKit Lifecycle method, start
and stop existing DriverKit
would have been capitalized.
That said, there are a few small
but important differences.
First, the DriverKit class
requires a different callback
with an additional attribute.
This attribute indicates that
this method conforms to the
callback type defined by the
IOUSBHostPipe object and
enforces compile-time type
checking.
And second, there are no
instance variables declared in a
DriverKit class.
This is because all instance
variables must be allocated by
the driver during
initialization.
So, let's take a look at how
that's done for
MyUserUSBInterfaceDriver.
First, you need to declare a
structure to hold all of your
instance variables.
All instance variables that
would have previously been part
of your kernel class should be
part of this structure.
For this class, we have pointers
to the same USB kernel types
that a Kext would.
Such as an IOUSBHostInterface
provider and IOUSBHostPipe
object for performing IO.
And there's also an OSAction
object that will be used to
encapsulate the callback for
asynchronous IO.
Then, you simply need to
allocate the structure during
your INIT routine.
And here's the INIT routine for
MyUserUSBInterfaceDriver.
It calls INIT on the superclass
in the same way a Kext would.
And then, it allocates the IVAR
structure.
It should be noted that the
superclass defines an IVAR's
member that must be used to
assign the result of the
allocation.
So, next, we'll take a look at
the implementation of Start.
This portion of Start is
responsible for calling into the
superclass and validating the
provider.
Here, things are slightly
different than the kernel
implementation.
You can see the definition is
wrapped in a macro IMPL.
And this macro is required to
support the IPC communication
between the user process and the
kernel proxy object.
And you can also see that
calling super start takes a
different form.
Next, using the USB DriverKit
APIs, you open your IOUSB host
interface provider and you
allocate your Pipe object.
And then, allocate a memory
descriptor to be used for IO.
This should be a fairly familiar
paradigm and is basically
identical to what's done in a
Kext.
In this case, we're performing
asynchronous IO, so we need to
allocate an OS attribute object
to encapsulate the callback.
And then, finally, all that
remains is to enqueue the IO.
At this point, assuming the
setup was successful, there's an
asynchronous read which we'll
call ReadComplete when finished.
And the ReadComplete method for
this driver just prints the
number of bytes transferred and
the status.
If successful, it re-enqueues
the IO.
So, next, let's take a look at
the driver in action.
So, in this demo, you'll see
some logging I've added to
MyUserUSBInterfaceDriver that
will print some of the lifecycle
methods.
I've also added an infinite loop
which we'll debug using LDB.
And then, I've also introduced a
crash which we can see how is
recoverable now using the new
DriverKit Framework.
So, if I plug in the device you
can see INIT and start run just
like they would in a Kext.
And ReadComplete is being called
as data is being transferred to
and from the device.
Using PS, we can see that our
driver is running.
And now, we've hit my infinite
loop that I added and we can
take a look with LDB at what's
happening in the driver.
So, from earlier, we can see our
PID is 2572, so we need to
attach to that process.
We need to find the thread
running the ReadComplete method.
You can see, that's thread two.
And here, we've definitely got
an infinite loop.
And because we're running in
userspace, we can modify our
loop variable.
And before I continue, if you
look closely, you can see
there's definitely a null
pointer D reference, which will
crash the driver.
And you can see it's crashed.
But then, immediately restarted
without affecting the rest of
the system.
And then, on unplug you can see
your stop and free methods would
run as normal.
[ Applause ]
And so, that's--
[ Applause ]
-- that's how easy it is to
build and debug a new driver
with the new DriverKit
Framework.
Now, I'll hand it back to Joe to
talk about how to deliver System
Extensions in your apps.
>> Thank you, Scott.
Now, that we've seen how to
build a Driver Extension, I'd
like to tell you how to ship a
driver or other type of System
Extension in your app.
We'll talk about how your
extension relates to your app,
how to build and package the
extension bundle.
We'll talk about code signing
and entitlements and how to
install, update, and uninstall
your System Extension.
A System Extension is always
part of an app.
This is a basic principle of the
design.
There's no such thing as a
standalone System Extension.
This is because users think in
terms of apps.
They purchase apps, install
apps, and run apps.
Your System Extension should be
an implementation detail of your
app.
The app is how the user
interacts with and controls your
extension.
Once you've packaged your System
Extension into an app, you can
distribute that app directly to
your users using Developer ID or
through the Mac App Store, which
has never been possible with
Kernel Extensions.
[ Applause ]
Because of the close
relationship between your app
and its extensions, your System
Extension should be identifiable
by the user as related to your
app.
You should use the
CFBundleDisplayName key in the
extension's info.plist to give
it a good localized name and
give it a custom icon that
relates to your app's main icon.
This way, if the extension is
ever shown in the user
interface, a user will recognize
it as part of an app that they
use.
You should also include a usage
description string in your
extension's info.pList that
explains simply what the
extension is and does and why a
user would run it.
Think of this as similar to the
usage stings required for
calendar or camera access.
For Driver Extensions, use the
key OSBundleUsageDescription.
And for other types of System
Extensions, use the key
NSSystemExtensionUsage
Description.
Remember to localize these and
all other strings for all the
languages your app supports.
The System Extension itself is a
separate sub-bundle of your app
with its own executable and
info.plist embedded within your
application.
Here's a peek inside a sample
application showing a System
Extension in its Contents
Library System Extension's
folder.
Driver Extension bundles use the
.dext filename suffix and the
package type DEXT.
They use OSBundle keys in their
info.plist similarly to Kernel
Extension bundles.
Driver Extension bundles should
be flat bundles with no contents
folder, similar to iOS apps.
System Extension bundles of
other types use the .system
extension filename suffix and
the CFBundlePackageType SYSX,
System X tension.
In Xcode, your System Extension
is a separate target.
Xcode has templates for Network
Extensions and DriverKit drivers
built in.
When you create such a target,
Xcode will ask if you want to
embed it in an application
that's already part of your
project.
If you do, it will create a Copy
Files phase that copies the
extension build product into
your application.
Once you've built your System
Extension, you sign it with the
same certificate that you sign
your app.
There is no need for a special
Kext certificate, Kext specific
signing certificate, anymore.
[ Applause ]
Normally, the Team ID used to
sign your System Extension and
your main app must match.
This is a security measure.
However, you may be building an
extension that's intended to be
packaged in other developers'
apps.
For example, a Driver Extension
for a common type of USB serial
interface chip that's included
in many products.
If so, there is an entitlement
you can use on the System
Extension to allow it to be
packaged in a different
developer's app.
If you sign your System
Extension with Developer ID, it
must be notarized before it will
run on a user's system.
For more information on
notarization, please watch the
session from last year or come
to the Notarization Lab later
this afternoon.
Your System Extension uses
entitlements to describe its
capabilities to the operating
system; what type of extension
it is and what it can do.
For example, DriverKit Drive--
Extensions use the Transport and
Family entitlements that Simon
showed.
Your app that contains System
Extensions also should use the
com-apple-developer-system
extension-install entitlement.
For more information on these
entitlements and to request the
use of them by your development
team, visit developer.apple.com
/system-extensions.
In the Developer Seed of
Catalina for local development,
you can turn System Integrity
Protection off to disable some
of the checks for code signing
and entitlements while you are
testing.
Please, remember to turn System
Integrity Protection on again
when you're finished testing.
Now that you've built your app
with the System Extension, how
does the extension get installed
on a user's system?
There's no need for an installer
or a package.
Your System Extension stays in
your app bundle.
Your app uses the new System
Extensions Framework and creates
an activation request to request
that the extension be made
available for the system to use.
A system administrator will
approve this request.
Most apps should probably create
an activation request during app
launch so the extension is made
available right away.
If your extension has previously
been activated and approved, the
activation request will return
quickly with success.
However, you may wish to
activate your System Extensions
at different points in your
app's lifecycle such as after a
user has agreed to a license
agreement or made an in-app
purchase if that's required for
your extension.
Once your extension has been
activated, the system will
manage its lifecycle starting it
automatically when it is needed.
For example, a Driver Extension
will start when a matching
hardware device is connected.
To update your System Extension,
update your app bundle.
A user may install a new version
that they download from your
website, your auto-updater may
update the app bundle in place,
or if you release a new version
on the app store it will be
updated for the user.
The next time your app runs and
submits an activation request,
the system will notice that the
extension's version has changed.
It will ask your activation
request delegate to compare the
version numbers following your
own version number rules.
If your delegate determines that
this is an upgrade, the system
will stop the old version of the
System Extension and start the
new one.
If a user wishes to uninstall
your application, when they move
it to the Trash, all of its
extensions will automatically be
deactivated, as well.
There is a deactivationRequest
API if you wish to use it, but
there's no need for a dedicated
uninstaller.
So, today we introduced System
Extensions, the replacement for
Kernel Extensions that lets your
apps extend the system in ways
that are more reliable, more
secure, and easier to develop
than ever before.
We saw how to use the DriverKit
SDK and frameworks, which are a
modernized update of IOKit to
build a driver.
And we saw how to write and then
debug an example USB driver all
on one machine.
Finally, we talked about how to
include System Extensions in
your apps.
If you have any questions, we
would love to answer them at the
Core OS Labs.
There's one later today and one
Thursday morning.
You may also wish to visit the
Security Lab happening now or on
Thursday afternoon or the
Networking Lab on Friday
morning.
Thanks, very much, and enjoy the
rest of WWDC.
[ Applause ]