Transcript
[ Background Noise ]
>> Welcome everybody.
[ Applause ]
My name is Eric Tamura, and I am
a manager on the file systems
team primarily for iOS and our
embedded devices, and I'll be
talking to you about what's new
in Apple file system.
I'll also be accompanied by
Pavel Sokolov, who's a manager
of the file systems team largely
for the Mac.
So let's get started.
So this presentation will
roughly be divided into two
sections.
I'll tell you a little bit more
about what's new for iOS as well
as tvOS, watchOS, and then Pavel
will come up and tell you all
about what's new on macOS.
So let's jump right in.
So last year I got up on the
stage at this conference, and I
told all of you that Apple
[inaudible] Apple file system,
as the default for all devices
starting in 2017, and we've
certainly done that.
So as of iOS 10.3 as well as our
tvOS and watchOS analogs, all
devices have been migrated to
APFS.
We'll tell you a little bit more
about that.
We'll talk a little bit more
about the conversion process,
how did devices actually move
from HFS+ to APFS, and then a
little bit on normalization and
case sensitivity, particularly
as it relates to iOS devices.
So let's talk about that
conversion.
So at this point, millions,
literally hundreds of millions
of devices have now been
converted successfully to APFS.
[ Applause ]
So starting on iOS 10.3 as well
as tvOS and watchOS, Apple file
system became the default.
We had millions of devices
converted.
And one of the things you might
have realized that this update
actually does free up some
storage for end users.
In particular, there was space
previously occupied by the prior
volume manager, lightweight
volume manager, which is no
longer needed, and we were able
to free up that space and give
it back to users.
One of the other things is that
starting in iOS 11, we are able
to start taking advantage of the
APFS capabilities in iOS and
tvOS and watchOS.
Particularly iOS where iCloud
backup can now use Snapshots in
order to take a stable copy of
the file system and upload
everything to the Cloud without
worrying about whether or not a
process might be modifying a
file and getting rights
intermingled.
And finally we've been listening
to developer feedback in order
to improve and refine the file
system and all the software that
we've been working on.
So let's talk a little bit about
that conversion process.
So the APFS conversion occurs in
line during the update to iOS
10.3 and beyond.
So whether you're moving to iOS
10.3, 10.3.2 or even iOS 11 from
some prior update, it will
automatically get converted.
And one of the reasons that we
were able to do this so
successfully is that we actually
employed a dry run conversion
during iOS 10.0, 10.1, and 10.2
and be able to track how the
health of our conversion process
was actually going.
And we've also been monitoring
the health of the conversion
during iOS 10.3 and beyond
updates, and we've been watching
that as well.
Finally, we've been also looking
at the numbers and statistics
and everything that we're
watching and be able to improve
and refine the conversion on the
Mac as well.
So sometimes an illustration can
help make this easier to
understand, so I've got one
here.
So let's assume that this
rectangle represents the
entirety of a block device and
on the far left you'll notice
that there's and SB that really
refers to the superblock or the
HFS+ volume header.
Let's say for simplification we
have three metadata regions that
report all the entirety of what
the file system might be
tracking, things like file
names, time stamps, where the
blocks actually live on disk,
and that we also have two
regions labeled file data, and
if you recall during the
conversion process the goal is
to only replace the metadata and
not touch the file data.
We want that to stay exactly
where it is as if nothing had
happened to it.
So the first thing that we're
going to do is identify exactly
where the metadata is, and as
we're walking through it we'll
start writing it into the free
space of the HFS+ volume.
And what this gives us is crash
protection and the ability to
recover in the event that
conversion doesn't actually
succeed.
Now the metadata is identified.
We'll then start to write it out
to disk, and at this point, if
we were doing a dry-run
conversion, we'd end here.
If we're completing the process,
we will write the new superblock
on top of the old one, and now
we have an APFS volume.
The final step is to simply
remove the metadata that was
previously occupied by the HFS+
content.
We'll do that, and at this point
it's a completely converted APFS
volume.
So now a quick word on Unicode.
So if you know anything about
Unicode, you know there's
actually multiple variants, but
for the purpose of discussion
here, I will only be referring
two really, which are NFC and
NFD, and these roughly
correspond to precomposed
characters versus decomposed
characters.
And what this means is that you
can actually store the same
character visually, the same
visual representation can
actually be stored multiple ways
depending on which form you're
using.
Now prior to iOS 11, APFS stored
file names as nonnormalized
UTF-8, and what this means is
that if you have a file with one
representation versus the other,
they will actually be treated as
different file names for the
purposes of lookup and creation
and comparison.
So let's give another example to
make this easier to understand.
Let's say we have the character
n with tilde or [foreign
language] in Spanish.
You can represent this as the
Unicode hex f1 character.
Alternatively, you can also
store it as a decomposed
character in which the n is
stored separately from the
tilde.
Now in this case, on the right
the n and the tilde will be
stored as two separate code
points; however, they will be
treated as having to be combined
for the purposes of visually
representing that character on
your screen.
So the problem is that even if
the file system is taking care
to treat these as separate,
that's not necessarily what the
rest of the system expects and
interacts with.
This might be well-defined
behavior for NFS; however, you
want to make sure that the
entire software stack from file
system all the way up to
foundation and above is able to
treat everything properly.
So in order to rectify this,
we're introducing two new
normalization schemes for Apple
file system.
We have a native scheme as well
as a runtime normalization
scheme.
Now the native normalization
scheme is available in your high
Sierra beta that you got earlier
this week for case-sensitive
volumes only.
For case insensitive content,
Pavel will get up on stage in a
little bit and tell you all
about that.
The native normalization is
coming soon for iOS 11 for erase
restores only.
This means if you plugged in the
phone or other device to iTunes
and did a restore to factory
settings starting with iOS 11
will automatically pick this up.
However, we don't want anyone to
have to erase install in order
to get this behavior.
So we're also introducing a new
runtime normalization mechanism,
and the runtime normalization
will automatically convert
between the NFC content versus
the NFD for the purposes of file
comparison, being able to do
lookups.
If it doesn't find one, it will
automatically look up with the
other to make sure that your app
doesn't receive an ENOENT
[phonetic] error back from the
file system.
And a future update will
automatically convert devices
from whatever form they were in,
if they were already migrated
starting in 10.3 or 10.3.2 to
the new form in a future update.
So that's the update for iOS,
and with that, I'm going to hand
it off to Pavel, who is going to
tell you all about the Mac.
[ Applause ]
[ Background Noise ]
>> Good morning everyone.
My name is Pavel Sokolov, and I
manage file system team in
Apple.
Today I'm going to talk to you
about what's new in macOS with
regards to APFS.
Let's begin.
So first, I would like to talk
to you about what features are
here just in case you missed the
presentation from the previous
year.
Important features that we
have-- first of all, I would
like to remind you that APFS is
available to every
Apple-supported platform.
It works on iOS, tvOS, even on
your watch, and macOS, of
course.
One of the awesome features is
cloning.
It allows you to do fast copies
of the files.
It employs our copy-on-write
technology that also powers our
Snapshots.
Snapshots allow you to take a
[inaudible] Snapshot of your
file system, entire file system,
remember it's date, and later
possibly return to it.
We will also natively support
encryption.
We have space sharing.
I'm going to talk to you about
that a little bit later, and we
introduce defragmentation.
How will get to APFS you might
wonder?
So let's talk a little bit about
that.
Actually, all you have to do is
simply call a new macOS High
Sierra and your system volume
could automatically be converted
by installer.
In case you wonder what happens
with other volumes, they are not
going to be automatically
converted, but you can still
manually do so.
If you convert multiple volumes,
the important point to remember
after conversion each volume
becomes APFS container.
I'm going to talk about that a
little bit later.
And multiple containers do not
share space.
Space sharing is one of the most
important and awesome features
that's going to save you space.
And we would like you to take
advantage of that, and I can
show you how.
You might wonder what volumes
are supported, what we can
convert APFS.
We can convert HFS+ volumes,
CoreStorage, Fusion, and even
FileVault encrypted.
They all become APFS
automatically.
If you'd like to do a manual
conversion of your existing
volume, all you have to do is go
to disk utility [inaudible],
right click on the volume,
select convert to APFS, and here
you go.
As I mentioned, if you do so,
the volume becomes a container
and continues do not share
space.
We would recommend if you have
multiple volumes and would like
to take advantage of that
feature that you actually go and
add volumes to the system
container and move your data
over.
Why space sharing is so
important and what's so good
about it?
Let's talk a little bit about
that.
Here's the typical volume layout
that you might have as a
developer.
You probably have multiple
volumes, right?
What's the deal with multiple
volumes is as you start running
out of space in each of them,
you might have a little bit of
free space here, a little bit of
free space there, but you do not
have enough in one single
location so when you try to copy
your files, nothing you can do.
In APFS world, the container is
a partition on your volume, and
volumes share the free space of
that partition.
So let's look at that picture.
In a typical case, you would
have a GPT header, [inaudible]
service partition, and
[inaudible] each volume is a
partition.
In APFS world, as I said, we
have eight containers of this
partition and volume [inaudible]
side of it.
Let's say you want to fill up
the space.
As you can see in HFS you run
out.
In APFS, you still can use your
free space and have your data
here.
Let's explore the cloning.
Cloning is [inaudible].
It allows you to make fast
copies of the files.
In this particular case, let's
take a local [inaudible] file.
It has a couple of blocks, and
traditionally if you copy a file
you have to copy the data over,
and that takes time.
Not only it takes time, it takes
space.
In APFS, all we have to do is to
record the references to that
file, remember where the file
is, and only your metadata is
[inaudible].
It's also because it's fast and
you don't waste space.
You might wonder what happens if
I have to modify the file.
It's a single copy after all.
What happens is that copy and
write technology kicks in, and
the new file data is located in
the free space, and we just
create yet additional reference.
Quite simple, saves you space.
Let's explore another
interesting aspect of APFS is
sparse files.
Let me give you an example.
Let's say you have a picture of
a cat on your desktop, something
like that, and you want to edit
it.
Well let's say in APFS world and
both in HFS world, you would
have the same initial view.
You have a file that's 1
megabyte in size, is
[inaudible], and you want to
edit that.
What would happen if your
application for example decides
to seek past the end of the file
and do additional write.
Here's where the difference is.
In HFS, the file system would
fill the space that you seek
after with zeros, and your file
would be 3 megabytes of size,
right?
So mega portion of the file
would be zeroed out by the file
system, and it would take your
space.
In APFS, what you would have is
a sparse file.
So middle portion of the file,
still logically here, if you
read from that, you would read
zeros, but physically it takes
only two blocks.
Again, saves you space, makes
your life faster, but you have
to be aware of that feature.
So as you can see, there are two
sizes.
Logical size and the physical
size.
You need to be aware of that,
but it actually becomes a little
bit more complicated if you put
Snapshots or clones into the
picture.
So let's explore that a little
bit.
How big is your file is
[inaudible].
Let's say the same situation
applies and you have a file that
sparse, only takes two blocks
and just going to take a
Snapshot, record the location of
those blocks, and now we want to
modify the file.
As you see, copy and write
technique kicks in, will
allocate new space for the new
block that you just wrote.
So how big is your file?
It's still three megabytes,
right?
Physically you still only have
two blocks for it allocated, but
[inaudible] delete the file, you
only get the one block back,
important thing that you need to
be aware about.
So let's talk a little bit about
next important aspect in your
file system is Unicodes and
normalization support.
As you guys all know, that's
almost invisible until we have
problems.
So let's explore how the world
differs in APFS versus HFS.
By default, APFS is case
insensitive and normalization
insensitive, unlike HFS, which
is case insensitive and
normalizing file system.
What makes the difference?
Actually pretty simple.
In APFS, we keep the file names
in exactly the same way you pass
it to us.
We compute case insensitive and
normalization insensitive
[inaudible] together with the
file name and store it together.
So when you want to do a lookup,
we perform the same hashing
approach.
We do hash comparison, and
that's how we can find the name
fast and efficient.
Important caveat here, if you do
a [inaudible] of directory
structure, the names would come
out in the order of the hash,
not in the order of natural
lexicographical comparison like
you normally would expect.
We're also supporting Unicode
9.0, unlike HFS that supports
3.2.
We do not allow unassigned code
points unlike HFS, and this is
done for the sake of future
compatibility.
So whenever new Unicode standard
emerges, you can easily take
advantage of that by replacing
folds and tables.
In APFS, we support canonical
folds unlike in HFS, which
supports simplified folds, and
canonical folding is better
because it allows you natural
sorts and quarter in the same
way that users expect.
Let's talk a little bit about
Unicode 9.0 and why that's a big
deal.
You might wonder, who cares
about Unicode version.
Well, you should.
Unicode 9.0 makes your
applications truly global.
For example, in Unicode 8, the
support for Native American
Cherokee was added, and now we
have lower case symbols and
capital symbols to properly
support it.
We support pretty much
everything language that's
written and understood by man,
and if that's not global, I
don't know what is.
We support latest emojis.
You might wonder why?
Of course, because it
[inaudible].
And you might think, okay, HFS
supported that too.
So what's the big deal?
The big deal is actually
correctness.
HFS supported that sacrificed
some aspects of correctness.
Now the folds in APFS is
completely Unicode 9.0
compatible and for some of them,
we just were certain rules
succeed, let's say like capital
I was taught in Turkish language
or sharp S in German, they would
be capitalized properly.
Let's talk a little bit about
boot support.
Of course APFS is bootable.
You would like to boot your Mac,
right?
And will have embedded EFT
driver.
What embedded means in that
particular case is the driver
actually baked in a file system
for [inaudible] and it's very
easy, can be loaded by third
parties that support virtual
machines.
You don't really have to
complete the full driver and the
standard full format.
All you have to do is follow a
couple of pointers from the
superblock and load the driver
that's baked into the volume
itself.
That makes it future approved so
in case volume format changes
all you have to do is just load
new driver from the new volume.
We can boot from encrypted
drives.
We can boot from fusion drives.
An encryption boot actually
makes it more secure.
Let's talk a little bit about
security and encryption.
We support the same FileVault
model that we used to.
We have converted for existing
FileVault volumes.
We support the same recovery
keys and not only will we
support the same mechanisms, we
will also make it compatible so
whenever you convert, you
preserve your existing recovery
keys, you preserve your existing
passwords.
You do not have to move your
data all over the place, and
conversion doesn't take that
much time.
Also important point, your
Snapshots are encrypted as well,
and that means if you took a
Snapshot at some point and you
didn't have a password, and then
you enabled encryption, your old
data is protected as well.
We support Fusion drives.
The he same features that you
used to have before are still
there, is write back cache and
read cache.
To take advantage of faster SSD
devices, we made small
improvement, and now all your
metadata always going to be on
SSD, that makes this a little
bit faster.
Another important feature is
defragmentation.
We support intelligent
defragmenter.
What makes it intelligent?
It's smart enough to understand
[inaudible] performance first to
most were which files were most
fragmented and it defragments
them first.
It defragments only hard drives,
and defragmentation happens when
your machine is idle and it
doesn't interfere with your
normal workflow.
Another interesting topic is
Snapshots.
Let's explore this a little bit.
So what happens is imagine you
have a file system, a couple of
files here, and you want to
remember the state of that file
system for future reference.
What happens here, will remember
all the references to all the
files, create a Snapshot of the
metadata and record it.
If you later make [inaudible] to
your file system, let's say you
[inaudible] file, the same copy
on the write technology kicks
in, will allocate space in a new
location, write your files.
Your existing blocks are intact.
Now let's say you delete a file.
The blocks are not actually
gone, only the reference to that
block is gone.
So later if you want to revert
to a Snapshot, it will restore
the state the way it used to be.
For example, your [inaudible]
will be gone, and the file that
you deleted would be restored.
A Snapshot [inaudible] feature.
They're fast to take.
You need to be mindful that they
occupy space.
Because of that, creating
Snapshots requires entitlements,
and if you want to take
advantage of a [inaudible],
please talk to us.
With that, I'm going to show you
a demo.
I'm going to invite Pavel Cisler
and going to show you a demo of
the time machine.
[ Applause ]
>> Thank you, Pavel.
Hello everybody.
My name is Pavel Cisler, and in
case you wonder what's up with
this guy's first name and the
last guy's first name being the
same, let me just say, we did
some testing on clone file, and
[laughter]-- anyway, so I manage
the Finder and Time Machine
teams.
Who is a Time Machine user here?
Wow, cool.
A lot of you guys.
Anybody here know about the
mobile Time Machine feature?
Some of you.
Not as many.
Okay. So I'm going to show you
mobile Time Machine a little
bit.
Let me switch over to my demo
machine here.
So mobile Time Machine is this
convenient part of Time Machine
that's a feature that lets you
back up your changes when you
are on the go and you're away
from your backup device, your
time capsule, or your hard
drive, right.
Let's say you take your laptop
on the road.
Your time capsule is at home.
Your machine still continues to
make hourly backups through
mobile Time Machine, and that's
a great feature, right?
It doesn't necessarily help you
with things like you lose your
laptop or you lose your entire
hard drive, but if you
accidentally erase that
important spreadsheet or
whatnot, mobile Time Machine can
save the day.
So we have this feature since
MacQuest and Lion, and let me
show you real quick how it
works.
So here I am.
Here's the Time Machine restore
UI, and my laptop here, I
brought it from my office, I'm
not connected to any time
capsule, and yet I still have a
backup history here, and I can
browse and see my files.
And here's that secret budget
document that I was working on
earlier that I accidentally
deleted, and just like that I
restored it, and I can continue
working on it.
So simple feature.
You can really save the day when
you need it.
Now you guys are developers, and
you would like to know how it
works under the hood, right?
So on MacQuest and Lion when we
first developed this feature, it
was quite involved.
It takes no less than two demons
to do.
One of them is a virtual file
system overlay.
If any of you had a chance to
work on something like that, you
know it's pretty tricky.
So it's literally tens of
thousands of lines of code.
So for macOS 10 High Sierra, the
Time Machine team was super busy
deleting all this code.
So what we did, we reimplemented
mobile Time Machine on top of
APFS Snapshots, and that's
actually what I just demoed to
you here.
So never mind this UI.
Let me open up Terminal and show
you some of it under the hood.
When we talk about Snapshots,
what do they look like?
What do they feel like?
Well, they feel very much like
any other volume, and as such,
you can inspect them using a
variety of volume management
tools.
My favorite is Mount, so let me
go ahead and type Mount here.
Without any arguments, if you do
that, it lists all your disks,
and you can see here that at the
top there's my slash, there's a
swap space here.
A bunch of other common volumes,
but then there's these funny
volumes that are mounted at a
mount point that continues to
[inaudible] that Time Machine in
the path, and there's also this
strange time stamp.
Well, these are all APFS
Snapshots that represent the
individual hourly mobile Time
Machine backups, right.
Each of them is a Snapshot.
Each of them mounted as a
volume.
So if I go into the Time Machine
restore UI on the right here,
that's what they are.
So let me go back.
So these Snapshots aren't
actually mounted all the time.
They get mounted [inaudible], so
they don't have to stick around.
Mobile Time Machine only kicks
them in when it needs them and
[inaudible] them after.
So let me show you how that,
what happens when you do that.
So I'm going to simulate what
mobile Time Machine would do
when it's done with Snapshots
and unmount them all.
I'll use this command line too
called tmutil with a verb called
unmount local Snapshots.
And it's going ahead and
unmounting all the Snapshots.
So now if I entered the restore
UI, it actually has to remount
them, right.
So that was super quick, right.
So it just mounted what, like
some 20 volumes just like that.
So that's fantastic.
Now, I'm going to go ahead and
do a backup.
Mobile Time Machine backs up
once an hour.
It wakes up and creates a
Snapshot.
We're not going to sit around
for an hour here and wait for
that.
So I'll use a tool again.
Tmutil Snapshot and it just
backed up, right.
So you saw it took a little less
than a second to back up the
entire machine.
So that's fantastic.
[ Applause ]
So I'm going to go ahead and
show you that it actually
worked.
I'm going to accidentally delete
some data here.
Oh no, what did I do, my
important movie.
And I'm going to go back into
the restore UI and, ah, whew,
there it is, and restore it.
So again mobile Time Machine
saves the day.
So how come this was so fast?
Well, as the other Pavel
explained to you, snapshotting
is a copy and write technology,
so you basically just grab
references to your data, and the
actual copying happens later if
there's an actual change.
It's deferred.
It's lazy, right.
So much more faster, much more
efficient.
So the other thing I'm going to
point out here is that this
movie that I just restored is
actually pretty big.
It's a movie of our last year's
talk.
It's 1.1 gigs, and the old
mobile Time Machine had certain
limits, like it couldn't back up
changes to files over 20
megabytes.
Well, all these limits are gone
now.
Mobile Time Machine on top of
APFS Snapshots will back up
anything in a very speedy way
and efficiently.
So that's mobile Time Machine on
top of APFS Snapshots.
All right.
S otherwise that, I'm going to
switch back to the slides and
Pavel will tell you more about
some of the APIs we've used to
make this happen.
Thank you everyone.
[ Applause ]
>> Thank you, Pavel.
[ Background Noise ]
So you might wonder what about
APIs, how to take advantage of
all that.
So good news is if you program
[inaudible] like you probably
should, it's all taken care of
for you.
The copyItem API automatically
takes advantage of API files
clone feature.
How that happens behind the
scenes we see if volume is
compatible with cloning, and we
take clones automatically as you
copy the files.
There's also replaceItem API,
which takes care of [inaudible]
save save, again automatically
you don't have to do anything
for that.
There are a couple of guidelines
that I want to remind, good
practices, and important points.
In particular, [inaudible] data
is duplicated.
We announced that a year ago,
and not only is it duplicated,
it's also not going to work with
API files.
Instead of that, if you want to
take advantage of [inaudible]
file swap, [inaudible] save
save, use new API which is
called rename underscore np.
Also, please do not use carbon
APIs.
Again that's [inaudible].
It works in compatibility mode
with API files.
It's going to be much slower.
It's [inaudible] the file
system.
Instead, use foundation APIs.
If you want a little bit lower
layer, you can use [inaudible]
copyfile with copyfile function
that supports copyfile clone
flag.
There are a couple of caveats
for that [inaudible] preview.
In particular, we're not going
to automatically upgrade
HDD-based file systems.
We're still working on
performance, and if you'd like
to get bootable APFS, it's not
enough just to run the
converter.
You have to go through the
installer.
So APFS is coming to macOS.
It's here.
In fact, it's supported file
system.
It supports Fusion drives.
It will support native
encryption.
Space sharing is an awesome
feature.
There are clone files and
snapshots that are going to make
your storage more efficient and
faster.
Please test your apps as you
just noticed there are a couple
of important differences between
HFS and APFS.
We would like your apps to be
successful and awesome.
Please give it a try, and report
your bugs.
There's more information
available for you at that URL.
Thank you.
[ Applause ]