WWDC2018 Session 416

Transcript

[ Music ]
[ Applause ]
>> Hi, everyone.
My name's Kyle.
I'm a software engineer at Apple
and today we'd like to take a
deep dive into iOS memory.
Now, just as a quick note, even
though this is focused on iOS, a
lot of what we're covering will
apply to other platforms as
well.
So the first thing we'd want to
talk about is, why reduce
memory?
And when we want to reduce
memory, really, we're talking
about reducing our memory
footprint.
So we'll talk about that.
We have some tools for how to
profile a memory footprint.
We have some special notes on
images, optimizing when in the
background.
And then, we'll wrap it all up
with a nice demo.
So why reduce memory?
The easy answer is users have a
better experience.
Not only will your app launch
faster.
The system will perform better.
Your app will stay in memory
longer.
Other apps will stay in memory
longer.
Pretty much everything's better.
Now, if you look to your left
and look to your right, you're
actually helping those
developers out as well by
reducing your memory.
Now, we're talking about
reducing memory, but really,
it's the memory footprint.
Not all memory is created equal.
What do I mean by that?
Well, we need to talk about
pages.
Not that type of pages.
We're talking about pages of
memory.
Now, a memory page is given to
you by the system, and it can
hold multiple objects on the
heap.
And some objects can actually
span multiple pages.
They're typically 16K in size,
and they can come in clean or
dirty.
The memory use of your app is
actually the number of pages
multiplied by the page size.
So as an example of clean and
dirty pages, let's say I
allocate an array of 20,000
integers.
The system may give me six
pages.
Now, these pages are clean when
I allocate them.
However, when I start writing to
the data buffers, for example,
if I write to the first place in
this array, that page has become
dirty.
Similarly, if I write to the
last page, that, or the last
place in the buffer, the last
page becomes dirty as well.
Note that the four pages in
between are still clean because
the app has not written to them
yet.
Another interesting thing to
talk about is memory-mapped
files.
Now, this is files that are on
disk but loaded in the memory.
Now, if you use read-only files,
these are always going to be
clean pages.
The kernel actually manages when
they come in and off of disk
into RAM.
So a good example of this would
be a JPEG.
If I have a JPEG that's, say, 50
kilobytes of size, when it's
memory mapped in, that actually
is mapped into four pages of
memory, give or take.
Now, the fourth page is actually
not completely full, so it can
be used for other things.
Memory's a little bit tricky
like that.
But those three pages before
will always be purgeable by the
system.
And when we talk about a typical
app, their footprint and profile
has a dirty, a compressed, and a
clean segment of memory.
Let's break these down.
So clean memory is data that can
be paged out.
Now, these are the memory-mapped
files we just talked about.
Could be images, data Blobs,
training models.
They can also be frameworks.
So every framework has a DATA
CONST section.
Now, this is typically clean,
but if you do any runtime
shenanigans like swizzling, that
can actually make it dirty.
Dirty memory is any memory that
has been written to by your app.
Now, these can be objects,
anything that has been malloced
-- strings, arrays, et cetera.
It can be decoded image buffers,
which we'll talk about in a bit.
And it can also be frameworks.
Frameworks have a data section
and a data dirty section as
well.
Now, those are always going to
count towards dirty memory.
And if you might have noticed, I
brought up frameworks twice.
Yes, frameworks that you link
actually use memory and dirty
memory.
Now, this is just a necessary
part of linking frameworks, but
if you maintain your own
framework, singletons and global
initializers are a great way to
reduce the amount of dirty
memory they use because a
singleton's always going to be
in memory after it's been
created, and these initializers
are also run whenever the
framework is linked or the class
is loaded.
Now, compressed memory is pretty
cool.
iOS doesn't have a traditional
disk swap system.
Instead, it uses a memory
compressor.
This was introduced in iOS 7.
Now, a memory compressor or the
memory compressor will take
unaccessed pages and squeeze
them down, which can actually
create more space.
But on access, the compressor
will then decompress them so the
memory can be read.
Let's look at an example.
Say I have a dictionary that I'm
using for caching.
Now, it uses up three pages of
memory right now, but if I
haven't accessed this in a while
and it needs to, the system
needs some space, it can
actually squeeze it down into
one page.
Now, this is now compressed, but
I'm actually saving space or
I've got two extra pages.
So if, some point in the future,
I access it, it will grow back.
So let's talk about memory
warnings for a second.
The app is not always the cause
of a memory warning.
So if you're on a low-memory
device and you get a phone call,
that could trigger a memory
warning, and you're out.
So don't necessarily assume that
a memory warning is your cause.
So this compressor complicates
freeing memory because,
depending on what it's
compressed, you can actually use
more memory than before.
So instead, we recommend policy
changes, such as maybe not
caching anything for a little
bit or kind of throttling some
of the background work when a
memory warning occurs.
Now, some of us may have this in
our apps.
We get a memory warning, and we
decide to remove all objects
from our cache.
Going back to that example of
the compressed dictionary,
what's going to happen?
Well, since I'm now accessing
that dictionary, I'm now using
up more pages than I was before.
This is not what we want to do
in a memory-constrained
environment.
And because I'm removing all the
objects, I'm doing a lot of work
just to get it back down to one
page, which is what it was when
it was compressed.
So we really got to be careful
about memory warnings in
general.
Now, this brings up an important
point about caching.
When we cache, we are really
trying to save the CPU from
doing repeated work, but if we
cache too much, we're going to
use up all of our memory, and
that can have problems with the
system.
So try and remember that there's
a memory compressor and cache,
you know, get that balance just
right on what to cache and what
to kind of recompute.
One other note is that if you
use an NSCache instead of a
dictionary, that's a threat-safe
way to store cached objects.
And because of the way NSCache
allocates its memory, it's
actually purgeable, so it works
even better in a
memory-constrained environment.
Going back to our typical app
with those three sections, when
we talk about the app's
footprint, we're really talking
about the dirty and compressed
segments.
Clean memory doesn't really
count.
Now, every app has a footprint
limit.
Now, this limit's fairly high
for an app, but keep in mind
that, depending on the device,
your limit will change.
So you won't be able to use as
much memory on a 1-gigabyte
device as you would on a
4-gigabyte device.
Now, there's also extensions.
Extensions have a much lower
footprint, so you really need to
be even more mindful about that
when you are using an extension.
When you exceed the footprint,
you will get a exception.
Now, these exceptions are the
EXC RESOURCE EXCEPTION.
So what I'd like to do now is
invite up James to talk about
how we can profile our
footprint.
[ Applause ]
Thanks, James.
>> Thank you.
Thanks, Kyle.
All right.
I'm James.
I'm a software engineer here at
Apple.
And I'd like to introduce some
of the more advanced tools we
have for profiling and
investigating your application's
footprint.
You're all probably already
familiar with the Xcode memory
gauge.
It shows up right here in the
debug navigator, and it's a
great way for quickly seeing the
memory footprint of your app.
In Xcode 10, it now shows you
the value that the system grades
you against, so don't be too
concerned if this looks
different from Xcode 9.
So I was running my app in
Xcode, and I saw that it was
consuming more memory.
What tool should I reach for
next?
Well, Instruments, obviously.
This provides a number of ways
to investigate your app's
footprint.
You're probably already familiar
with Allocations and Leaks.
Allocations profiles the heap
allocations made by your app,
and Leaks will check for memory
leaks in a process over time.
But you might not be so familiar
with the VM Tracker and the
Virtual Memory Trace.
If you remember back to when
Kyle was talking about the
primary classes of memory in
iOS, he was, he talked about
dirty and compressed memory.
Well, the VM Tracker provides a
great way to profile this.
It has separate tracks for dirty
and swapped, which, in iOS, is
compressed memory, and tells you
some information about the
resident size.
I find this really useful for
investigating the dirty memory
size of my app.
Finally, in Instruments is the
VM Memory Trace.
This provides a deep view into
the virtual memory system's
performance with regards to your
app.
I find the By Operation tab
really useful here.
It gives you a virtual memory
system profile and will show you
things like page cache hits and
page zero fills for the VM.
Kyle mentioned earlier that if
you approach the memory limit of
the device, you'll receive an
EXC resource exception.
Well, if you're running your app
now in Xcode 10, Xcode will
catch this exception and pause
your app for you.
This means you can start the
memory debugger and begin your
investigation right from there.
I think this is really, really
useful.
The memory debugger for Xcode
was shipped in Xcode 8, and it
helps you track down object
dependencies, cycles, and leaks.
And in Xcode 10, it's been
updated with this great new
layout.
It's so good for viewing really
large Memgraphs.
Under the hood, Xcode uses the
Memgraph file format to store
information about the memory use
of your app.
What you may not have known is
that you can use Memgraphs with
a number of our command-line
tools.
First, you need to export a
Memgraph from Xcode.
This is really simple.
You just click the Export
Memgraph button in the File menu
and save it out.
Then, you can pass that Memgraph
to the command-line tool instead
of the target [inaudible] and
you're good to go.
So I was running my app in Xcode
10, and I received a memory
resource exception.
This isn't cool.
I should probably take a
Memgraph and investigate this
further.
But what do I do next?
Well, obviously to the terminal.
The first tool I often reach for
is vmmap.
It gives you a high-level
breakdown of memory consumption
in your app by printing the VM
regions that are allocated to
the process.
The summary flag is a great way
to get started.
It prints details of the size in
memory of the region, the amount
of the region that's dirty, and
the amount of memory that's
swapped or compressed in iOS.
And remember, it's this dirty
and swap that's really important
here.
One important point of note is
the swap size gives you the
precompressed size of your data,
not what it compressed down to.
If you really need to dig deeper
and you want more information,
you can just run vmmap against
the Memgraph , and you'll get
contents of all of the regions.
So we'll start by printing you
the nonwritable region, so,
like, your program's text or
executable code, and then the
writable regions, so the data
sections, for instance.
This is where your process heap
will be.
One really cool aside to all of
this is that all these tools
work really well with standard
command-line utilities.
So for example, I was profiling
my app in VM Tracker the other
day, and I saw the, an increase
in the amount of dirty memory.
So I took a Memgraph, and I want
to find out, are any frameworks
or libraries I'm linking to
contributing to this dirty data?
So here I've run vmmap against
the Memgraph I took.
And I've used the pages flag.
This means that vmmap will print
out the number of pages instead
of just raw bytes.
I then piped that into grep,
where I'm searching for a dylib,
so I need dynamic library here.
And then, finally, I pipe that
into a super simple awk script
to sum up the dirty column and
then print it out as the number
of dirty pages at the end.
I think this is super cool, and
I use it all the time.
It allows you to compose really
powerful debugging workflows for
you and your teams.
Another command-line utility
that macOS developers might be
familiar with already is leaks.
It tracks objects in the heap
that aren't rooted anywhere at
runtime.
So remember, if you see an
object in leaks, it's dirty
memory that you can never free.
Let's look at a leak in the
memory debugger.
Here I've got 3 objects, all
holding strong references to
each other, creating a classic
retain cycle.
So let's look at the same leak
in the leaks tool.
This year, leaks has been
updated to not only show the
leaked objects, but also the
retain cycles that they belong
to.
And if malloc stack logging was
enabled on the process, we'll
even give you a backtrace for
the root node.
One question I often ask myself
is, where's all my memory going?
I've looked in vmmap, and I see
the heap is really large, but
what do I do about it next?
Well, the heap tool provides all
sorts of information about
object allocations in the
process heap.
It helps you track down really
large allocations or just lots
of the same kind of object.
So here I've got a Memgraph that
I took when Xcode caught a
memory resource exception, and I
want to investigate the heap.
So I've passed it to heap, which
is giving me information about
the class name for each of those
objects, the number of them, and
some information about their
average size and the total size
for that class of object.
So here I kind of see, like,
not, lots and lots of small
objects, but I don't think
that's the problem.
I, that, I don't think that's
really the problem here.
By default, heap will sort by
count.
But instead, what I want to see
is the largest objects, not the
most numerous, so passing the
sortBySize flag to heap will
cause it to sort by size.
Here I see a few of these
enormous NSConcreteData objects.
I should probably attach this
output and the Memgraph to a bug
report, but that's not going far
enough, really.
I should figure out where these
came from.
First, I need to get the address
for one of these NSConcreteData
objects.
The addresses flag in heap.
When you pass the addresses flag
to heap with the name of a
class, it'll give you an address
for each instance on the heap.
So now I have these addresses, I
can find out where one of these
came from.
This is where malloc stack
logging comes in handy.
When enabled, the system will
record a backtrace for each
allocation.
These logs get captured up when
we record a Memgraph, and
they're used to annotate
existing output for some of our
tools.
You can enable it really easily
in the scheme editor in the
diagnostics tab.
I'd recommend using the live
allocations option if you're
going to use it with a Memgraph.
So my malloc's, my Memgraph was
captured in malloc stack
logging.
Now, to find the backtrace for
the allocation.
This is where malloc history
comes in helpful.
You just pass malloc history,
the Memgraph, and an address for
an instance in memory, and, if
there was a backtrace captured
for it, it'll give it to you.
So here I've taken the address
for one of those really big
NSConcreteDatas.
I've passed it to malloc
history, and I've got a
backtrace.
And, interestingly, it looks
like my NoirFilter's apply
method here is creating that
huge NS data.
I should probably attach this
and the Memgraph to a bug report
and get someone else to look at
it.
These are just a few of the ways
you can deeply investigate the
behavior of your app.
So when faced with a memory
problem, which tool do you pick?
Well, there are 3 ways to think
about this.
Do you want to see object
creation?
Do you want to see what
references an object or address
in memory?
Or do you just want to see how
large an instance is?
If malloc stack logging was
enabled when you record, when
you, when your process was
started, malloc history can help
you find the backtrace for that
object.
If you just want to see what
references an object in memory,
you can use leaks and a bunch of
options that it has in the
[inaudible] page to help you
there.
And finally, if you just want to
see how large a region or an
instance is, vmmap and heap are
the go-to tools.
As a jumping off point, I'd
recommend just running vmmap
with the summary flag against a
Memgraph taken of your process
and then follow the thread down
there.
Now, I'd like to hand back to
Kyle, who's going to talk about
what can be some of the largest
objects in iOS apps, and that's
images.
Kyle?
[ Applause ]
>> Thanks, James.
So images.
The most important thing about
images to remember is that the
memory use is related to the
dimensions of the image, not its
file size.
As an example, I have this
really beautiful picture that I
want to use as a wallpaper for
an iPad app.
It measures 2048 by 1536, and
the file on disk is 590
kilobytes.
But how much memory does it use
really?
10 megabytes.
10 megs, that's huge!
And the reason why is because
multiplying the number of pixels
wide by high, 2048 by 1536, by 4
bytes per pixel gets you about
10 megabytes.
So why is it so much larger?
Well, we have to talk about how
images work on iOS.
There's a load, a decode, and a
render phase.
So the load phase takes this
590-kilobyte JPEG file, which is
compressed, loads it into
memory.
The decode converts that JPEG
file into a format that the GPU
can read.
Now, this needs to be
uncompressed, which makes it 10
megabytes.
Once it's been decoded, it can
be rendered at will.
For more information on images
and how to kind of optimize
them, I'd recommend checking out
the Images and Graphics Best
Practice session that was
earlier this week.
Now, 4 bytes per pixel we got by
the SRGB format.
This is typically the most
common format that images in
graphics are.
It's 8 bits per pixel, so you
have 1 byte for red, 1 byte for
green, and 1 byte for blue, and
an alpha component.
However, we can go larger.
iOS hardware can render wide
format.
Now, wide format, to get that
expressive colors, requires 2
bytes per pixel, so we double
the size of our image.
Cameras on the iPhone 7, 8, X,
and the, some of the iPad Pros
are great for capturing this
high-fidelity content.
You can also use it for super
accurate colors for, like,
sports logos and such.
But these are only really useful
on the wide format displays, so
we don't want to use this when
we don't need to.
On the flip side, we can
actually go smaller.
Now, there's a luminance and
alpha 8 format.
This format stores a grayscale
and an alpha value only.
This is typically used in
shaders, so like Metal apps and
stuff.
Not very common in our usage.
We can actually get even
smaller.
We can go down to what we call
the alpha 8 format.
Now, alpha 8 just has 1 channel,
1 byte per pixel.
Very small.
It's 75% smaller than SRGB.
Now, this is great for masks or
text that's monochrome because
we're using 75% less memory.
So if we look at the breakdown,
we can go from 1 byte per pixel
with alpha 8 all the way up to 8
bytes per pixel with wide
format.
There's a huge range.
So what we really need to do is
know how to pick the right
format.
So how do we pick the right
format?
The short answer is don't pick
the format.
Let the format pick you.
If you migrate away from using
the UIGraphics BeginImageContext
WithOptions API, which has been
in iOS since it began, and
instead switch to the UIGraphics
ImageRenderer format, you can
save a lot of memory because the
UIGraphics BeginImageContext
WithOptions is always a
4-byte-per-pixel format.
It's always SRGB.
So you don't get the wide format
if you want it, and you don't
get the 1-byte-per-pixel A8
format if you need it.
Instead, if you use the
UIGraphics ImageRenderer API,
which came in iOS 10, as of iOS
12, it will automatically pick
the best graphics format for
you.
Here's an example.
Say I'm drawing a circle for a
mask.
Now, using the old API with the
highlighted code here is my
drawing code, I'm getting a
4-byte-per-pixel format just to
draw a black circle.
If I instead switch to the new
API, I'm using the exact same
drawing code.
Just using the new API, I'm now
getting a 1-byte-per-pixel
image.
This means that it's 75% less
memory use.
That's a great savings and the
same fidelity.
As an additional bonus, if I
want to use this mask over
again, I can change the tint
color on an image view, and that
will just change it with a
multiply, meaning that I don't
have to allocate any more
memory.
So I can use this not just as a
black circle, but as a blue
circle, red circle, green circle
with no additional memory cost.
It's really cool.
One other thing that we
typically do with images is
downsample them.
So when we want to make like a
thumbnail or something, we want
to scale it down.
What we don't want to do is use
a UIImage for the downscaling.
If we actually use UIImage to
draw, it's a little bit less
performant due to internal
coordinate space transforms.
And, as we saw earlier, it would
decompress the entire image in
the memory.
Instead, there's this ImageIO
framework.
ImageIO can actually downsample
the image, and it will use a
streaming API such that you only
pay the dirty memory cost of the
resulting image.
So this will save you a memory
spike.
As an example, here's some code
where I get a file on disk.
This could also be a file I
downloaded.
And I'm using the UIImage to
draw into a smaller rect.
This is still going to have that
big spike.
Now, instead, if I switch to
ImageIO, I still have to load
the file from disk.
I set up some parameters because
it's a lower-level API to say
how big I want this image to be,
and then I just ask it to create
it with CGImageSource
CreateThumbnail AtIndex.
Now, that CG image I can wrap in
a UIImage, and I'm good to go.
I've got a much smaller image,
and it's about 50% faster than
that previous code.
Now, another thing we want to
talk about is how to optimize
when in the background.
Say I have an image in an app,
full screen.
It's beautiful.
I'm loving it.
But then, I need to go to my
Home screen to take care of a
notification or go to a
different app.
That image is still in memory.
So as a good rule of thumb, we
recommend unloading large
resources you cannot see.
There are 2 ways to do this.
The first is the app life cycle.
So if you background your app or
foreground it, the app life
cycle events are great to, are a
great way to know.
Now, this applies to mostly the
on-screen views because those
don't conform to the
UIViewController appearance life
cycle.
UIViewController methods are
great for, like, tab controllers
or navigation controllers
because you're going to have
multiple view controllers, but
only 1 of them is on screen at
once.
So if you leverage like the
viewWillAppear and
viewDidDisappear code or
callbacks, you can keep your
memory footprint smaller.
Now, as an example, if I
register for the notifications
for the application entering the
background, I can unload my
large assets -- in this case,
images.
When the app comes back to the
foreground, I get a notification
for that.
If I reload my images there, I'm
saving memory when in the
background, and I'm keeping the
same fidelity when the user
comes back.
It's completely transparent to
them, but more memory is
available to the system.
Similarly, if I'm in a nav
controller or a tab controller,
my view controllers can unload
their images when they
disappear.
And before they come back with
the viewWillAppear method, I can
reload them.
So again, the user doesn't
notice anything's different.
Our apps are just using less
memory, which is great.
And now, I'd like to invite up
Kris to kind of bring this all
together in a nice demo.
Kris?
[ Applause ]
>> Okay. I'm going to switch to
the demo machine.
There we go.
So I've been working on this
app.
What it does is it starts with
these really high-resolution
images from our solar system
that I got from NASA, and it
lets you apply different filters
to them.
And here we can see a quick
example, applying a filter to
our Sun.
I'm really pleased with how it's
going so far, so I sent it off
to James to get his opinion on
it, and he sent me back an email
with 2 attachments.
One was a Memgraph file, and the
other one was this image.
Now, James is a pretty reserved
and understated guy, so when
he's got 2 red exclamation
points and a scream emoji, I
know he's pretty upset.
So I went to James and I said,
"You know, I don't understand
what the big deal is.
I clearly have at least half a
gigabyte before I'm even in the
red, you know.
I have all this available
memory.
Shouldn't I be using it?"
And James, who's a much better
developer than I am, pointed out
a few things that's, a few
things that are wrong with my
logic.
First of all, this gauge is
measuring a device with 2
gigabytes of memory.
Not all our devices have that
much memory.
If this code was running on a
device with a, only 1 gigabyte
of memory, there's a good chance
our app would already be
terminated by the operating
system.
Second, the operating system
doesn't just, doesn't use just
how much memory your app is
using when designing when to
terminate your app, but also
what else is going on in the
operating system.
So just because we're not to the
red yet doesn't mean we're not
in danger of being terminated.
And third, this represents a
terrible experience for the
user.
In fact, if you look at the
usage comparison chart, you can
see other processes has zero
kilobytes of memory.
That's because they've all been
jettisoned by the operating
system to make room for our app.
You should all take a good look
at me and give me the stink eye
because now when the user has to
go watch your app, you have to
load from scratch.
So James makes some pretty good
points, so I think, in general,
we want to get this needle as
far to the left as possible
instead of as far to the right.
So let's see what we can do.
Let me go ahead and take a look
at the Memgraph file.
And I have a couple go-to tricks
that I use when working with a
Memgraph file or go-to
strategies.
And the first -- actually, let
me bring this up a little bit --
is to look for leaks.
So if I go down to the Filter
toolbar and click on the leaks
filter, that'll show me just any
leaks that are in my Memgraph
file.
Turns out this Memgraph file has
no leaks.
Well, that's both kind of good
news and bad news.
It's great that there are no
leaks, but now I have to figure
out what's actually going on
here.
The other thing that Memgraph is
really good for is showing me
how many instances of an object
are in memory and if there's
more than I expect.
But if I look at this Memgraph,
I can see if I actually just
focus specifically on the
objects from my code, there's
only 5 in memory, and there's
actually only 1 of each of
these.
If there were, you know,
multiple root view controllers,
or multiple noir filters, or
multiple filters in memory, more
than I expect, that's something
else I could investigate.
Well, there's no more instances
here than I expect, but maybe
one of these is really big.
It's not very likely, but I
might as well check.
So I'm going to go to the memory
inspector.
I'm going to look at these.
Each of them, it lists the size
for each of the objects.
So I can see my app delegate is
32 bytes.
The data view controller is
1500.
As I go through each of these,
none of these are clearly
responsible for the, you know, 1
plus gigabytes of memory my app
is using.
So that's it for my bag of
tricks in dealing with Memgraph
in Xcode.
Where do I go now?
Well, I just watched this great
WWDC session about using
command-line tools in Memgraph
files.
So let me see if I can find
anything by trying that.
And thinking back, the first
thing James suggested was using
vmmap with the summary flag.
So let me give that a try, and
let me pass in my Memgraph file.
And let's take a look at this
output.
So now, what should I be looking
for in here?
Now, in general, I'm looking for
really big numbers.
I'm trying to figure out what's
using all this memory, and the
bigger numbers mean more memory
use.
Now, there's a number of columns
here, and, you know, some of
them are more important than
others.
First of all, virtual size, I
mean, virtual means not real.
I can almost practically ignore
this column.
It's memory that the app has
requested but isn't necessarily
using.
Dirty sounds like something I
definitely don't want in my app.
I'd much rather my app be clean
than dirty, so that's probably
something I want smaller.
And then, swapped, which,
because this is iOS, is
compressed, remembering back to
both Kyle and James pointed out
that it's the dirty size plus
the compressed size that the
operating system uses to
determine how much memory my app
is really using.
So those are the two columns I
really want to concentrate on,
so let's look for some big
numbers there.
I can see immediately CG image
jumps out.
It has a very big dirty size and
a very big swapped size.
That's a giant red flag, but
let's keep looking.
I can see this is IOSurface has
a pretty big dirty size but no
swapped size.
MALLOC LARGE has a big dirty
size but a really small or
smaller swapped size.
And there's nothing else in here
that's really that big.
So I think, based on what I see
here, I really want to
concentrate on the CG image VM
regions.
So I'm going to go ahead and
copy that.
So what's the next step?
Well, we want to know more about
some virtual memory, so vmmap
seems like the place to go
again.
This time, instead of the
summary flag, I'm just going to
pass my Memgraph file.
But I really only care about the
CG image memory.
I don't care about all the other
virtual memory regions that
vmmap will tell me about.
So I'm going to go ahead and use
grep to just show me the lines
that talk about CG image.
So let's see what that looks
like.
So now, I have three lines.
I see there's two virtual memory
regions.
There, and I see their start
address and their end address.
And then, I can see these are
the same columns as above.
This is virtual, resident,
dirty, and compressed.
And this last line here is
actually the summary line again.
So that's the same data that was
above.
Looking at my two regions, I can
see I have a really small region
and a really big region.
That big region is clearly what
I want to know more about.
So how can I find out more about
that particular VM region?
Well, I went looking through the
documentation from vmmap, and I
noticed it has this verbose
flag, which, as the name
implies, outputs a lot more
information.
And I wonder what that can tell
me.
So let's go ahead and pass
verbose and the Memgraph file.
And again, I only care about CG
image regions, so I want to use
grep to filter to just those.
Oh, now I see there's actually a
lot more regions.
What's going on here?
Well, it turns out that vmmap,
by default, if it finds
contiguous regions, it collapses
them together.
And in fact, if you look
starting on the second line
here, the end address of this
region is the same as the
starting address of this one.
And that pattern continues all
the way down.
So vmmap, by default, collapses
those into a single region.
But looking at the details here,
I can see there are actually
some differences.
And in particular, some of these
use a lot more dirty memory and
some have a lot more compressed
memory, so this gives me an idea
of maybe something I want to
focus on.
But I'm actually going to use a
different strategy here.
I know that the operating
system, not necessarily, but as
a general rule, the later the VM
region was created, the later in
my app's life cycle it happened.
And since this Memgraph was
taken during this huge spike in
memory use, chances are these
later regions are more closely
tied to whatever caused that
spike.
So instead of trying to find the
one with the biggest dirty and
compressed size, I'm going to go
ahead and just start at the end
here.
I'm going to grab the beginning
address of that final region.
Now, where do I go from here?
Well, one of the tools that
James mentioned was heap, but
that's about objects on the
heap, and I'm dealing with a
virtual memory region, so that
doesn't help.
Then, there's leaks, but leaks,
I don't have a leak here.
I already know from looking at
the Memgraph there's no leaks,
so that doesn't seem like the
tool I want to use.
But I went looking through the
help information for leaks, and
it turns out leaks can do lots
of things and including telling
me who has references to either
an object on the heap or virtual
memory region.
So let's go ahead and see what
that tells us.
So I'm going to use leaks, and
then I'm going to pass this
traceTree flag.
And what that does is it gives
me a tree view of everything
that has a reference to the
address I'm passing in.
In this case, I'm passing in the
starting address of my virtual
memory region.
And then, finally, we give it
the Memgraph file.
So what does this look like?
So what we see here is this tree
of all these references.
If we scroll up to the top,
which is way up here, I can
actually see here's my VM
region, here's my CG image
region, and then I can see
there's a tree view here of all
the things that have references,
and what references them, and
what references them, and so on
and so forth.
And in fact, if we go back to
Xcode, and we actually filter on
the same address, and I go ahead
and look at this object, this
tree is the exact same tree I
see from leaks.
And if I wanted to, I could go
through and expand every single
one of these nodes and look at
the details for each of them,
but that's going to take a
while, and it's kind of tedious.
The nice thing about the leaks
output is not only can I kind of
quickly scan through it, if I
want to, I can search or filter
it, or I can put it into a bug
report or an email, which I
can't really do with the
graphical view that's in Xcode.
So what am I looking for here in
this output?
Well, ideally, I would find
something, a class that I'm
responsible for, a class from my
application.
I happen to have looked through
this already, and I know there's
none of my classes in here, so
what's the next best thing?
Well, a class that I know I'm
creating, like a framework
class, that's either being
created on my behalf or that I'm
directly creating.
So I know that, you know, my app
has UI views.
It has UI images.
And I'm using these core image
classes to do the filtering.
And so if we go ahead and we
look through here, and I'm using
a very sophisticated debugging
tool called my eyeballs.
And we go ahead and we look for
-- let me see if I can find what
I want.
It's a very big terminal output.
Makes it a little more
confusing.
Well, so, for example, you know,
here's a font reference, and I
know, you know, my application
uses fonts, but chances are the
fonts aren't responsible for a
lot of my memory use, so that's
not going to help.
If we go down further, I can
actually see there's a number of
these CI classes, and those are
the core image filters, or
that's core image, you know,
classes it's creating to do the
filtering work in my
application.
So that may be something I want
to investigate further as well.
I happen to have already done
that and haven't found anything
useful.
So I can't really get anywhere
further looking at the leaks
output, which is unfortunate.
So what should I go to next?
Fortunately, James had
memory-backed trace recording,
allocation-backed trace
recording turned on when he
captured this Memgraph, which
means I can use the other tool
he talked about to look at the
creation backtrace of my object.
So I'm going to use malloc
history.
And this time, I pass it, the
Memgraph file, first.
And then, I'm going to pass it
from the help documentation,
this fullStacks flag.
And what that does is it prints
out each frame on its own line,
makes it a lot more human
readable.
And then, I'm going to pass it
the starting memory address of
my VM region.
Let's see what this looks like.
Well, this actually is not that
big of a backtrace, and I can
see actually my code appears on,
here on several lines.
Lines 6 through 9 actually come
straight from my application
code, and I can see here on line
6 that my NoirFilter apply
function is what is responsible
for creating this particular VM
region.
So that's pretty good smoking
gun as to where I want to look
in my app for who's creating all
this memory.
And in fact, if we go back to
the Memgraph file, I can
actually see that's the same
backtrace that appears in Xcode
here.
You can actually see right here
is also the NoirFilter apply
method.
We don't get the nice
highlighting you normally see in
the backtrace view here because
we're not debugging a live
process.
We're loading a Memgraph file.
But you can see it's the exact
same output as we get from
malloc history.
And in fact, to just kind of a
confirm things even further, if
I go ahead and I look at my full
list of CG image VM regions and
I collect, I grab the second one
from the bottom, the next one
up, and let's look at the
backtrace for that one.
And it turns out it's the same
backtrace.
So the same code path is
responsible for that region as
well.
And in fact, looking at several
of those regions, it actually
uses the same backtrace.
So now, I have a really good
idea of what in my application
is responsible for creating
these VM regions that are using
up a whole bunch of the memory
in my application.
So what can we do about it?
Well, let's go back to Xcode,
and I can go ahead and close the
Memgraph file.
And the first thing I want to do
is let's take a look at the code
here.
If I look at my filter, I can
see right here is the apply
function, and I can actually see
right away something jumps out
at me, which is I'm using the
UIGraphicsBegin ImageContext
WithOptions and the
UIGraphicsEnd ImageContext,
which I remember Kyle said you
shouldn't be using.
There's a better API to use in
those circumstances.
So that's something I definitely
want to come back to, but the
first thing I need is I need
some kind of baseline.
I need to get an idea of how
much memory my app is using so I
can make sure my changes are
making a difference.
So I'm going to go ahead and run
the application, and I'm going
to go to the debug navigator and
look at the memory report.
So now, I can see the memory my
app is using as I run it.
Now, I really like this image of
Saturn's north pole.
It's this weird hexagon shape,
which is both kind of cool and a
little freaky.
So let's take a look at that,
and let's apply the filter and
see what we get.
So 1 gig, 3 gigs, 4 gigs, 6
gigs, 7 gigs.
This is bad.
And actually, this actually
brings up a good point, which is
this would not fly on a device
at all.
So when you're running in the
simulator, you have to remember
that it's useful for debugging
and testing changes, but you
need to validate all that stuff
on devices as well.
But the other thing that's nice
is the simulator is never going
to run out of memory.
So if I have a case where my app
is getting shut down on a
device, maybe try it in the
simulator.
I could see what's, you know, I
can wait for a really big
allocation, not get shut down,
and then investigate it from
there.
And one thing I would like to
point out is actually we do show
you the high-water mark over
here.
And in this case, I'm up to 7.7
gigabytes.
It's terrible.
So let's see what we can do
about that.
I'm going to go back to my apply
function.
And now, you know, I do want to
come back to this
beginImageContext WithOptions
thing, but thinking back to what
Kyle said, when you're dealing
with images, what's the most
important thing in terms of
memory use?
It's the image size, so let's
take a look at what that looks
like.
I'm going to go ahead and apply
the filter again.
And then, once I'm stopped in
the debugger, I want to go ahead
and see the size of this image.
And I'm actually just going to
take a sip of water before I hit
return here.
Actually, I'm not going to have
any water.
This is 15,000 by 13,000.
Now, I checked in the
documentation.
On UIImage that size, that's
points, not pixels.
If this is a 2X device or a 3X
device, you have to multiply
that by a big number.
You know, Kyle was upset because
an image was taking 10
megabytes.
Nobody tell him about this.
And in fact, just to confirm
things, I want to try this.
I'm going to do a, the 15,000
times 13,000, and the iPhone X
is a 3X device, so it's 3 times
the width times 3 times the
height times 4 bytes per pixel.
That number looks kind of
familiar.
So I'm pretty sure I know
exactly what's using up my
7-and-a-half gigabytes of
memory, and it's not necessarily
my beginImageContext thing.
It's the size of this image.
And there's no reason the image
needs to be this big.
What I want to do is scale it
down so it's the same dimensions
as my view.
And that way, it'll take up far
less memory.
So if I go back to the image
loading code that's up here --
actually, before I do that, I
want to go ahead and disable
this break point -- so let's
take a look at what this does.
Well, it's pretty
straightforward.
It's getting the URL from a
bundle, it's loading some data
from that URL and loading it
into UIImage, then that, then,
which gets passed to the filter.
So what I want to do is, before
I send it to the filter, I want
to scale down the image.
However, I remember what Kyle
said.
I don't want to do the scaling
on UIImage because it still ends
up just loading that whole image
into memory anyway, which is
what I'm trying to avoid.
So I'm going to go ahead and
let's collapse this function.
And I'm going to replace it with
the code Kyle suggested.
Okay, so let's take a look at
what this code is doing.
So here again, we're getting the
image from the bundle, but now,
this time -- just a little
lighter -- I'm calling
CGImageSource CreateWithURL to
get a reference to the image and
then passing that to
CGImageSource CreateThumbnail
AtIndex.
So now, I can scale the image to
the size I want without having
to load the whole thing into
memory.
Let's give this a shot and see
if it makes a difference.
I'm going to go ahead and
rebuild and wait for it to
launch on the application.
And then, once it's there -- oh,
there's a warning.
I have an extra this.
Let's see.
Okay, building.
Building, building, building.
Okay, launching.
Good. All right.
Now, let's go ahead and take a
look at the memory report.
Let's go back to Saturn's north
pole, which is something I've
always wanted to say.
And let's apply our image and
see what it goes to now.
So now, we're at 75, 93
megabytes.
Our high-water mark in this case
is 93 megabytes.
Significant improvement.
[applause] Much better than the
almost guaranteed to get shut
down 7-and-a-half gigabytes.
But now, I remember there's
actually, I want to go back, and
let's go ahead and stop.
And I still want to go back to
my filter method and change this
UIBeginImageContext code and do
what Kyle suggested here.
So I'm going to go ahead and
delete this code and add in my
new filter.
So now, in this case, I'm going
ahead and creating a UIGraphics
ImageRenderer.
And I'm using my CI filter
within the renderer to do the
filter, apply the filter.
Let's go ahead and run this --
hopefully, it'll build -- and
see if this happens to make any
difference in terms of my memory
usage.
So let's go back to the debug
navigator and to the memory
report.
And once again, we get to go
back to Saturn, and let's go
ahead and apply our filter.
Now, let's see what our
high-water mark ends up this
time.
Ninety-eight.
So that's actually the same, but
it turns out that's, if you
think about it, that's what I
expect.
My image is still going to be
using, in this circumstance, 4
bytes per pixel, so I'm not
actually getting any memory
savings by using this new
method.
However, if there was an
opportunity for memory savings
-- for example, you know, if the
operating system could determine
that it could use fewer bytes
per pixel or if it determined
that it needed to use more, it
would do the right thing, and I
don't need to worry about it.
So even though I don't see a big
improvement, I know my code is
still better for having made
these changes.
So there's still more I could do
here, right.
I want to make sure we unload
the image when the app goes into
the background, and I want to
make sure we're not showing any
images in views that aren't on
screen.
There's a lot more I can do
here, but I'm really pleased
with these results, and I want
to send them back to James.
So I'm going to go ahead and
grab a screenshot and add a
little note to it for James just
to show him how pleased I am
with all of this.
And I think we're going to go
ahead and send him the
starry-eyed emoji.
And hopefully James will be
happy with these results.
So now, I'm going to hand it
back to Kyle, who's going to
wrap things up for us.
Thank you.
[ Applause ]
>> Thanks, Kris.
Thanks, Kris.
That was awesome.
With just a little bit of work,
we were able to greatly reduce
our memory use by orders of
magnitude.
So in summary, memory is finite
and shared.
The more we use, the less the
system has for others to use it.
We really need to be good
citizens, and be mindful of our
memory use, and only use what we
need.
When we're debugging, that
memory report in Xcode is
crucial.
We can just turn it on when our
app is running because then, as
we monitor it, the more we can
notice the regressions as we're
debugging.
We want to make sure that iOS
picks our image formats for us.
We can save 75% memory from SRGB
to alpha 8 just by picking the
or by using the new UIImage
GraphicsRenderer APIs.
It's really great for masks and
text.
Also, we want to use ImageIO
when we downsample our images.
It provides us or prevents a
memory spike, and it's faster
than trying to draw a UIImage
into a smaller context.
Finally, we want to unload large
images and resources that are
not on the screen.
There's no sense in using that
memory because the user can't
see them.
And even after all of that,
we're still not done.
As we just saw, using Memgraphs
can help us further understand
what's going on and reduce our
memory footprint.
That combined with malloc
history gives us great insight
into where our memory's going
and what it's being used by.
So what I'd recommend is
everyone go out, turn on malloc
history, profile your tool, and
start digging in.
So for more information, you can
go to our slide presentation.
And also, we'll be down in the
technology labs shortly after
this for a little bit if you
have additional questions for
us.
Thanks, and have a great
remainder of WWDC.
[ Applause ]