Transcript
[ Music ]
>> Good morning.
[ Applause ]
>> Welcome to Session 403,
What's New in Testing.
My name is Honza Dvorsky and I
will share the stage today with
my colleague, Ethan Vaughan.
Today, we'll start by talking
about the code's coverage
improvements we made in Xcode
9.3.
Then we'll look at the test
selection and ordering features
new in Xcode 10.
And finally, Ethan will come up
on stage to tell you about how
you can speed up your tests by
running them in parallel.
Let's kick off with code
coverage.
We completely reworked code
coverage in Xcode 9.3 and this
resulted in many performance and
accuracy improvements and it
allowed us to add a new feature
so that you have the granular
control over which targets
contribute to code coverage.
We created a new command line
tool called xccov and last but
not least, we gave code coverage
in the source editor a visual
refresh.
Let's look at all of these in
detail.
First, to give you a sense of
how much better code coverage is
now, we measured it on a large
internal project at Apple.
To understand the speed, we
measured how long it took Xcode
to load and show code coverage
in the source editor.
In Xcode 9, it took about six
and a half seconds to do that.
In Xcode 9.3, however, we got it
down to less than a half a
second, which is more than 95%
faster.
[ Applause ]
But we also wanted to make the
coverage file smaller, as Xcode
can potentially write out many
of them, if you run your test
often, as you should, so I'm
happy to say that here, the
improvements are just as
dramatic.
Xcode 9's coverage files were
over 200 megabytes in size.
And this is a lot but remember
that we're talking about a large
project with thousands of source
files.
But files written by Xcode 9.3
were less than a tenth of that.
[ Applause ]
I'm sure you appreciate this if
you maintain a continuous
integration machine or if you're
just running low on disk space.
But the best part is that not
only are the coverage files
smaller and faster to read and
write, they're also more
accurate than ever before.
One example of this is header
files.
Xcode 9 didn't correctly collect
and show coverage in header
files and this was a problem for
a code basis that used C++ as in
that language, you can have a
good portion of your executable
code in headers.
So, if you're one of the people
affected, you'll be happy to
hear that Xcode now correctly
gathers and presents code
coverage for both implementation
and header files.
Now let's talk about code
coverage features.
The first one is target
selection.
This is a new option to control
not only whether code coverage
is enabled or disabled but when
it's enabled, you actually
control which targets are
included.
This can be important if your
project builds third-party
dependencies that you're not
expected to cover by your tests
or if you work at a company
where another team supplies you
with a framework that they
already tested.
You can customize the included
targets in this scheme's test
action under options.
This means that it can continue
to include all targets in code
coverage or only hand pick some
of them.
Now in the name of more powerful
workflows, we created a new
command-line tool called xccov.
It can be integrated into
automation scripts easily, as it
produces both human readable and
machine parseable outputs.
And at the high level, it gives
you detailed view of your
coverage data.
So, I've mentioned the code
coverage data a couple of times,
let's look at how it looks under
the hood.
When tests are run with code
coverage enabled, Xcode
generates two files.
First is the coverage report, or
the xccovreport file extension,
and that contains the line --
line coverage percentages for
each target, source file, and
function.
The second one is the coverage
archive and that contains the
raw execution counts for each of
the files in the report.
And these coverage files live in
your project's derived data
directory and additionally, if
you pass the result bundle path
flag to Xcode build, these files
will be placed into the result
bundle as well.
Let's look at an example.
Here, we use xccov to view the
coverage data of our Hello World
app.
We can see the overall coverage
for each target but also
detailed coverage for each file
and even for each method.
And of course, by passing the
right flag identification, you
can get the same exact
information in JSON.
This can make it much easier to
integrate with other tools.
So, as you can see, xccov is
very flexible, so I would
encourage you to explore its
documentation.
Now we talked about use --
viewing code coverage on the
command line but the most
convenient way is still going to
be to see it right next to your
source code.
You control whether code
coverage is shown by selecting
editor, show, or hide code
coverage.
When you do enable it, you'll
see the refreshed look of code
coverage in the source editor.
The whole line highlights when
you hover over the execution
count on the right-hand side.
And this is just the best way to
keep an eye on which of your
code is already covered by tests
and which still needs more work.
Now let me show you all of these
improvements in action.
So, here I have a project called
Dev Cast.
It's a simple messaging app and
I created the iOS version for
this talk last year.
So, this year, I wanted to
create a Mac version.
But I wanted to share the
business logic between the iOS
and Mac versions, so I put it
all into a framework called
DevCastkit and today, my goal is
to explore and maybe improve the
code coverage of this framework.
So, right now, I'm not
collecting code coverage, so I
just need to turn it on.
I'll do that by selecting the
scheme, selecting edit scheme,
go into the test action, go into
options.
And down here, I will just
enable code coverage.
Now I will run my test by going
to product test.
Right now, my framework, my test
bundle and my Mac app are
getting built, and my tests are
being run.
Now the test finished so let's
look at the results.
We'll go to the report navigator
and to the latest coverage
reports.
Here, I can see that I have the
-- both targets, the Mac app and
also the framework.
Now the Mac app only serves as a
container for running tests, so
I'm not really concerned about
its low code coverage.
In fact, I would actually not
want to see it in this report at
all.
So, I can do that by editing the
scheme again.
And in the test action, here --
instead of gathering coverage
for all targets, I'll change to
just some targets.
This gives me a hard cover list
and I will add just the target
that I'm interested in, which is
the framework.
Now I will rerun my tests and
look at the updated coverage
reports.
So, now you can see that I only
have the framework in the
coverage report.
This is what I wanted.
So, now I can just focus on this
one target.
So, looking at the coverage
percentage, is at 84%, which is
not bad, but I know I can do
better.
So, to understand which of the
files need more attention, I
will disclose the target and see
that the first file is only
covered at about 66%.
So, I'll jump to the file by
clicking on the arrow here.
And here, I'll look at my local
server class.
On the right-hand side, I can
see the execution counts for
each region of the code so I can
see that all of the executable
parts of my class are covered by
tests.
This is good.
Unfortunately, my convenience
function, get recent messages,
hasn't been called for my tests
at all, so I don't know if it's
working properly.
To fix that, I'll go to the
corresponding test file and I
will add a new test.
This test just puts a couple of
messages into a local server and
then I call, get recent
messages, on it to verify that
it returns what I'm expecting.
So, with the new test added, I
will rerun my tests one more
time.
So, great, the test finished.
I'll go to the coverage report
again and now when I focus on my
framework, I can see that it's
covered at 100%.
This is exactly what I wanted.
So, we saw how I used the target
selection feature to only focus
on some of the targets.
Then I used the coverage report
to know exactly which file to
focus on.
And finally, I used the code
coverage integration in the
source editor to know exactly
which part of my code still
needs covering.
So, this is the improved code
coverage in Xcode.
So, moving on from code
coverage, let's talk about some
new features in Xcode 10.
First, we'll see how we can now
better select and order our
tests.
Now why is this important?
Well, not all the tests in your
suite serve the same purpose.
You might want to run all 1000
of your quick running unit tests
before every single commit but
only run your 10 long-running UI
tests at night.
And you can achieve this today
by disabling specific tests in
your scheme.
The scheme encodes the list of
disabled tests so that XE test
knows which tests to skip.
And this has an interesting side
effect.
Whenever you write a new test,
it's automatically added to all
the schemes that contain the
corresponding test targets.
But if that's not what you
wanted, you have to go through
all those schemes and disable
the test there manually.
So, in Xcode 10, we're
introducing a new mode for
schemes to instead, encode the
tests to run.
If you switch your scheme to
that mode, only the test that
you hand pick will run in that
scheme.
You control this mode in the
scheme editors test action,
where in the list of test
targets, there's a new options
pop-up, where you can declare
whether the scheme should
automatically include new tests
or not.
This way, some of your schemes
can continue to run any new
tests you write, while other
schemes will only run a list of
hand-picked tests.
So, we've discussed how we can
better control which of our
tests run and when, but the
order of our tests can be
significant too.
By default, tests in Xcode are
sorted by their name.
This means that unless you
rename your tests, they will
always run in the same order.
But determinism can be a
double-edged sword.
It can make it easy to miss
bugs, where one of your tests
implicitly depends on another
one running before it.
Let's look at an example of when
this can happen.
Imagine you have tests A, B, and
C.
They always run in this order
and they always pass.
But when you look at your tests
in detail, you realize that test
A creates a database.
Then test B goes and writes some
data into it.
And then finally, test C goes
and deletes it.
Now these tests only pass
because they run in this
specific order.
But if you tried to shuffle them
around, for example, by renaming
them, then you try to run them
again, you might have test B
writing into a database that
doesn't exist and your tests
would just fail.
So, to prevent issues like this,
your tests should always
correctly set up and tear down
their own state.
Not only will they be more
reliable, but it will also make
it possible for your tests to
run independently of all the
other ones and this can be
really beneficial during
development and debugging.
So, to help you ensure that
there are no unintentional
dependencies between your tests,
in Xcode 10, we're introducing
the new test randomization mode.
If you turn it on, your tests
will be randomly shuffled before
every time they're run.
And if your tests still pass
with this mode on, you can be
more confident that they are
really robust and
self-contained.
Randomization mode can be
enabled in the scheme editor,
just like the other features
we've seen today.
So, these are the new test
selection and ordering features
in Xcode 10.
Now I'm really excited about
what comes next.
To tell you all about the new
parallel testing features in
Xcode, I would like to welcome
Ethan Vaughan to the stage.
Ethan.
[ Applause ]
>> Thanks, Honza.
So, many of you have a
development cycle that looks
like this.
You write some code, you debug
it, and then you run your tests
before you commit and push your
changes to the repository.
By running your tests before you
push, you can catch regressions
before they ever make it into a
build.
However, one of the bottlenecks
in this process can be just how
long it takes to run your tests.
Some of you have test suites
that take on the order of 30
minutes to hours to run.
And if you have to wait that
long before you can confidently
land your work, that represents
a serious bottleneck in your
workflow.
We want to make sure that your
tests run as fast as possible in
order to shorten this critical
part of the development cycle.
So, last year, we introduced a
feature in Xcode 9 to help you
run tests faster, it's called
Parallel Destination Testing.
This is the ability to run all
of your tests on multiple
destinations at the same time.
And you do this from the command
line by passing multiple
destinations specifiers to
xcodebuild.
Previously, if you were to run
your tests, let's say on an
iPhone X and an iPad, xcodebuild
would run all of the tests on
the iPhone X and then all of the
tests on the iPad.
Neither device would be running
tests at the same time.
But in Xcode 9, we changed this
behavior so that by default,
tests run concurrently on the
devices and this can
dramatically shorten the overall
execution time, which is great.
However, there are some
limitations to this approach.
First, it's only beneficial if
you test on multiple
destinations.
If you just want to run unit
tests for your Mac app, for
example, this doesn't help.
Also, it's only available from
xcodebuild, so it's primarily
useful in the context of a
continuous integration
environment, like Xcode Server
or Jenkins.
I'm excited to tell you about a
new way to run tests faster than
ever called Parallel Distributed
Testing.
With Parallel Distributed
Testing, you can execute tests
in parallel on a single
destination.
Previously, testing on a single
destination looks like this, a
continuous straight line with
one test executing after the
other.
Parallel Distributed Testing
allows you to run tests
simultaneously so that testing
now looks like this.
In addition, it's supported both
from Xcode, as well as
xcodebuild, so no matter where
you run your tests, you'll get
the best performance.
Now in order to tell you about
how Xcode runs your tests in
parallel, we first have to talk
about how your tests execute at
all, what happens at runtime.
Let's start with unit tests.
Your unit tests get compiled
into a test bundle.
At runtime, Xcode launches an
instance of your app which
serves as a test runner.
The runner loads the test bundle
and executes all of its tests,
so that's how unit tests
execute.
What about UI tests?
For UI, tests the story is
similar.
Your tests still get compiled
into a bundle, but the bundle is
loaded by a custom app that
Xcode creates.
Your app no longer runs the
tests.
Instead, the tests automate your
app by launching it and
interacting with different parts
of its UI.
If you'd like to learn more
about this process, I'd
encourage you to check out our
session from 2016, where we go
into even more detail.
So, now that we understand how
our tests execute, we can
finally talk about how Xcode
runs them in parallel.
Just like before, Xcode will
launch a test runner to execute
our tests but instead of just
launching a single runner, Xcode
will launch multiple runners,
each of which executes a subset
of the tests.
In fact, Xcode will dynamically
distribute tests to the runners
in order to get the best
utilization of the course on
your machine.
Let's go into more detail.
When Xcode distributes tests to
the runners, it does so by
class.
Each runner receives a test
class to execute and it'll
execute that test class before
going on to execute another one.
And then testing finishes once
all the classes have executed.
Now you might be wondering why
Xcode distributes tests by class
instead of distributing
individual test methods to the
runners.
There are a couple reasons for
this.
First, there may be hidden
dependencies between the tests
in a class, like Honza talked
about earlier.
If Xcode were to take the tests
in a class and distribute them
to different runners, it could
lead to hard to diagnose test
failures.
Second, each test class has a
class level set up and tear down
method, which may perform
expensive computation.
By limiting the tests in a class
to a single runner, XE tests
only has to invoke these methods
once, which can save precious
time.
Now I'd like to talk about some
specifics to parallel testing on
the simulator.
When you run tests in parallel
on the simulator, Xcode starts
by taking the simulator that
you've selected and creating
multiple distinct copies or
clones of it.
These clones are identical to
the original simulator at the
time that they're created.
And Xcode will automatically
create and delete these clones,
as necessary.
After cloning the simulator
several times, Xcode then
launches a test runner on each
clone and that runner then
begins executing a test class,
like we talked about earlier.
Now the fact that your tests
execute on different clones of
the simulator has some
implications that you should be
aware of.
First, the original simulator is
not used during testing.
Instead, it serves as a sort of
template simulator.
You can configure it with the
settings and the content that
you want and then that content
gets copied over to the clones
when they're created.
Next, there will be multiple
copies of your app, one per
clone, and each copy has its own
data container.
This means that if you have a
test class that modifies files
on disk, you can't expect those
file modifications to be visible
to another test class because it
could have access to a
completely separate data
container.
In practice, the fact that class
is executed on different clones
will likely be invisible to your
tests, but it's something to be
aware of.
So, where can you run tests in
parallel?
You can run unit tests in
parallel on macOS, as well as
unit and UI tests in parallel on
the iOS and tvOS simulators.
And with that, I'd like to give
you a demo of Parallel
Distributed Testing in action.
All right, so this is the Solar
System app, which you may have
seen at some of the other
sessions here at WWDC.
As one of the developers of this
app, I want to run all of my
tests before committing and
pushing my changes to the
repository.
However, as I add more and more
tests, this is starting to take
longer and longer, and it's
becoming a bottleneck in my
workflow.
So, let's see how parallel
testing can help.
I'll go ahead and switch over to
the Xcode project and I already
ran my tests and I want to see
how long they took to run.
All right, let's go back to the
test report.
So, here, you can see next to
each method, we show exactly how
long that method took to
execute.
Next, for each test class, we
show the percentage of passing
and failing tests, as well as
how long the test class took to
execute.
And finally, in the top right
corner, you can see exactly how
long all of your tests took to
run.
So, here, we can see that our
tests took 14 seconds.
Now I'd like to enable
parallelization, which I can do
by going to the scheme.
So, I'll select the scheme and
choose edit scheme.
Next, I'll click the test action
and I'll click the options
button next to my test target.
Finally, I'll select the execute
in parallel checkbox and that's
all I need to do to enable
parallel testing.
So, let's go ahead and run our
tests now by choosing product
tests.
So, I want you to look at the
doc.
Xcode launches multiple copies
of our Mac app to run our unit
tests in parallel.
So, great, it looks like that
finished.
So, now, let's go back to the
report and select the most
recent report.
So, whereas previously, our test
took 14 seconds to run, now they
only take 5 seconds.
So, just by enabling
parallelization, we've seen the
greater than 50% improvement in
test execution time.
[ Applause ]
So, the Solar System app isn't
just available for the Mac, it
also has an iOS companion.
And one of my responsibilities
was to write a UI test suite to
exercise the various screens of
my iOS app.
Now I already enabled
parallelization for the iOS
scheme, so I'll go ahead and
just switch to that now.
And then I'll run tests by
choosing product, test.
Now we'll go ahead and switch
over to the simulator.
So, here, you can see that Xcode
has created multiple clones of
the simulator that I selected,
and these clones are named after
the original simulator so that
they're easy to identify.
And on each simulator, Xcode has
launched a test runner and each
runner is executing a different
test class in my suite.
Now while my tests are
executing, I'm going to switch
back to Xcode and show you the
test log.
You can find the test log
associated with the test report.
The log is a great place to see
how your classes are being
distributed to the different
runners.
You can see an entry in the log
for each runner and beneath a
given runner, you can see the
exact test class that it is
currently executing.
So, when the tests are
completely finished, this is a
great place to see how your
classes were distributed, which
gives you a picture of the
overall parallelization.
And with that, let's switch back
to the slides.
[ Applause ]
So, let's briefly recap what we
learned in the demo.
First, we saw how to enable
parallelization in the scheme
editor.
Next, we saw how to view results
in the test report, as well as
how our classes are distributed
in the test log.
Then we saw Xcode launch
multiple instances of our Mac
app to run unit tests in
parallel.
And finally, we saw multiple
clones of the simulator running
our UI tests in parallel.
Like I mentioned earlier,
xcodebuild has great support for
parallel testing as well.
We've added some new command
line options that allow you to
control this behavior and I'd
like to point out two of those
now.
First, we have
parallel-testing-worker-count,
which allows you to control the
exact number of workers or
runners that Xcode should launch
during parallel testing.
Normally, Xcode tries to
determine an optimal number of
runners based off of the
resources of your machine, as
well as the workload.
This means that on a higher core
machine, you will likely
experience more runners.
But if you find that the default
number isn't working well for
you, you can override it using
this command line option.
Next, we have
parallel-testing-enabled, which
allows you to override the
setting in the scheme to
explicitly turn parallel testing
on or off.
Now for the most part, all you
need to do to get the benefits
of parallel testing is just to
turn it on.
But here are few tips and tricks
to help you get the most out of
this feature.
First, consider splitting a
long-running test class into two
classes.
Because the test classes execute
in parallel, testing will never
run faster than the longest
running class.
When you run tests in parallel,
you may notice a situation like
this, where one test class
dominates the overall execution
time.
If you take that class and
divide it up, Xcode can more
evenly distribute the work of
test execution between the
different runners, which can
shorten the overall execution
time.
Now don't feel like you need to
go and take all of your classes
and divide them up.
That shouldn't be necessary but
if you notice a bottleneck like
this, this is something to try.
Next, put performance tests into
their own bundle with
parallelization disabled.
This may seem counterintuitive,
but performance tests are very
sensitive to system activity so
if you run them in parallel with
each other, they'll likely fail
to meet their baselines.
And finally, understand which
tests are not safe for
parallelization.
Most tests will be fine when you
run them in parallel, but if
your tests access a shared
system resource, like a file or
a database, you may need to
introduce explicit
synchronization to allow them to
run concurrently.
Speaking of testing tips and
tricks, if you'd like to learn
more about how to test your
code, I'd encourage you to check
out our session on Friday hosted
by my colleague Stuart and Brian
[assumed spellings].
You won't want to miss it.
To wrap up, we started today's
talk with code coverage and the
new improvements to performance
and accuracy.
Next, we talked about new test
selection and ordering features,
which allow you to control
exactly which tests run, as well
as the order in which they're
run.
And finally, we talked about
Parallel Distributed Testing,
which allows you to distribute
test classes between different
runners to execute them in
parallel.
For more information, I'd
encourage you to download our
slides from developer.apple.com
and come see us in the labs
later this afternoon.
Have a great WWDC, everyone.
[ Applause ]