Transcript
[ Music ]
[ Applause ]
>> Good morning, and welcome to
Testing in Xcode.
My name is Ana Calinov and I'll
be presenting along with my
colleagues, Stuart Montgomery
and Ethan Vaughan.
In today's session, we'll start
with an introduction to testing
in XCode with XCTest.
Then, Stuart will tell you about
the test plans feature.
Finally, Ethan will show you
applications of XCTest with
continuous integration.
Let's get started with XCTest.
XCTest is the automation testing
framework provided in Xcode with
built-in support to help you set
up and execute your tests.
Testing is an important step of
developing any project which can
help you find bugs in your
source code.
You can also use test to codify
requirements, meaning you make
test for expected behaviors for
your app, and further work by
you and your team can be
qualified against these
expectations.
We're going to start with a
summary of how you should
consider planning out the
automation test suite of any
project.
The pyramid model approach to
testing helps to strike a
balance between thoroughness,
quality and execution speed.
Unit tests are the foundation of
our pyramid.
Unit tests help verify a single
piece of code, generally a
function.
This is done by inputting
variables to the function and
checking that they return the
expected output.
Unit tests are short, simple,
and run very quickly.
This is the foundation of all of
our testing so you want to write
many unit tests to cover all
your functions.
Next, we have integration tests.
Integration tests are used to
validate a larger section of
your code.
These tests should target
discrete subsystems or clusters
of classes to make sure that
different components behave
correctly together.
Integration tests sit on top of
unit tests, because you want to
make sure that the individual
functions behave correctly
before testing this larger piece
of your code.
You generally also won't need as
many integration tests as unit
test.
They may take slightly longer to
run but they do test more of
your app at once.
Lastly, user interface or UI
tests observe the user-phasing
behavior of your app.
This makes sure that your app
truly does what you expect it
to.
UI tests take the longest to run
but they're vital to
demonstrating that everything
behaves correctly.
UI tests also require more
maintenance because your app's
UI may change more frequently.
The full pyramid of tests can
therefore help you balance
between these three different
test types and ensures your test
suite gives you the coverage
that you need.
So we just went over how to
balance your test suite, so now
let's switch to the tools
provided by XCTest to help you
implement it.
Unit tests in XCTest are all of
your tests targeting your source
code.
This includes both your standard
unit test and also your
integration tests.
UI tests execute on top of your
app's UI to provide end-to-end
qualification of your app.
UI tests are also black box test
because they won't rely on any
knowledge of the functions or
classes actually supporting your
app.
UI tests will be able to make
sure that everything behaves
correctly at the very end.
Lastly, performance tests run
multiple times over a given test
to look at the average timing,
memory usage or other metric
given to it to make sure you
don't introduce regressions in
these areas.
Today, we'll be focusing on unit
and UI tests.
The easiest way to get started
with testing in your Xcode
project is to choose to include
both unit and UI tests when
starting a new project.
In our brand new project, you
can see the unit tests targeting
class and the UI tests targeting
class automatically created and
displayed in your project
navigator.
You are also provided with the
template to start writing each
of these test classes and the
test cases within them.
Now let's take a closer look at
a test class that uses XCTest.
The class imports the XCTest
framework along with the target
to be tested.
The class itself is a subclass
of XCTest case which allows its
methods to be used by Xcode to
execute tests.
Each method we want to use as a
test case must start with the
word test and then hopefully be
named something that indicates
what it will do.
You'll also see a test diamond
appear to the left of the test
to show that Xcode can execute
it.
Inside of the test, assertion
APIs are used to evaluate and
validate your source code.
In this case, XCTAssertEqual
will compare the first two
values given to it and make sure
that they're equal.
If they differ, the test will
fail instead.
Now, once we run this test, we
hope that it passes and the test
diamond turns green with the
checkmark inside.
We know this isn't always the
case though.
If the test fails, the test
diamond turns red with an x and
the relevant line will
highlight.
We also get an error message
that shows what went wrong in
the test.
The string we pass to
XCTAssertEqual also shows up now
to give us more information to
debug the issue.
In this case, we may have an off
by one error in our source code
or you may have initialized to
an unexpected value.
We can go back to the source
code, fix this issue, and run
our test again until it passes.
Each test class template will
also include a setUp and a
tearDown method.
These blocks serve as a way for
you to do any work you need to
around your test in order to
keep your tests specific to
their purpose.
SetUp is called before each of
your test cases executes.
In UI test, this helps ensure
that your app is launched before
you try interacting with it.
XCTest then runs your test
method and afterwards tearDown
can be used to clean up any
changes you've made to the data
or the global state of your app,
to make sure your test leaves
nothing behind that could impact
subsequent tests.
Now, let's go into a demo to see
how to write unit and UI tests
for your app.
[ Applause ]
So, we've been working on a
travel app.
This shows different
destinations around the world
and can help you plan a
vacation.
I'd like to add a new feature to
my app that shows how far away
each of these destinations is
from our current location in San
Jose.
For this, I've written a new
class called DistanceCalculator.
This class defines a city struct
that contains a string with the
name of the city and a tuple for
its coordinates.
I currently have my list of
cities stored in a dictionary.
I'm planning on moving these two
database later though.
I have a function called city
that will return an optional
city struct type to be able to
search my dictionary for the
cities.
The main function I'm planning
on using from this class is
distanceInMiles.
This takes in two strings with
city names and returns the
distance between them as a
double.
If either of the cities can't be
found in our dictionary, an
error will be thrown instead.
Lastly, I have another helper
function also called
distanceInMiles.
This one takes in our city
struct and returns the distance
between them.
This function uses the core
location framework so that it
does the heavy lifting for me.
Now, to start writing unit test
for this class, I've created a
new test class called
DistanceCalculatorTests.
As I start writing tests, I want
to see my source code so I'm
going to open another editor
below, and then navigate to my
DistanceCalculator class.
The first test that I want to
write is testing my city
function.
So I'm going to start writing a
unit test and I can call it
testCoordinatesOfSeattle.
For each of my unit test
classes, I'm going to choose a
specific test case to make sure
that my function works.
In this case, I'm going to input
Seattle to my city function and
make sure that its correct
coordinates are returned.
The first thing I'll do is to
find calculator by initializing
my DistanceCalculator.
Now, I can start writing and
make a call to my function.
But one thing to remember is
that city returns an optional
city struct.
So I want to make sure that the
value I get is not nil.
To do this, I can use this API
and try using XCTUnwrap to make
sure that the value is valid.
Then I can call calculator for
Seattle.
Now, I'll get an error message
that appears, because it looks
like there's an error that could
be thrown that is not being
handled.
If city returns a nil value,
then I have to make sure that my
test case also throws to
properly show the issue.
Now, I have my city variable for
the value return for Seattle and
I can use my assertion APIs to
make sure that both the latitude
and the longitude are right.
Now, I'll run my test by
clicking on the test diamond.
This will launch the app, run
the test, and tell us what
happened.
It looks like our test
succeeded.
It's great.
I can write another test now for
my class and I'm going to focus
on my distanceInMiles class.
My function is going to be
called testSanFranciscoToNewYork
to specifically look at what my
functional return for the
assistance.
Now the first thing I'm going to
have to do here is to find
another calculator by
initializing DistanceCalculator.
But at this point, my code is
getting repetitive because I
have to do this at the beginning
of each of my test cases.
So instead, I'm going to declare
a class variable called
calculator and make use of my
setUp function to initialize it
before each test begins.
Now I can start writing my test.
I'll define distance in miles by
trying to call distanceInMiles
from my function from San
Francisco to New York.
And then I'll use my assertion
APIs to make sure that the
distance is correct.
I'll run my test from the test
diamond.
And it looks like this test
failed instead.
I can look at the error message
at the failure message in my
issue navigator.
This time our test runs and we
get an error message with an
actual test failure.
It looks like the value that
we're expecting and comparing to
has a lot more precision than
the one we're actually looking
for.
In this case, I don't care about
all this precision because I
only want to show my user a
whole number of miles between
these cities.
So, I can add an accuracy
argument instead of equal to
one.
This will allow XCTAssertEqual
to have a leniency and allow the
numbers to defer by one and
still pass the test.
I'll write my test again to make
sure it passes.
Great.
[ Applause ]
Now I've written two tests so
far for my unit test class, but
both of these tests take valid
inputs to the functions and
check that a valid input is
returned.
But there are some other test
cases I want to check for my
function.
I want to make sure that errors
are also handled correctly in my
class.
So, the next function I'm going
to write is going to be a
negative test case.
I don't expect Cupertino to be
my database of cities because
I'm only considering large
cities, and Cupertino won't be
one of them.
So, when I call Cupertino, I'll
expect an error to be thrown to
say that the city is unknown in
our database.
I can use XCTAssertThrowsError
to do this.
I can then use a closure to look
at the exact errors that were
thrown and compare them with
XCTAssertEqual to make sure
they're correct.
Now, I've written three tests
and I want to run all of them at
once to make sure that
everything is working right.
So I can go to the test
navigator and this will show me
row by row, all of the different
test targets, test classes, and
test cases in my project.
I can select at each level to
run all of the tests below it.
So, if click the play button
next to DistanceCalculatorTest,
it will run all three of these
tests.
My Apple launch runs them and
all the green checkmarks means
they all passed.
I can also command and click on
different rows in order to
select different values to run
or different test categories to
run if I don't want to run all
of them at once.
By context clicking, I can
choose to run them.
If I do choose a subset to run
and then want to rerun the same
subset later after I fix some
issues, I can also go to the
Product menu, click on Perform
Action and choose to rerun just
the last test that was run.
Now that I've written some unit
tests for my class, I want to
implement it in the UI.
I'll do so by showing a
distanceText and running my
class.
Now, under the city name, we
have a distance that shows how
far away these cities are from
our current location.
So, I've created a new UITest
class called DiscoverUITest.
Our UITest class has set up
populated with two things.
Continue after failure is set to
false.
Once we failed a UI test, it
usually means we've gone into a
UI state that's unexpected.
Chances are we won't be able to
interact with anything further
because we don't know what's
actually on the screen.
We also want to make sure that
our app is launched before we
try interacting with it.
We can start writing our UI
tests just as we were writing
our unit test.
I'm going to write
testMilesToParis and the goal
will be to open the app, swipe
to the Paris icon and then make
sure that the distance shown is
correct.
The first thing I'm going to do
is actually make sure that we're
on the discover tab when our
test starts.
I can write UI test by looking
for elements in the app's UI and
interacting with them.
In this case I'm going to look
for a tab bar button called
Discover.
And then I'm going to tap on it
to make sure we're on the right
tab.
After every UI action we do, we
want to verify that the correct
screen is now displayed.
So I'm going to then make sure
that San Francisco is visible on
the screen to make sure we're in
the expected state.
I'll use an XCTAssert statement
to make sure that San Francisco,
a static text isHittable.
IsHittable will ensure both that
the element exists and that is
on the screen so we can interact
with it.
Now, the next thing I want to do
is swipe left on this image to
get to the Paris location.
But I'm actually not sure how to
interact with this image because
they may have a custom label
that I'm not sure how to define.
So I can use the debugger to get
more information about my app's
UI.
I'm going to set a breakpoint on
line 26 and then run my test
from the test diamond.
The Apple launch click on
Discover, make sure San
Francisco is visible and then
pause in the debugger.
From the debugger, we can get
more information about the app.
We can print out the exact view
hierarchy by using po app to get
this information.
Here we have a list of all of
the UI elements in our app.
This is a little overwhelming
though so I'm going to further
specify that I want to print out
all of the images in my app
only.
Great.
Now we have a list of all of
these images and it looks like
San Francisco is the one I want.
So I'm going to copy the string
and I'll close the debugger and
then define my sfImage to be an
image with this identifier.
Now, I can simply call on this
query to swipe left.
Once I've done this, I want to
confirm that Paris is actually
visible on the screen so that no
changes in the UI would
interrupt this test.
So I can use XCTAssert to make
sure that Paris is hittable.
Lastly, I want to make sure that
this correct distance is
displayed.
So I'll use one more XCTAssert
statement to make sure that 5586
miles, a static text, is
visible.
Now we can run our test from the
test diamond and we'll see the
app launch and execute through
each of these steps, once I
remove my breakpoint.
So we'll see that the app
launches, we press Discover,
check for the text, swipe left
and then make sure that all of
our correct strings are
displayed.
[ Applause ]
One more thing to consider with
this test is that we're looking
for a static text, which says
exactly 5586 miles.
You may want to make sure that
you're mocking your location or
simulating that you're in
exactly in San Jose whenever you
run this test because otherwise
the distance will vary based on
your actual location.
And now let's go back to the
slide to talk about test
organization in your project.
So when starting to write your
test class, you'll start out
with two test targets,
generally, one for unit tests
and one for UI tests.
Unit test and UI test must be
separated by type because of the
differences in how they execute
on your app.
Each of these test classes--
test targets will contain your
test classes.
Your test targets can have as
many test classes as necessary
to test your app.
Your test classes then contain
each of your test cases.
Together, your unit test target
and UI test target can fully
test your app.
But there are some cases where
you may need more test targets.
As your project becomes more
complicated you may want a new--
to create a new framework.
This framework should have its
own unit test target to contain
its test.
Additionally, Swift packages
that you create and write tests
for in Xcode already define test
targets.
These test targets behave the
same way in Xcode as any other
unit test target.
Finally, once you have a full
test rewritten for each of your
test targets you may wonder how
well your test actually cover
all of your source code.
For this you can use Code
Coverage.
Code Coverage is a feature in
Xcode in XCTest that will
measure and visualize the number
of times each line of your
source code was executed during
your test run.
After enabling the feature and
running your test, you can go to
the report navigator and select
the coverage data for your test
run.
There, you'll see a list of each
of your test targets and classes
along with the percentage that
shows how much of your source
code in that section was
actually executed.
You can select each test class
to switch directly to the file.
When showing coverage data in a
source file, the source editor
gutter on the right will show a
number indicating how many times
that line was executed when you
ran your tests.
You can also hover over the
numbers to see information
directly in your source editor.
Lines that were executed will
turn green.
Sections that were not hit
during your test will highlight
red.
You can also see complex
information showing individual
code paths that were not hit
during your test, such as
conditionals that were never
selected.
The code coverage tool overall
gives you more information about
your test and can help you
identify areas you may want to
write more tests for.
When committing new work to
repository including your test
along with your code ensures
that everything is quality
controlled and checking your
code coverage ensures that you
don't miss anything.
If you feel like your tests
don't fully cover your changes,
it may be a good time to go back
and write more tests.
You'll get the most value out of
your tests by writing them
early.
By having a test suite that goes
along with your source code, you
can ensure that each new
function you write is reliable
and behaves as expected.
And remember, testing is an
ongoing process and vital to the
health of your app.
Now, I'd like to invite Stuart
to the stage to tell you more
about how you can get the most
out of your tests in Xcode.
[ Applause ]
>> Thanks, Ana.
So now that you've learned the
basics of testing in Xcode, I'd
like to talk about a new feature
in Xcode 11 called Test Plans,
which helps you get the most out
of your tests.
So whether you're just getting
started with writing test or if
your project already has a large
and robust test suite, there's
one piece of advice we'd like to
share to help you get the most
out of your tests.
We actually recommend that you
run them more than once in
different ways.
Now even if you don't modify
your tests at all, if you
leverage more of Xcode's build
in testing options and its
advance capabilities, you can
get a lot more out of your tests
and catch more bugs.
Let's walk through an example to
explain this.
So, imagine that the app we've
been working on is localized
into several different
languages.
And now that we've learned how,
we've written several UI tests
for our app as well.
Now, once we've done this, our
test will most likely succeed
when we run them in our
development language, which in
this case is English.
But imagine that one day we
discover a bug that only
reproduces in certain languages
where a localization string is
missing for that language.
It's using a placeholder string
instead which breaks the UI
layout.
Now, once we're aware of this
bug, we can adjust Xcode
settings to manually run our UI
tests in that language.
And if we do that we might see a
UI test failure.
Of course if we don't, that's a
great opportunity to write a new
test that reproduces the issue
and fails until we fix the bug.
But once we fix that issue and
we have a test covering it,
ideally we should always run our
tests in this language in
addition to our development
language to make sure it doesn't
break again.
So that's just one example but
there are other cases too of
bugs that might only be caught
if you run your tests more than
once in a different way each
time.
For example, you might choose to
run your test in both
alphabetical and random order
since running your tests in a
randomized order is really
helpful for finding hidden
dependencies between your test
methods.
Or you might want to run your
tests using more than one
sanitizer, such as both address
sanitizer and thread sanitizer.
And I'll explain a little bit
more about what these are later
on if you're not familiar.
Or you could even vary arbitrary
command line arguments or
environment variables each time
you test.
And this can be helpful if the
code that you're testing needs
to modify or fake certain things
when you're testing such as
using a testing version of your
web server or maybe mock data
sets.
Xcode allows you to configure
various options about how your
app is run using the scheme
editor today.
You can visit the Arguments,
Options or Diagnostics tabs to
control various things about how
your app is launched.
But this only allows you to run
your app interactively a single
time using whichever settings
you pick.
What we really like and what
we've been talking about is the
ability to run our tests
multiple times.
And for that, we're introducing
a new feature in Xcode 11 called
test plans.
So test plans allows running
your test more than once with
different settings.
Using a test plan, you can
define all your testing variance
in one place.
And then, you can share that
between multiple schemes.
Now, if you've previously
duplicated your schemes just so
that you can run your test more
than once, you might be able to
undo that and consolidate your
schemes back down to just one
using a test plan.
So test plans is supported in
Xcode as well as in xcodebuild
for use on continuous
integration servers and in Xcode
Server.
And it's really easy to adopt in
existing projects.
So rather than keep talking
about it, I'd like to go back to
my demo project and show you how
it works.
[ Applause ]
All right.
So back in the project that I've
been working on, I'll get
started showing your test plans
by clicking on the New Test Plan
File in my project.
So here, I can see all the
details about my test plan.
I can see all of the tests
organized first by test target
then by test class and then by
all of the test method inside of
each class.
Now, using this view, I can see
a few different things.
I can see the full list of all
of my tests, of course, or if I
want to find a specific test, I
can use it-- the filter field to
search for it.
Or if I ever need to temporarily
disable a test for any reason;
for instance, if this one is not
working at the moment, I can
just uncheck it in enabled
column.
I can also modify settings
related to test targets by
clicking on the options button
on the right.
And in this case, I happen to
know that this test target is my
UI test target and it would
really benefit from running all
of its test in parallel on
multiple clone simulators so
that they'll run a lot faster.
So I'll do that by enabling that
here.
So next, I'll go to the
Configurations tab of my test
plan.
And this is where I can control
how my tests will run and how
many times that they'll run.
On the left, we have a list of
what are called test
configurations.
And we also have an item at the
top called Shared Settings.
And the Shared Settings is where
I can control options for-- that
are common to all the times in
my test run.
And if we look at what all I can
control in a test plan, there's
a lot of things I can set.
I can modify different arguments
for how my tester should be
launched.
I can modify settings about
localization or alter how long
my UI testing screenshots are
preserved.
I can modify the test execution
order, enable code coverage or
enable things like run time
sanitizers or memory
diagnostics.
Now, you may notice that some of
the items here are in bold text
such as the Environment
Variables row.
That signifies that I've given
it a custom value.
And here, I've given it a custom
environment variable to run my
test with.
I've also customized my tests to
always run them in a random
order instead of alphabetical.
So what I really like to do in
this test plan is to modify it
to run similar to the example I
gave earlier so that it will run
two times using different
languages.
I'll do that by adding a second
configuration.
And because I'm going run it in
a different language, I'll give
it a custom name of the language
I'll use which is German.
And while I'm here, I'll
customize the name of the first
configuration to US English.
Now, in the US English
configuration, I'm actually
going to leave it as is with all
of its default values plus the
shared settings.
But in the German configuration,
I will customize the language
and the region.
And I want to point out that
since I'm editing the-- a
configuration right now, in the
pop-up menu, I see an extra item
at the top labeled Plan Default
Value.
And this item represents the
value that it would be inherited
from the shared settings level
of the plan.
So if I ever want to revert this
customization and go back to the
inherited value, I can just
select this.
All right.
So, now I've configured my test
plan how I want it.
Next, I like to show you a few
ways that you can use test plans
throughout the rest of Xcode.
I'll go to an event-- a unit
test file I've been working on
called EventTests, which tests
the event struct in my app.
And it's just got a couple of
small unit test about the
struct.
And if I click on any of the
test diamonds in this test file,
because I've configured my test
plan with two configurations, it
will run my tests two times
total.
And that's great for situations
when I want to run all of my
tests especially on my
continuous integration server.
But as I'm developing my test, I
may not run-- run them all those
times.
So I can choose to just run a
single configuration by option
clicking on the test diamond and
here I see a menu with just so I
could select one configuration.
I can-- Yeah, thank you
[applause].
I can also do this in the test
navigator.
If I control click on any test
in the test navigator, I see a
similar menu allowing me to run
either all the configurations or
just one.
And while I'm here, I also want
to mention that the test
navigator now shows the-- which
test plan is actually active at
the moment.
I only have one test plan in my
scheme but there can be multiple
and we'll talk about the
benefits of doing that a little
bit later on.
All right.
So now, I like to just run my
unit tests in all of the
configurations.
And now, we'll see Xcode quickly
build my entire project and then
run my test two times in the
simulator.
And it's already done.
So there, it looks like there
was at least one issue though.
So let's go see the details
about that.
I'll see those details by
clicking on the report navigator
and going to the most recent
test action.
So here, we can see that most of
my tests succeeded.
So they get the green check mark
and one of the tests had an
issue of some kind, so it has a
red icon with a dash through it.
And the test report is really
good at showing me all of the
details about everything that
happened in my testing session.
And if I expand this, I can
see-- it looks like this test
method succeeded in one
configuration and failed in
another.
And if I open it further, I can
see exactly the issue.
It looks like this test method
received English text but it was
expecting German text.
So that looks like a real issue
that I'll need to go back to
my-- either my app code or my
test and adjust and I'll do that
later on.
But before I wrap up, I want to
show you a few other
enhancements to the test report
here.
If I want to only see test
methods like this one which had
a mixed status between the two
configurations, I can do that
easily by just clicking on the
Mixed button in the scope bar.
Or if I only want to see the
results from a certain
configuration, I can just click
on the test configurations
popover and select one of the
configurations.
All right.
So that's a quick tour of test
plans in Xcode.
And now, let me just go back to
slides.
[ Applause ]
So, now that you've seen test
plans in action, I'd like to
mention a few details about how
it works.
A test plan file is really just
a JSON file with a .xctestplan
file extension.
And it contains all of your
tests to run as well as all of
the test configurations which
describe how your test will run.
Now, a test plan file is
included in your regular project
structure and it can be
referenced by one or more
schemes.
Now, a test configuration
meanwhile describes a single run
of your entire plan's test.
Now, each test configuration has
a customizable unique name and
it's a really good idea to give
each test configuration a
meaningful name for your project
since we'll see that name in
places like the test diamond
popover menu and the test report
like we saw.
Now, each test configuration
includes all of the options for
how to build and run your test.
And they can inherit any common
options from the shared settings
level of the plan.
So if you have any settings that
are the same each time you run
your tests, you can just define
them in one place and not need
to repeat yourself.
So if you're curious, here is
the full list of all the options
that you can set on each test
configuration.
And all this can be found in the
test plan editors
configuration's tab that I
showed.
So you might be wondering, how
can I begin using test plans.
If you have an existing project,
you'll first need to convert
it-- your scheme to use test
plans.
To do that, first, edit the
scheme then go to the schemes
test action.
And there, you'll see a button
labeled convert to use test
plans.
Clicking on this button will
show a sheet offering the
different ways that you can
convert the scheme.
Well, the first option is to
create a brand new test plan
file from the scheme's existing
settings.
If this is the first or the only
scheme that you're going to
convert, this is probably the
choice you want.
But another option is to choose
an existing test plan in a
project.
And if you choose this, it will
show a sheet allowing you to
pick an existing test plan in
the work space.
And this is a good option if the
scheme you're converting-- if
you've already converted one
scheme to use test plans and you
want a different scheme to share
that same plan.
And you can also use this if
you've created a test plan file
from scratch.
OK.
So now that we've covered what a
test plan is and you've seen how
to begin using one, I'd like to
offer a few potential ways that
you could use test plans in your
own project.
So here is one basic example of
a test plan that you could
create.
Each of the red boxes represents
one configuration in the plan.
And the first one has the
address sanitizer enabled and
the other has thread sanitizer.
Now, if you're not familiar with
these, sanitizers are tools
built into Xcode that instrument
your code and help identify bugs
that can be really hard
reproduce manually.
And some sanitizers like these
two can all be combined with
each other.
But if you construct a test plan
this way, you could still run
your test using both of them and
get all of their benefits.
Now, to get even more value out
of this test plan, if your
project includes C or
Objective-C code, you can expand
the plan by enabling the
undefined behavior sanitizer in
each of the configurations as
well.
So you might look at this and
notice that the undefined
behavior sanitizer is set in two
places.
It's repeated in each of the
configurations.
This would be a great setting to
move up to the shared settings
level of the plan instead.
And then, it will be
automatically inherited by every
configuration in the plan.
Now, one thing to be aware of is
that if you configure a test
plan like this with mutually
incompatible sanitizers is that
if you run both configurations
in the plan, Xcode will need to
build your project twice, once
for each set of sanitizers.
This is ideal for continuous
integration environments where
you don't mind your test taking
a little bit longer to build
since they're performing more
thorough testing.
But test plan isn't just about
picking different sanitizers.
As I showed in my demo earlier,
you can also configure a plan
with configurations representing
different languages or locales.
For example, I've picked the US,
South Korea, and Italy here and
there is no limit on the number
of configurations you can have.
Now, if you configure a plan
like this, you could use it to
run your UI tests.
And then, you could collect the
screenshots from those tests by
enabling the new localization
screenshots feature in Shared
Settings.
Localization screenshots is a
new feature in Xcode 11, which
will cause your UI tests to
preserve all of their
screenshots even for test that
succeed.
And it will gather data about
the localized string that your
app uses.
So this allows you to reference
the screenshot for context when
you're localizing your app or if
you've already finished
localizing, you could use the
screenshots in your app store
listings around the world.
[ Applause ]
Now for more information about
this and other localization
enhancements this year, check
out our the great-- Creating
Great Localized Experiences
session on the WWDC website.
Now, finally, I'd like to
emphasize that you can mix and
match these settings in a test
plan however it makes sense for
your project.
For example, you could construct
a test plan like this one with
three very different
configurations.
The first one focuses on memory
safety, and so it has the
Address Sanitizer and the Zombie
Objects memory setting.
The second one is all about
concurrency and it's got the
Thread Sanitizer, the Undefined
Behavior Sanitizer, and it runs
tests in a random order each
time.
Then the last configuration is
set up to collect extra
diagnostics while running test.
And it does this by setting a
custom environment variable that
the code being tested knows
about in order to trigger more
log collection.
And it also enables the option
to keep all custom file
attachments even for tests that
succeed.
So this is a fairly complex
example but hopefully it shows
the power and the flexibility of
a test plan to run your tests
however you would like.
So that's test plans, a new
feature in Xcode 11 to get more
value out of your tests by
running them multiple times in
different ways.
Now, I'll hand it over to Ethan
to share some ways you can use
Xcode for continuous
integration.
[ Applause ]
>> Thanks, Stuart.
To leverage the full power of
test plans, you likely want to
run your tests under many
different configurations.
A great place to do this is in
continuous integration which
automates the process of
building and running your tests.
While at your desk, you focus on
getting individual tests to
pass, continuous integration
runs all of your tests across
all of your devices giving you
maximum coverage.
When it comes to continuous
integration with Xcode, there
are two primary solutions to
choose from.
The first is Xcode Server which
is built directly into Xcode.
With Xcode Server, you can
easily set a box to build and
test your app with minimal
configuration.
The second option is to build
your own continuous integration
setup.
While more advanced, this option
is ideal if you have custom
requirements or need to
integrate with existing
infrastructure.
If you do require a custom
setup, you're in luck.
Xcode comes with powerful tools
that you can use to build your
own automation.
In this section of the talk,
we're going to focus on option
two and learn how to build a
completely custom continuous
integration pipeline.
End to end, our pipeline will be
composed of four steps involving
the use of different tools at
each step.
In step number one, we'll build
our tests on a dedicated builder
machine.
In step two, we'll take the
tests that were built and
execute them on a suite of
devices.
These devices will be connected
to a second machine that we've
set aside for running the tests.
The first two tests will produce
a set of build and test results.
These results will serve as the
source of data for our next two
steps.
In step three, we'll mind the
build and test results for
failures in order to populate
our favorite issue tracker.
And finally in step four, we'll
track our code coverage over
time to get a sense of how we're
doing in terms of our overall
test coverage.
Let's start with the first two
steps-- building and running our
tests.
For these tasks, we'll be using
xcodebuild.
Xcodebuild is the command line
interface to Xcode that will
power the core of a workflow.
Behind xcodebuild, sits the
xcodebuild system and the XCTest
testing harness.
With xcodebuild, there are two
ways to run tests.
The first is to build and test
in the same invocation.
For this, you use the test
action, passing in the name of
the project and scheme you want
to test, as well as the
destination on which tests
should be run.
The second is to build and then
test.
This puts up the actions of
building and testing into two
separate invocations of
xcodebuild.
One of the primary use cases of
this functionality is to have
one machine dedicated to
building and another dedicated
to running tests, which is the
workflow that we're trying to
achieve.
To accomplish this, you first
invoke the build for testing
action passing in the same
parameters as before.
This will produce both the build
products that are necessary for
testing, as well as an xctestrun
file.
The xctestrun file is a manifest
that describes the build
products and instructs
xcodebuild what to do at test
time.
Next, invoke the test without
building action, passing in the
xctestrun file that was produced
earlier.
This will cause your test to
actually execute.
It's actually possible to craft
your own xctestrun files giving
you more control over what
happens during test without
building.
If you like to learn more about
the format of these files, check
out the man page.
Also, note that the format can
change between Xcode releases.
In general, use the same version
of Xcode, both for building and
running your tests.
Speaking of running tests,
xcodebuild supports testing on
multiple devices or simulators
at the same time.
This can give you maximum
coverage across a variety of
device types and sizes.
And this is especially useful
for UI tests since your app's UI
likely varies according to size
class.
If you'd like to learn more
about xcodebuild support for
multiple destinations, check out
the What's New in Testing
session from 2018.
So far we've covered the basics
of xcodebuild, how to build and
run our tests.
Now I'd like to talk about a
couple options that are specific
to test plans.
If you have a scheme with
multiple test plans, you can
list them all using the show
test plans option.
Having a scheme with multiple
test plans opens up some
compelling workflows.
For example, you could have a
long running test plan
containing your full suite of
tests, and a short running test
plan with a handful of smoke
tests.
If you do decide to have
multiple, one of those test
plans is considered the default
which you can configure in the
test action of the scheme
editor.
The default plan is the one that
xcodebuild will run unless you
tell it otherwise.
To overwrite the default test
plan, use the test plan option
passing in the name of the
particular plan that you want to
run.
Armed with all that we've
learned about xcodebuild, we can
start to fill in some of the
gaps in our pipeline.
Starting with the builder
machine, we'll use xcodebuild
build-for-testing to produce the
build products and xctestrun
file that we need.
This will get passed to the
runner machine, which invokes
xcodebuild test-without-building
to execute the test on our suite
of devices.
The product of these two steps
is the build and test results,
which brings me to the next
thing I'd like to talk about,
and that is result bundles.
We have some really exciting
stuff to share with you about
result bundles this year.
First off, what is a result
bundle?
A result bundle is a file
produced by Xcode containing
structured data describing the
outcome of building and running
your tests.
It contains assets such as the
build log, revealing which
targets and source files were
compiled.
The test report showing you
which test passed and failed.
The code coverage report
reviewing which code was covered
by the test ran, and any test
attachments that were created by
the tests using XC test
attachment APIs.
So how do you produce a result
bundle?
Just pass the result bundle path
option to xcodebuild.
Now that we know how to produce
a result bundle, we can fill in
another missing piece in our
story.
We'll add the result bundle path
option to our Xcode build
invocation to start producing
those build and test results.
Now, result bundles have been
around for a while but in Xcode
11 we completely redesigned the
underlying file format, which
has come with several benefits.
First, the new format is highly
optimized to be efficient on
disc.
In our own testing, we found
result bundles to be four times
smaller on average compared to
the previous format.
This is especially useful in
continuous integration where
result bundles can be generated
and stored at a very high rate.
Second, it's now possible to
open result bundles directly in
Xcode allowing you to easily dig
into the results of an
integration.
And third, for the first time we
are providing a way to
programmatically access the
contents of the result bundle
which we will be leveraging in
our own continuous integration
setup.
To open a result bundle in
Xcode, simply double click the
file to view its contents using
the UI that you're already
familiar with.
See failing and passing test in
the test report, dig into the
build failures in the build log,
and see how you're doing on
coverage in the code coverage
report.
[ Applause ]
Thank you.
To access the result bundle's
contents programmatically, you
can use a new command line tool
in Xcode 11 called xcresulttool.
Xcresulttool gives you complete
access to the structure data
contained in the result bundle.
It emits this data as JSON and
the format of the JSON is
publicly documented and
versioned.
We'll be leveraging xcresulttool
in our next step, which is to
populate our issue tracker with
any failures that occur during
building or testing.
To extract build failures,
invoke xcresulttool using the
get command passing in the path
of the ResultBundle.
In the JSON output that follows,
you can find build failures
nested in one of the objects.
Each build failure includes both
the failure message as well as
the source file and line number
that failed to build.
Test failures are also nested
inside of the JSON with the name
of the test that failed, as well
as the assertion message.
Don't worry if you didn't follow
those steps 100%.
Like I mentioned previously, a
big benefit of xcresulttool is
that the JSON it produces is
publicly documented.
In fact, the tool itself can
describe the schema of the JSON
using the format description
command.
The schema lists all of the
possible types of objects that
can be present in the output.
So I encourage you to refer to
it as you write your own
automation on top of the tool.
Last but not least, check out
the tool's man page for more
information.
With our newfound knowledge of
xcresulttool, we can now throw
in step number three.
We'll use xcresulttool get to
extract those build and test
failures from the result bundle
and put them in our issue
tracker.
We've come a long way at this
point but we have one last step
to accomplish.
We want to track our code
coverage over time to know if it
ever decreases.
For this, we can use another
command line tool called xccov.
Xccov provides programmatic
access to the code coverage
report either as human readable
text or JSON.
To view the coverage report
using xccov, invoke the view
command passing it the result
bundle.
In the output that follows, you
can see the line coverage for
every target, source file, and
function or method in your
project.
Now, simply viewing the coverage
report may not be exactly what
you want.
If instead you want to compare
two reports to see if coverage
has gotten better or worse, you
can use the diff command.
Pass the paths to two result
bundles of the tool which will
produce output similar to this.
In this example, we can see that
the code coverage for
AppDelegate file increased by
50% between the two result
bundles.
Like xcresulttool, xccov also
has a man page, so check that
out for more info.
And with that we can finally
fill in our last step.
We'll use xccov to extract code
coverage from the bundle to
track our progress over time.
Awesome.
Our continuous integration
workflow is complete.
Using xcodebuild to build and
run our tests, xcresulttool to
extract, build, and test
failures, and xccov to view code
coverage, we built a fully
functional end-to-end pipeline
that automates the testing of
our app.
Hopefully this gives you a sense
of just how much power and
flexibility is available to you
using the tools that come with
Xcode.
We've only covered one possible
workflow but with these building
blocks the sky is really the
limit.
We've covered a lot of content
in our talk today so let's
briefly recap what we've
learned.
We started off today's talk with
an introduction to testing in
Xcode.
We learned how to write unit and
UI test using XCTest and how to
run them to catch bugs.
Next, we learned about Test
Plans, a new feature allowing us
to better organize our tests, as
well as run them multiple times
under different configurations.
And finally, we learned about
the tools that we can use to
build a custom continuous
integration pipeline.
If you'd like to learn more,
grab a copy of these slides from
developer.apple.com and be sure
to check out the release notes
for Xcode 11.
Finally, if you're interested in
hearing about new APIs in XCTest
for measuring the performance of
your code, check out the
Improving Battery Life and
Performance session later today.
Come see us in the labs and have
a great WWDC.
[ Applause ]