WWDC2013 Session 409

Transcript

X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[ Silence ]
>> Good morning.
[ Applause ]
Wow. Thank you for coming
to Testing in Xcode 5.
I'm Mike and before
we get started,
could I have everyone
raise your hand
if you've ever written
a unit test before.
Wow. Actually, could
you keep your hand
up if you've written a
unit test in Objective-C.
Awesome. All right.
Thank you.
What I'm here today
to show you is all
of the great unit testing
features that we've added
in Xcode 5 and show you how
easy it is to test your apps.
So easy that I hope
by the end of today,
we'll have absolutely
every hand raised.
So, first, why would anyone
actually want to do testing?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Perhaps you've heard a few
times this week that we want
to help you build better apps.
So first off, wouldn't
it be great
if you could catch crashers
before your app got submitted
into the app store?
Did you know that crashers are
actually the number one cause
of app rejection?
Any crashers that make their way
to your customers are
probably only going
to lower your app's rating.
And if you can catch
your crasher in testing,
it means you don't have
to re-submit your app.
But besides crashers,
you'll also want
to catch logical errors.
Suppose, you have a game with
a character who's running along
and picks up a health pack,
it'd be pretty embarrassing
if picking up that
health pack actually ended
up killing the character because
of some sort of integer overflow
if the health was already
topped off at 100 percent.
Or, perhaps the pack was
poisoned or maybe the character,
it was just getting greedy, but
either way you're going to want
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to write a test to define
what the correct behavior is
for your app.
So once you have a bank of
this test built up over time
from release to release,
you'll find that these sorts
of test will start
to catch regressions
as the rules of your app change.
As your code evolves over time,
these sorts of test will begin
to surface logical errors
and sometimes just
find plain old bugs.
Also, in some of the other
sessions, you may have been
at earlier this week you
may have seen how to set
up OS X server to run your
tests on a huge mix of OS
and device configurations.
Many more configurations than
we could reasonably run by hand
because we as developers want
to actually spend time writing
features and not spend all
of our time running tests.
And most importantly, for
every test that you commit
to your source repository,
OS X server can run that test
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
when someone else on
your team commits.
Because, maybe the new guy
doesn't have the entire model
of your app in his head yet,
or perhaps he just forgot
some sort of subtle edge case.
I mean, who among us
hasn't actually introduced a
jaw-dropping regression
that nobody picked up on
until it was way too late
in the release cycle.
So, all this is great, right?
Well, let's see how we're
going to do all that.
So first, we're going to briefly
touch on what a unit test is.
Then I'm going to show you
the new testing framework
that we've added to Xcode 5.
And then we're going to cover
how you can create your first
test and then for the
bulk of the session,
we're going to show you some
of the exciting new UI
enhancements we've added
to Xcode 5 that will allow
you to run single tests just
with one click or all your
tests in really large batches.
Then we'll show you
how to debug your code
when it's running
inside of the test rig.
And, we'll show you
how testing fits
into the continuous integration
the OS X server provides.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And finally, we'll look at
some advanced configurations
for OS X server and how
you can drive testing
on the command line.
So first, we should
define what we mean
when we talk about unit tests.
[ Pause ]
A unit test is a small test
of one unit of functionality.
It either passes or it fails
and the behavior should be
completely deterministic.
If you want to run
a lot of them,
they should run pretty quick
because when they fail,
you're going to want to
diagnose them really fast.
Though, unit testing
is not necessarily good
for some things.
For example, unit tests
are not performance tests,
a pass-fail result isn't going
to tell you how long it takes
to run a benchmark or
the standard deviation
of that benchmark over time, or
the variation of that benchmark
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
across different hardware.
Also, UI testing is
pretty hard to do
from a unit testing framework.
There are a lot of complex
interactions in the UI
and it's hard to
isolate down precisely
where a failure originates
from when you're running
tests at such a high level.
You're also going to want
screen recordings and a lot
of other support you don't get
from a simple pass-fail result.
And this sort of speaks to a
greater point and that it's hard
to understand failures that are
introduced by the integration
of a complex system that isn't
already granularly unit-tested.
You want to have the confidence
that the individual pieces
of your application
are well-tested
so that you can trust that your
higher level tests are actually
reporting higher level failures.
What this means is that
you're going to want
to pick the right place
to start testing your app.
So here, we have a high
level block diagram
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of how your app might
be structured.
You probably have
some sort of model,
view and controller classes
which are some pattern familiar
to anyone who's been
working with Cocoa.
And you might have
some sort of connection
to the network behind some sort
of database that's behind
some sort of web service,
but what we're really
focusing on today is
where your model classes
meet your controller.
This is a good place to start to
build up tests because you want
to know with certainty that your
model classes are well-tested
before you work your
way up to running test
for the controller
classes which start
to touch the other more
complex parts of your app.
Alternately, if you're a
framework or a library author,
you could write tests around
the external API of your code
and work your way into
the internal classes.
It's really up to you, but
the new testing framework
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and UI features in
Xcode 5 are designed
to make running model tests
really easy and really fast.
So, the Xcode unit testing
framework Xc test is brand new
to Xcode 5 and iOS 7.
If you've used testing before
an Xcode, you may recognize
that Xc test is actually
directly derived from OC unit.
But it's been significantly
cleaned up and modernized.
We even have a migration tool
to migrate your existing
send tests to Xc test.
Xcode builds your test
into an Xc test bundle
which is injected directly into
your application and invoked
after application
did finish launching.
Or, when testing a library, the
test rig will load your library
and the test bundles together.
So what does a test look like?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Tests are just ordinary
Objective-C code.
A test class is simply a
sub-class of Xc test case.
Tests are simply methods
inside of a test class.
These methods are
identified by the word test
at the beginning of
their method name.
And, test methods return
nothing and take no arguments.
So how do you know if
a test passes or fails?
Well, tests make assertions
in the bodies of their method.
What this means is that you can
actually have multiple failures
inside of a test because you
can make multiple assertions.
The test classes are discovered
by the test rig interrogating
the Objective-Code runtime
and asking for all of the
subclasses of Xc test case
from the Xc test bundle
which has been injected
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
into your application.
So, these are the basics
of how tests are
structured at the code level.
What I'd like to show you now is
the most significant UI feature
that we've added to
Xcode 5 for testing.
It will help you add, run, and
navigate all of your tests.
This is the new test navigator.
As you can see here,
nestled between the issue
and the debug navigators
is where you'll find all
of your tests in your project.
When we started designing
Xcode 5 to make the act
of testing your app a
first class experience,
we quickly realized that
the workflow of being able
to view your tests
and their statuses was
critically important.
At the top level of the
navigator, you'll see all
of your test bundles
in your project.
Those test bundles
contain your test classes
which contain your test methods.
We've also made it really
easy to run any test or group
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of tests with just one click.
As the tests run, the result
show up live along the side
of the navigator, this gives
you a great top level overview
of what's passing and
failing and lets you drill
down to any specific test.
Also, if you just want to focus
on only the failing tests,
we have a filter for that.
In this view, you can see
only the current failures
but still run all of your tests
as you work your
way down to zero.
We think that the new navigator
will give you a very nice
overview of your tests.
But we didn't stop there.
Next to each test method
in the source editor,
we added new test indicators.
You can click to run just
that test and see its status.
Running each test
is really easy even
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
without the test navigator open.
So let's look at each
of these features
and how they work together
over on the demo machine.
[ Pause ]
So here, we have a project
that's a board game class
that has several classes which
represent the model of our game.
There are classes for how the
parts, the pieces on the board,
the scoring and how the board
itself actually interacts.
So, one of the rules in a game
like this is that normally,
two pieces cannot occupy
the same space on the board
at the same time, but we've
actually noticed this has been
happening in our board class.
So let's try and write a test
to tease out this problem.
I can do that by going
to the test navigator.
As you noticed we have
no tests for our project,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
go down to the plus menu
and choose new test target.
What this will do is create a
new bundle inside of our project
that is linked against
our application
which contains the board
class that we want to test.
As you can see, the new test
bundle shows up right away
and takes just a
moment for the indexer
to find the board class
and the test method.
The test class and
the test method.
We can run the test just by
clicking on the indicator
in the test navigator.
And the test fails.
So we can find out why
by clicking on the test
which will navigate us to that
test and the source editor.
And as you can see here,
the test is unimplemented
so by default it has a
built-in unconditional failure.
So, we created a code
snippet ahead of time
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that has a pre-cooked test
for this demo which I'm going
to autocomplete into
place, so you don't have
to watch me type
this all at on stage.
This is test unit locations and
it's got a couple of imports
that we need to put
here at the top.
There we go.
So this test creates four model
objects in our game, the board,
two pieces and one location to
try and move them both into.
This test makes four assertions.
First, it asserts that
moving the first piece
into the location on
the board succeeds
and then it actually
checks to see
if the piece is at
that location.
Then, it tries to move a second
piece into the same location
on the board and asserts that
that move actually fails.
Then, we check to
see if the second--
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
if the first piece is
still on the board.
So let's run this
test by clicking
on the test indicator next to
the method here on the editor.
Well, it looks like
our test is failing.
We should probably take a
look at the move piece method
to see why it's not
working as we expect.
I'll do that just
by command-clicking
on the move piece method.
And it looks like there's--
that the location to piece map
is always getting the piece
slammed in and there's no point
where this method is actually
trying to reject a move
to a location if there's
a piece already in place.
So, we can fix this pretty
easily just by adding a check
to a location to
piece map to see
if there is something that's
already in the location
and just return early with
a no to reject that move.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So, we can go back to the test
and rerun it with our change
in place and our test succeeds.
So, now our board class is
correctly rejecting moving a
piece into the same space
where another piece already is
and we'll go back to the slides.
[ Applause ]
So, what did we just see?
We made our first Xc test
bundle in our project
to test our board class inside
of our app using
the test navigator.
Then we wrote our first test
and asserted what the correct
behavior was for the board
and the piece classes and how
they were supposed to interact.
Then we ran the test
which exposed the bug
in our games model and once we
fixed up the implementation,
we reran the test without
ever leaving the editor just
by clicking on the
test indicator.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So now we have a test
that we can commit
into our source repository and
be sure that any future changes
to the board or piece classes
won't reintroduce the same bug
as long as that test gets run.
Pretty easy, start getting to--
easy to get started
with testing, right?
So let's take a closer
look at how Xc test works.
So the whole point of a
test method is to assert
that certain assumptions
are met.
If you can think of a condition
that you might want to test,
Xc test's got an
assertion for that.
We're just going to take a look
at a couple of common
assertions.
For example, XCT assert equals
objects, takes two objects
and the optional message.
It passes if the two objects is
equals methods actually agree
that the two objects
are in fact the same.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
XCT assert not nil
is a favorite of mine
because it can catch
situations where you message nil
and it returns nil and
propagates that nil farther
than you might expect.
XCT assert true and false.
Both take a single
Boolean expression
and an optional message and do
basically what you would expect.
And XCT assert throws is really
good for testing your API
to ensure that an exception
is thrown in situations
where it's actually being
passed in valid parameters.
So let's take a look at a few
things that are good to assert.
Most tests fall into
a few categories.
First, are expected successes.
These are situations where
your code is working basically
as designed.
An addition method takes 2 and
2 and returns 4, a character,
gets a health pack and health
is bumped up by 5 percent.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
This may include EDGE Cases,
but this is still situations
where your code is
doing the right thing
by whatever definition it is
that you've defined it to be.
Some people actually
write tests first
which help them define
their implementation.
They'll take use cases
and express them as tests
and they'll keep writing
their actual implementation
until they get all of
their tests to pass.
The second category are from
bugs that have been reported
by your users or have been
found in other testing
that you've distilled down
into making unit tests.
These kinds of tests
grow over time and ensure
that you don't reintroduce
the same kinds of problems
with later code changes.
The third category is a
little less intuitive.
To fully cover your code,
you should consider cases
where your code is
actually expected
to fail however it emits errors,
how it might throw an exemption
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
if your addition method takes
2 billion plus 2 billion,
it could result in overflow or
out of nowhere, a wild infinity
or not a number could appear as
a result of some calculation.
It makes sense to
write a test to ensure
that an exception gets thrown
or some other error is
surfaced for these cases.
Sometimes with higher
level objects,
getting past nil is
totally unexpected
or you could be passing an
array which is shockingly empty.
The untyped collections in
Cocoa can be problematic
if you assume things like keys
and dictionaries
and always strings.
Or, you could get an NS
null value that could sneak
in as a side effect to
somebody observing through KVO.
And finally, if your
code uses NS errors
or venns them throughout
parameters, you want to assert
that those errors are actually
created with useful information
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
in them and that they're
handled in a sensible way.
So writing tests for
all of these sorts
of situations will give your
code the most complete coverage.
So now, I'd like to back up and
take a look at what sort of set
up might be needed for each
of your test methods
before they're run.
The set up method is your
opportunity to run common bits
of code before each
of your test methods.
If you find that you're
copying and pasting code
from test method to test method,
set up might actually
be a pretty good place
to centralize some of that.
You can set up shim objects
that might represent things
like a server which would
not be appropriate to contact
in a testing environment.
You can also load
Plist, JSON, and XML data
from your Xc test
bundles resources just
by calling NS bundle,
bundle for class
and giving it a reference
to your test class.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Putting data and documents
into your test bundle
is a great place
to store pathological used cases
that may have been
reported by as bugs.
And finally, if you need to
undo anything that you did
in your set up method,
tear down is there for you.
So let's see how
this works in action.
First, the test rig finds all
subclasses of Xc test case
and starts iterating through
them in no particular order.
Then for this example, it finds
exempt-- the example tests class
and notices that there
are two methods in it.
First, the set up method
for the class is run,
then a new instance of
the test class is created
and set up as called on that.
Then your actual test
method itself is run
which in this case
will pass and then tear
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
down is called on that instance.
After that, another
instance is created
and setup is called on it.
Then the second test method gets
run which in this case will fail
and tear down gets called on
the instance then gets called
on the class and the test rig
moves on to the next class.
Pretty straightforward, right?
So, I'd like to take a
moment before another demo
to mention OC unit.
OC unit is Xcode's unit
testing framework or has been
since version 2.1 and that's--
you may already have some tests
that are written in it.
A lot of you actually have
tests that are written in it.
That's great and we don't want
to break any of your tests.
So we ensured that OC unit
and Xc unit could coexist side
by side as we make
more disruptive changes
to Xc test in the future.
Xcode 5 actually recognizes
all of the same test classes
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and will show you your classes
and your test methods inside
of the test navigator
as well as all
of the editor indicators
in the side bar.
If you're-- if you want to
run your tests on iOS 6 and 7,
OC unit is actually the
right choice for you.
So-- but when you're
ready to adopt Xc unit,
Xc test we have an
option available under--
to the run the migration tool
under the Edit menu
refactoring option
and then convert
to Xc test case.
So now that we've dove
into Xc test all the way
down to the code level.
I'm sure you're going to want
to see how to debug your tests.
We've added some
great new UI features
to Xcode 5 beyond just
the test navigator
and the editor indicators
to help you quickly run
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and debug your tests
with a workflow I think
you're going to love.
So to show this to you in our
next demo, I'd like to turn it
over to my friend and
colleague Bino George.
[ Applause ]
>> Thanks Mike.
This is the demonstration.
So this is a newer version of
the project that we saw earlier
and it has many more test that
covered the various EDGE Cases
for our board games
model classes.
Before we get started and I need
to add a new break point
that's new in Xcode 5
which breaks on test failures.
To do that I'm going
to switch the test--
to the breakpoint navigator
and I'm going to click
on the plus button and select
Add test failure breakpoint.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So this special breakpoint stops
when a test assertion fails.
You'll probably want
to understand
when you're debugging your test.
It's helpful because it stops
the debugger right at the point
of the failure, at the
point of the test assertion
where you can inspect the live
objects that caused the failure.
So let's switch back to test
navigator and run all the test.
I can do that by clicking on
the one button for the bundle.
[ Pause ]
Huh, looks like there's
a failure.
Mike must have checked in
something before he went
on the stage that
broke one of my test.
It looks like they're
failing on and this--
it looks they're
failing on an assertion
for our rule checker class.
In the out of bounds move test,
it's creating an invalid
location and then trying
to move the piece there.
[ Pause ]
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The debugger tool
tip was telling us
that the rule check is
actually allowing this.
So now we know why the test is
actually failing but let's find
out why it's actually failing,
why the rule checker
is allowing this.
To do that, I'm going to add a
new manual breakpoint before the
test failure.
And then I can rerun
the test again.
Now, on the second run we hit
the manual breakpoint before the
actual failure.
So lets step into the
valid location method.
So the first thing it does is
actually check that the piece
that you're moving is
actually already on the board
and to do it I just
find the old location.
I'm going to step
over it and it looks
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
like all location is not nil
so that means the piece
is already on the board.
So the next thing it does
is it actually checks
that the location you're
moving to is actually
within the bounds of the board.
And that's exactly what our test
is actually doing in this test.
So let's see if that's the case.
[ Pause ]
Clearly the rule check is
clearly wrong here and--
no the logic is clearly
giving us the wrong result.
So the board's frame
is actually 0088
and the coordinate is actually--
the removing tool is
negative one, negative one
which is clearly outside
of the bounds of the board.
The bug is probably in
this giant nest of code
and since we already
have the board's frame
and the performance of
NSRect and the boards--
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
the location we were going to--
in the form of an NSPoint,
I think there's a foundation
method that does this for us
and we can remove all those
bunch (inaudible) away.
It's again it's called
NSPoint/NSRect.
It takes a point and NSRect.
So the point in this
case is new coordinate
and the rect is board frame.
That's a lot simpler.
So let's see if this actually--
if the test is passing now.
To do that, I'm going to
use a new test command
in the Product menu.
The test again command
is very powerful.
It allows you to rebond the last
set of test that you're in over
and over again and it can help
you debug your test while you're
iterating over the test.
So in the Product menu
under Perform Action,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
I'm going to select test again.
It looks like I hit
the manual breakpoint
that we added earlier.
So I'm going to remove
that and continue.
Great, our test is
passing and the one checked
in code is working now.
So now that we fixed this
test, I want to make sure
that we didn't break any
other test in my code.
So I want to find out
which other test are there
for this API.
To do this, I can use the--
a new assessment category
at the other Xcode 5.
So I'm going to switch
the assessment editor
and select Test Colors.
And I'm going to switch
to the primary editor back
to the API review testing and
here is our first test method
that we already tested.
Let's see if there
are any other test.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
There's another test called test
vertical move and let's switch
to that and run that
test as well.
Great, this test
is also passing.
So now let's make sure that
everything else is working
and to do that I'm going to
switch the test navigator.
It looks like we never
completely finished the full run
of tests.
So I can rerun all the tests
by using the Test
Menu on the product.
[ Pause ]
Great. Looks like
there are no failures
and we see green all the
way down to test navigator.
Everything looks great
on my local machine
and I should probably
check in this fix,
but for now let's switch
back to the slides.
[ Pause ]
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So we just saw a
lot in the demo.
There are three new Xcode 5 UI
features I'd like to call out.
The first is the test
failure breakpoint.
When we add it to our project
it stopped the test execution
in the debugger right
at the test failure.
So this allows us to
inspect the live objects
that cause test failure.
Second we saw that there are
new assistant categories.
We use a test color's
category in the demo
to find the second method
that was calling our test
a valid location API.
It works just like the
general colors category
but it focused just
on the test methods
that are calling the
method that you're editing
in the primary editor.
Very tight, very
focused results.
The test pluses category we
didn't see it in the demo
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
but it's a good way-- what it
does is it gives you a wider
range of classes that have test
methods in them that call the--
that referenced the classes
you are editing in the primary.
It's a good way to step back
and see where you might want
to add new test especially for
new code that you're developing.
And third, we saw
the-- we use a test--
a new test command in the
product Perform Action menu.
The test again command
is incredibly powerful.
It allows you to rerun the last
set of test that you run over
and over again with
just one key stroke.
You can be in the middle
of editing any file
in your source editor
and would want it
and you can work the
Test Again command
and you get immediate results.
You will now just then
rather, changes you made
in the editor actually fix the
test that you're working on.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
We think it's a really big deal
and it's one of those features
that you love once
you start using it.
So now that we've
shown you how to write
and debug test locally
inside test Xcode,
I'd like to shift gears and show
you how OS X server can save you
time and catch problems
by running all
of your tests all the time.
So earlier this week you
may have seen how to set
up OS X server, so we will
make something how to set
up OS X server in this session
but in the Continuous
Integration with Xcode 5 session
on Tuesday this was covered and
depth so you might want to refer
to that for your-- when
you leave the session.
So after you set up the export
service, you can create a bot
to integrate your project.
Integration is the act of
checking out your code, building
and running your tests.
Integration can happen on
every commit, on every hour
or whatever interval
is right for your team.
[ Pause ]
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So, you could have multiple bots
-- integrate all of the products
in your project, one
for your app NSTest,
one for your framework NSTest,
one for your library NSTest.
But there is a much
simplier way.
You can create a shared scheme
and this shared scheme
everything in your app,
your app NSTest, your library
NSTest, they're all going
to this shared scheme.
All the test bundles for all
of the products you create
are going to the scheme
which you can then edit
in the scheme sheet
or in the test navigator
directly
in the test navigator itself.
You will then check
in this sheet--
this scheme into
source repository
and then the bots can integrate
your project, the scheme.
Since the shared scheme,
everybody in your team can
also run the same scheme.
So once you've set up a bot
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
for your project any time
there's a new test failure
or new build issue will
bring that to you right away.
On the score board over
the next code itself right
into the activity
area where you can--
when you open that project.
So besides running all of your
test all the time for you,
there's an, you will now an
exciting reason why you want
to have OS X server do your
continuous integration for you.
You can cover-- your test
will run on a wide variety
of configurations that
simply isn't possible to do
on your laptop or
while you are sitting
at your desktop,
sitting at your desk.
You can have many more
devices connected to the server
and the server will basically
run all of your test on any
or all of these devices
for you automatically.
This allows you to cover
a wide variety of hardware
and software configurations
automatically.
You can also run your test
on the simulator although
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
in the current OS X server
seed that's not supported,
it's not available.
Between the simulator and the
devices connected to the server,
you can cut-- you can cover
a wide variety of hardware
and software cointegrations
automatically
and you can cover
all the OS versions
that you support as well.
If you're integrating an OS
X project, server will check
out your project, build
your app and NSTests
and run everything locally on
the server itself without having
to have a user logged
in graphically.
So to see this in action,
let's switch to a demo machine.
So this is the same
project that we saw earlier
that I just checked out
from my OS X server here.
This red indicator shows us
that there's a problem with one
of the boards that's
integrating this project.
So I'm going to click on this
indicator to see what's wrong.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[ Pause ]
Let's take a look at the board
to see exactly what's going on.
[ Pause ]
Here we see all the
recent integrations
that we performed on--
that the board performed
for this project.
I can see all the test
failures as well, a few.
So we've had a run
of clean integrations
and everything was going
up until this point
but somebody broke one
of our tests just now.
So to find out which
test is failing,
let's click on the
integration test results.
[ Pause ]
So I have-- this is where you
can see all your tests and all
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of the devices that are
connected to the server
and the results for all of the
test that ran on the server.
You can see various
configurations
like I have an iPad mini here
and an iPhone running
iOS 6-- iOS 7.
I only have two devices here
but you could have many more
devices connected to the server.
The first device I have is an
iPad mini and its running iOS 6,
the second device I have
is an iPhone running iOS 7.
So we can look at the failing
test, the code of failing test
by clicking on the
red test indicator.
So this is a test that
basically parses high score data
from our server.
In this test we fake the
data because we don't--
this is a unit test and we
want it to be really fast.
We also want the test
to work in isolation
so that it's not relying
on any data from the server
that could change
from one to one.
So what could have gone wrong?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Perhaps integration
summary could help us out.
So let's go back to
integration summary.
So the integration
details tell us
that we are calling an
unrecognized selector called
scan on-sign long, long.
This is really weird because we
didn't have any build failures.
So let's switch back
to the code and see
where we're actually using it.
[ Pause ]
We're not actually using
in the test meter itself.
I don't see it here,
it's quite small.
Maybe we are using again
the high scores class.
So let's command-click on
it and see if we're using it
in the high scores class.
It's quite a big file so I'm
going to search for it in here.
There it is.
So how could this fail on an--
on the iPad but not
on the iPhone?
Perhaps there's a
difference in the OS versions?
So let's look at the header
for this method to find
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
out when it was introduced.
I can do that by
command-clicking on the method.
So this method was newly
introduced in iOS 7.
So we're running an iOS 7 API,
we're calling an iOS 7 API
on a device that's
running iOS 6.
This is not going to work.
No wonder it's failing.
So right about that, I
see another method called
which scans the sign numbers.
So we could probably use this
because it's exists iOS
6 and I think it'll do.
It just means that the highest
score can only be 9 quintillion
instead of 18 quintillion,
I guess that will work.
So let's switch back to the
code and fix it-- fix the AP--
fix our code to use the
new AP-- the older API.
[ Pause ]
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And now I think I fixed
the test and fixed my code,
so let's run all
the test locally
to see if it works locally.
During that I am going to
switch to Test Navigator
and select product test
to run all the test
locally in the simulator.
Great, everything
is passing locally.
So let's check it in by
using the source control menu
and selecting commit.
[ Background Conversation ]
Push it to the server.
So now I checked it in and I can
click off a manual integration
because I want it to happen
right away by switching
to the bott and performing
integrate.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
[ Pause ]
So, when I push the
integrate button,
Xcode sends integration
request to the server.
The server then checks out all
of the code from my project
from the source repository into
a temporary build location.
Then it builds all of the
apps code and the test code
into an app bundle and a
test code, test bundle.
Server then tries to
figure out all the devices,
it scans all the
devices connected to it.
Server then uploads your app
and its test bundles into the--
to each of the devices.
In this case I have
an iPad and an iPhone.
Then your application
is launched as we talked
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
about earlier in the application
that finished launching
it calls your test
and it got-- runs your test.
Once the last test is
finished on the last device,
basically server collects
all of the data for all
of the test results from
each device and stores it
into the box integration
history.
So you can go back and see where
and how things have
done in the past.
And the whole integration
results are stored
in the history.
Then, all the connected
clients Xcode here
and the score board
we saw earlier and any
of the web sessions are
all informed of the result.
And now what we can see here
in Xcode that all of the test
and all of the configurations
have passed.
Excellent.
Thanks to OS X server we've
tested many more configurations
automatically than we
have time to do by hand.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So let's switch back
to the slides.
I'd like to invite Mike back up
to recap and then wrap it up.
[ Applause ]
>> Great. Great, thank you Bino.
So, what did Bino just show us?
Well, first to set up the bot,
you need to add your tests
to a shared scheme and commit
that into your source
repository.
In the demo, we showed-- we
saw how OS X server can run all
of your tests on a variety of
devices and configurations.
Then, Bino showed us the bot and
test summary results in Xcode 5
and how you can use
it to find failures
that are not necessarily
reproducing in the IDE.
Now, all of this is great
if you can start fresh
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
with a brand new OS X
server and set that up
as your continuous
integration environment.
But what if you already
have one?
Through all the new features
that we've added to Xcode 5
for testing, we haven't
forgotten
about the command line.
This is your opportunity
for your existing continuous
integration system to hook in.
The Xcode build command
line tool can drive testing
in exactly the same
way as the Xcode IDE
or the Xcode service
inside Xcode server.
The only mandatory argument
is the shared scheme.
You can also specify
different destinations
that you want the
integration performed to.
For example, My Mac 64, or
if you have devices plugged
in to the server, you
can call them now by name
and their device identifier.
You can also use a simulator
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to target all the
different device form factors
and OS combinations that
that simulator supports.
Best of all, you can chain all
of these destination
arguments together
and an integration we
performed across all of them.
If there are any issues that
came up in either the building
or the actual tests themselves,
Xcode build will return a
non-zero exit code and this is
where your automation
can pick up on that
and store off the
transcript of the log for you
to investigate the
failure later.
So, we think this is really
easy for command line tool
and that is the last thing I
have to show that's new today.
So, to recap, today we covered
what a unit test is in general
and how they test
just one thing.
And how to quickly
write those tests
in Objective-C using Xc test.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
In the demo, we showed
just how easy it was
to start getting writing-- get
started writing your unit tests
and how to run them quickly
in the test navigator
and using the test indicators
in the source editor.
Then, Bino showed us how to
use the test failure breakpoint
to step through your
code at the moment
that a test assertion failed.
Then, he also showed us how to
use the new assistant categories
to find other tests that might
be calling the implementation
that you're working on
in the primary editor.
Then, he also showed us how the
Test Again command can quickly
help you rerun the same test
over and over and over again
as your iterating on
the implementation
without even having the test
navigator or to test itself up.
In the demo, he showed us
how to run a wide variety
of configurations
on different devices
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and how you can run all those
configurations all the time
and then finally we
covered just now how
to drive the same integrations
at the command line.
So throughout the three demos
today, we showed three bugs
that could've been
potential show stoppers
for any application.
The first, being a pure logic
bug where the two pieces
on our board collided,
the second was a case
of overly complicated
balance check
in that could've been replaced
by a simple library function.
And the third was the case
of using a two new API
when deploying on to
an older OS version.
What I hope that you've seen
today is how testing can catch
these kinds of bugs in your
app and just how easy it is
to start the testing right
now if you haven't already.
So, if you have any
questions or feedback,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
please let our evangelist
Dave know.
Here are the links
to the documentation
and developer forums.
To learn more about
OS X's server--
learn more about OS X server and
the Xcode service, please check
out the Continuous Integration
with Xcode 5 session
that was on Tuesday.
The video is available
right now from the wwcf
that you have in your pocket.
And with that, I'd
like to say thank you.
[ Applause ]