WWDC2004 Session 301

Transcript

Kind: captions
Language: en
good morning everybody welcome to
session 301 automated testing on Mac OS
10 I'm John Mont Brian from apple
developer relations and I'm pleased to
introduce your host John Comiskey will
be speaking to you he's a senior
engineer from applescript engineering
and welcome Jon Missy thank you good
morning thank you for being here on John
Chris gamma engineering the applescript
group I'm going to talk to you today for
about 15 minutes about using Apple
script to test your application and then
some gentlemen from Redmond from red
stone software will be talking about
eggplant much more high-powered way of
testing your application so this session
will cover a ways to use apple script to
test your application thorough testing
is crucial you want to make sure that
your program works correctly before you
ship it to your customers and the only
way to do that is with with thorough
testing and making sure everything works
exactly right we like to encourage
people to use apple script to test their
applications because it has a lot of
advantages it's very flexible you can
test what you want to test you write the
test yourself you control them it's
extensible every time you add something
new to your program you can add new
Apple scripts that test those features
it can be very complete you can probe
every single corner of your application
press it into all of its boundary cases
this way it improves your accuracy you
can keep track of what the results of
your tests were from one execution to
the next and make sure that you're
getting the right answer you can repeat
those tests time after time each time
you change your application you can
repeat all your tests and prove to your
boss that you haven't broken anything
and it's controllable you had you say
what's going to be tested you can have a
small test that you run every single
time that you build the application a
larger test that you run just before you
turn it over to your testing
organization and then they can have a
massive suite of tests like I said that
probe every single corner of your app
education and the best thing is that it
all goes a lot faster than if you're
trying to do it by hand so we can cover
lots of things today you have to plan
for script ability it just doesn't
happen all by itself you're going to
want to instrument your code so that
when you're driving it with scripts you
know what's happening you can follow the
flow and make sure that that's what you
wanted to have happen you want to ensure
that you have code coverage like I said
probe into every single corner of your
application make sure that it all works
not just the most frequently used stuff
last year in session 311 we talked a lot
about using Apple GUI scripting to test
your application we say a little bit
more about that today and then the most
important part is to sustain the effort
if it's nice to have a good set of tests
but it's very important to keep them
up-to-date as your program changes and
then like I said fellas from eggplant
we're from redstone we'll be talking to
you about eggplant last year we had a
very long session about how to design a
dictionary that's section 414 if you
have the dvds you can watch it there if
not it's available on the developer
website we also promised you some
guidelines for putting together your
dictionary and Chris Neville did an
excellent job of that and those are now
available also on the developer website
so you should definitely look at the
scripting interface guidelines even if
your programs already scriptable you
should look at the guidelines because it
might it might help you out even if all
you do is go through and change some of
the comments your dictionary might be
easier to use if you're using cocoa to
do your script ability it's necessary to
align the objects in your dictionary
with the cocoa objects inside your
program even if you're not using cocoa
you're going to have to have some kind
of mapping from your dictionary to your
coat in cocoa it's very direct in
one-to-one you're going to want to
streamline this though from the users
point of view you don't necessarily want
to show them all the ugly guts of your
program you want to name things what
your pro what your users call them and
make them interact the way your users
are familiar with using them through the
gooey
if you absolutely have to you can put in
a private test suite that doesn't shift
to your customers which will allow you
to get to some of the lower level things
that you know you need to test but are
not likely to be useful to the end users
and you're going to want to cover all
the functionality or program you're
going to want to make sure that there's
some way to test and get out everything
your program does that might be a pretty
daunting task at first so we encourage
you to go ahead and say is this
operation and each release add a little
bit more script ability than the time
before and that way you'll get there at
least in a couple of steps to know what
your code is doing inside it's got to be
instrumented and you can do this lots of
different ways if your company are
standardizes on a particular kind of
profiling software then of course you
should use that but it's not there's
other ways to do it first thing even
want to do is build a skeleton of your
code if your application already exists
you're going to skip this part but if
you're writing a new application you're
going to need cocoa objects for each of
your each of the objects in your Apple
script dictionary you're going to have
to have accessors for all the properties
of those objects and you're going to
have to have accessors for all the
elements that can be inside of those
objects if you have commands in your
dictionary you're going to need methods
to handle those as well if the command
is directed at a particular object such
as send mail then the method goes on the
mail object if if it's directed towards
the application itself such as sleep
then that message is going to be coded
slightly differently and it's going to
attach to the application rather than
one of your objects the you're going to
need to add some kind of instrumentation
to your code what I do is really very
simple I want to create a record of
where my programs been I want to know
every routine that I've called maybe
what the input looked like and maybe
whether it fast or failed so you're
going to want to create a record of
where you've been you're going to want
to log any relevant error conditions
that occur if the files supposed to open
it doesn't you're going to want to log
that if you've got problem with leaks in
your program you can track allocation
and de-allocation and make sure that
everything balances and like I said it
doesn't have to be fancy this is what i
use it's just a couple of macros and if
you set that switch from 0 to 1 all of
these macros expand when you run an
apple script you'll get a very extensive
list of what your programs done and then
if you set the switch back to zero and
recompile it all just goes away so it's
not part of the program that you ship
when you look at these at these logs
you're going to do a couple of things
you're going to have to understand why
your code did what it did and justify it
you don't want to be running off into
the weeds and doing a lot of
confrontation that's not necessary if
you generate an error condition you're
going on and investigate why you might
be testing an error condition so you're
doing it on purpose or you might be
trying to like I said open a file and
for some reason it's not there you're
going to want to balance all of your
allocation and de-allocation and make
sure that you're not leaking and you're
going to want to retain a record of all
of this so that when you repeat the test
later you can make sure that it did the
same thing the same way or it didn't do
the same thing the same way and that's
the fix to the problem and this is an
example of what I see when when I run
one of my programs with the logging
turned on I've got my initials in there
because it gives me a nice string to
search for that's not likely to occur
anywhere else it lets the people i work
with know who it was that put this log
message in there in case they want to
know what it's about and this here is
fetching the startup disk from the
system events program it calls the
startup disk access or method and then
it creates a disk object a disk appens
to be a node and no it happens to be an
item an item happens to be an element
and you can see all of those things
happening and then at the end of the
program it's going to dispense with all
of these objects
okay you're going to want to ensure code
coverage it doesn't do you any good to
test part of your program your users are
going to find the parts that don't work
so you want to set everything but it's
like I said it can be a very daunting
task to do this right away especially
with a large existing program so go
ahead and stay is the implementation do
something useful and meaningful in your
next release and then add to it each
time you rev your program and there are
ways to exercise code without
necessarily exposing it in your
applescript dictionary like I said
before you don't want to expose all the
inner workings you want to want to keep
some of that private but there are
certain operations that you can do that
will cause those inner workings to be
exercised you want to make sure that you
make that part of your test and if
there's no other way to do it you can
put a private test suite in your program
and just dispense with it just before
you ship it we would rather you leave
your scripting in place for the end
users to use as well but if there's
something that's just too nasty you can
hide it from them you're going to want
to test all the corners of your program
you want to test trivial things things
that you don't expect to work edge
conditions really large numbers really
small numbers you're going to want to
force all the errors what happens when
you tell your program to open a file
that doesn't exist and you want you want
to make sure that the user the end user
is going to receive a meaningful
response that token something's going
along and you need to change your script
and one thing that applescript can't do
particularly well is get at the glue
code that you write to bind your view
and your model objects together
applescript drives your model objects
more or less directly if you've got a
lot of glue code that stages things back
and forth between your model objects and
your view objects you're going to need a
way to test that one way is to write
less code if you use something like
cocoa bindings you'll write a lot less
glue code and you'll have to do a lot
less test and you don't have to test the
code that you don't write
Apple's GUI scripting we talked about
last year it can it can be used to do a
lot of things mostly it can access
things that are not scriptable to begin
with and allow you to get past blockages
in your workflows there are several
things that does particularly well it
does exercise all of your glue code
since it does come in through the view
it's going to exercise your blue coat
and if that's what you're trying to do
GUI scripting can be a good way to do
that it recreates end user scenarios
exactly if you've got a report from the
field that if I select this menu and
then this menu and then press this
button something bad happens then you
can recruit you can recreate that
scenario exactly in your test and see
what it is that the users talking about
and then repeat that test over and over
again if you're doing a phased
implementation of Apple script in your
program you're still going to want to
catch the rest of it and you can use GUI
scripting to do that you don't have to
write any code for GUI scripting to work
you just have to turn on accessibility
and if you're testing your program
specifically for 508 compliance there's
a tool that's going to be introduced
here this week for testing accessibility
compliance you can also use GUI
scripting to do that yourself starting
right away you're going to need to use
GUI scripting if your workflow depends
on other applications that you don't
control perhaps your application
interacts with an application written by
another company and that application
either isn't scriptable or some key part
of it that you need isn't scriptable you
can use GUI scripting to get past that
this is what GUI scripting was invented
for for unblocking workflows for
scripting the unscripted ball and it may
be what your customers are using if your
applications not fully scriptable yet
you're your end users may have worked
around that by writing GUI scripts of
their own if they've integrated those
into a mission-critical workflows you're
going to have to make sure you don't
break those things before you ship a
revision of your code it gooey scripting
has got some limitations though it
shouldn't be used as a substitute for
your own script ability
it's easy to get started on GUI
scripting but it's hard to maintain one
of the one of the reasons is a lot of
applications the view layer has a
complicated containment hierarchy
there's controls inside of controls
inside of controls and if you ever
revise your application relay out any of
the screens then you're going to have to
also revised all of your GUI scripts as
well and whatever it is you're trying to
do is going to be bounded by the GUI you
might not be able to generate the stress
conditions that you'd like to just
through the GUI and if that's the case
you're going to want to use object model
scripting to do that the most important
part i said is sustaining the effort
it's easy to get started add a little
bit of script ability or program be very
happy with it use it to do some
important testing and then set it aside
you'll revise your program every year
every couple of months when you add that
functionality you're going to want to
add testing for it and the way to add
vesting is to add script ability at the
same time whenever you do revise your
program you're going to want to run all
of your prior tests everything that
you've ever done make sure that you
haven't regressed anything make sure
that you haven't broken anything so you
should hang on to all of your tests and
you should hang on to the results of all
of your tests so you can run them over
again and make sure they do the same
thing each time and when you add bug
fixes to your program or you get reports
from the field from end users you're
going to want to write test cases that
cover that make sure that the end user
has what he needs you going to put those
in your in your testing suite as well
and repeat them every single time you
revise your program so these are the
things that I talking about today got a
plan for script ability it doesn't just
happen we're going to instrument your
code so you know what happened and why
you've got a probe into all of the
corners of your code so that your only
cyllage you're testing all of it instead
of just part of it you want to use GUI
scripting where it's appropriate but
you're going to want to use in your own
model object scripting to the greatest
extent possible and you want to keep
that over and over again don't give up
and I like to thank you and turn this
over to
Doug Simon's from redstone and he's
going to talk about eggplant thanks John
good to see everyone this morning I'm
Doug Simon's one of the principal
developers at the red stone software and
as John said we'll be talking about our
eggplant to testing tool we also want to
help we also want to give you a little
bit of an overview of some different
approaches you might want to consider in
testing your application for each type
of testing we're talking about so we'll
look at what are the goals of testing
and what value automation brings to that
process and we'll try and give you some
concrete examples of each of these to
give you some idea of how you want to
spend your testing efforts so as I said
we'll talk about eggplant eggplant is a
an interface level testing tool and
we'll give you some idea of how that
works because we using that in some of
the examples that we're presenting some
of the different kinds of testing that
we're going to talk about this morning
our unit testing which is to test your
code at a low level look at functional
testing which is a testing how a user
might interact with your application and
through youth cases stress testing you
want to test your application in depth
and really push it to its limits in a
number of different ways performance
testing to measure how fast your
application is running and integration
testing to test your application and a
broader picture of the context where
we'll be running give you a little bit i
do some of the types of information and
results you might be looking to guests
from your testing process and finally
we'll take a look at some additional
uses of automation beyond testing let me
tell you a little bit about how eggplant
works is that as an interface level
testing tool it's in fact allows allows
you to drive your application exactly as
a user would so how does that work well
it's a two computer system
eggplant runs on your Mac os10 machine
and connects over a tcp/ip network to
any computer running a VNC server how
many of you have used VNC or heard of it
yeah quite a few VNC is a neat program
that it's an open source software that
allows you to access a computer remotely
people use it to access their home
computer from work or vice versa and
eggplant has a VNC client built into it
which allows us to access any computer
running a VNC server which is pretty
much any computer out there the VNC
process gives eggplant of view of the
screen of the other computer and allows
you to control the keyboard and the
mouse to that eggplant adds scripting
which provides automation of course we
have a scripting language called sent
stock which is a very easy to understand
English like language that gives you all
the control that you need to develop
some sophisticated tests and so the end
result is that eggplant can drive any
software it doesn't matter what language
it's written in since we're driving
directly through the user interface and
can be running on any operating system
let's talk a little bit about the unit
testing now the goal here is to verify
the correctness of your application at
the code level ordinarily unit tests are
written to test an individual function
or method within your application and in
recent years a number of agile
development methodologies have become
popular such as extreme programming any
of you extreme programmers or agile
developers not too many in this crowd
but in the one common components too
many advil approaches to a software
development is a test driven development
and the idea of test-driven development
is too for developers to write tests to
verify you know their code and quite
often as the tests are written before
the code and then they you can develop
the code to satisfy those tests
there are some good open source free
tools for doing this kind of testing or
the first one was j unit which is used
for doing unit testing of java
applications and since then there have
been a lot of other tools of a similar
sort there's one called o SI units for
example to do unit testing of
objective-c code that we use for our
unit testing applescript the course can
also test your code at a fairly low
level interacting either with the
individual modules or testing the model
of your application and John did a good
job of covering a lot of the benefits
that automation brings in terms of
repeatability and young consistency one
of the other things that's a great
advantage course of automation is that
so you can build these unit tests into
your build process providing a continual
feedback to the developers as they're
developing the application and coming
you know revising everything so that can
be a big benefit functional testing is a
little bit higher level testing now
we're looking at how users interact with
the system and the goal here is to
verify the requirements of your
application most application are
develops beginning with some
requirements but to tell you what it is
that the application is supposed to be
able to do and frequently these
requirements are written up in the form
of use cases which are individual
scenarios of how a user would walk
through some particular process within
the application a lot of functional
testing is done manually and frequently
this is done by developing test cases
that are modeled after those use cases
again just so the manual tester can walk
the application through some particular
sequence and make sure that it's
performing the way that it should using
an interface level tool such as eggplant
you can automate those kinds of tests
and this allows your regression test to
be run again after every build for your
functional test and very much the same
way that your unit tests are run I'd
like to invite my colleague Jonathan
Gillespie up he's going to be doing the
demos today and we're going to show you
how we would use a eggplant to do some
functional testing thanks Doug good
morning as our first demonstration I'm
going to be showing you an example as
doug said of a simple functional test
script but for those of you who haven't
seen our product eggplant before I also
want to just give you an idea of how
easy it is to start writing test scripts
using that so as Doug mentioned expat
works by connecting to a remote machine
so let's go ahead and start by
connecting to my test system over here
and you can see you can see here that
I'm controlling the remote machine
directly sending it keyboard and mouse
events just like this so if we wanted to
create let's say we had a simple use
case that we wanted to create a
functional test form for my
demonstration my first demonstration
this morning I'm going to go ahead and
just do a little simple use case using
Apple's iTunes application so here you
can see we've just got a real simple use
case we want to go ahead and launch
i-tunes we want to search the iTunes
library for a particular song we want to
play that song and when that song
finished we want to go ahead and quit
itunes just a typical use case scenario
so let's see how we'd go about scripting
that using eggplant the first thing we
need to do is create a test suite a
suite in eggplant is just a collection
of scripts and other resources necessary
to run a text there's a sweet and let's
go ahead and create our first simple
functional test
now I could just start writing a test
script right here using eggplant
scripting language sense talk but
instead let me take advantage of
eggplants script generation mode that
makes it real easy to write a test so I
do that by switching from live mode into
capture mode now I'm no longer
interacting with the remote machine
directly instead I have a selection
rectangle that I'm using to identify
elements of the remote machine that I
want to work with graphically so the
first the first step in our use case was
we wanted to launch i-tunes so we're
going to go ahead and do that the exact
same way a user would by clicking on the
itunes application down here in the dock
so I select the items application I tell
eggplant using the toolbar that I want
to click on that image it can't brings
up a save image dialog and I go ahead
and name the image and save it and when
that happens when I do that three things
happen the first is that eggplant stores
that image over here in our test suite e
itunes icon the second thing that it
does is it goes ahead let me make that a
little bigger so you can all see it
second thing is it does is that actually
writes a comment that script command
into our script and the third thing is
it does is it actually execute to that
line click the itunes icon and when that
line is executed what eggplant does is
actually searches through the remote
screen to find that to find that image
and when it finds it it clicks on it you
can see here is that it's brought up the
remote screen or brought up itunes on
the remote system so the next step in
our youth cases want to search the
library for a particular song as a user
we do that by clicking and entering
something up here in the search field
and we identify the search field by this
search label right here because we don't
want to click directly on the search
label so eggplant allows us to set what
we call a hot spot so that we can
interact it's two positions relative to
an image so I'll just move that hot spot
directly above the search label and
again tell eggplant we want to click
there in the search field
you can see we have it in you can see we
have an insertion point now and we can
send keystroke commands to the remote
machine just as easily so let's go ahead
and see if there are any eggplant songs
over here in my itunes app and sure
enough there's one so the next step in
our use case is to go ahead and play
that song so we just do that again the
same way a user would by clicking on the
play button
[Music]
you said a lot now oh man that a
staccato
tell hungry haha and Rachel
[Music]
so we wanted to continue on with our
youth case we want to make sure to wait
for that song to end itunes goes ahead
and changes the play button to a pause
button while the song is playing so what
we want to do is wait for that pause
button to return back to a play button
that's how we know the song has finished
again just like a user does so X
plankton waits for certain elements to
appear on the remote screen using the
wait for command the way for command
allows us to specify a maximum period of
time for a particular to em for a
particular image to show up in this
example we'll go ahead and just give it
90 a maximum of 90 seconds if the image
doesn't appear in that time it's going
to it's going to quit it raise an
exception and fail the script mmhmm yeah
I'm just naming these images as I record
the script yes actually if you look over
here in our test script we can actually
see all these images being recorded I
could have actually reused the play
image from before and that would have
been a better way to go so then the last
step in our script is to go ahead and
quit the iTunes application and we'll do
that here through the menu
so here we've written our first simple
functional test script using eggplant
and I can go ahead and run it here now
one two three four five Oh to hell Alex
are you said a lot now oh man that ate
Chicago still hungry and Rachel
so that's our first demonstration on
functional testing using eggplant thanks
Jonathan we have slides again okay let's
talk a little bit now about the stress
testing functional testing and unit
testing are great to ensure that your
application is is working and doing what
it's supposed to do what it's designed
to do but you also we're going to want
to push your application a little harder
and see if you know as John mentioned
you know push put in that some
unexpected conditions some things that
are likely to cause errors touch some of
the boundary conditions there in your
application you don't really expect your
user to enter negative numbers perhaps
for a duration of time or something but
that you know they will eventually and
so you want to be sure to test some of
those exceptional situations so the goal
here is is not just to verify that your
application is doing what it's designed
to do but that it's not doing what it's
designed not to do and so you want to
want to push it a bit there and so the
goal here is really to find bugs and
track them down to their scores if you
can automation it plays a really
critical role here it's a always of
course it's very nice to be able to
automate things to save some time and
effort but when it comes to to this kind
of stress testing you really want to be
able to do things repeatedly and scripts
are do that a lot better than people do
so you want to want to be able to
iterate over large sets of data and to
try your application with different
combinations of input and also to be
able to run tests repeatedly to
reproduce any intermittent crashes you
might have how many of you ever had a
bugs or reports of bugs that were very
hard to reproduce they happen and in
fact we had one at eggplant a few months
ago and people ask us sometimes whether
we use eggplants
eggplant and the designer of fact we do
we have a test we call our EP squared
test or eggplant squared in which
eggplant is driving another machine
which is also running eggplant to create
and run from tests on a third machine
and when we had this occasional crash
showing up we were able to run the
eggplant squared test repeatedly
overnight it takes about two three or
four minutes for each time through the
test and not something that a person
would want to sit there and do 100 times
over but by doing this we were able to
find that this particular crash occurred
once or twice every 100 run and we were
able to track down where it was
occurring and isolated and fix the
problem so I'd like to ask Jonathan to
give us a little demo of a data-driven
test thanks again as Doug and John
Comiskey beforehand pointed out many
times when you're looking to stress test
your application you want to try
providing large amounts of varied data
and sometimes you want to try doing the
exact same actions over and over again
so for this for this demonstration I've
created a simple little data file here
and I'm going to use this data file to
drive the address book application to
just create some create some records in
address book and and then look up the
phone numbers once we've done that so
let me bring up this script here I'll
try and make that a little bit larger
there we go
so I'm actually gonna walk through the
script using a new feature of our
upcoming release of eggplant which is
actually an interactive debugger so when
I run the script it's going to pause
here after the pod script command the
first line of the in the script is to
call another script here you see that
eggplant is fully modular we can call
other scripts passing parameters this
particular script actually will will
launch an application on the remote
system using the finder let's step
through that or actually i'll continue
through that we can watch it
so here we are in the address book
application we have a it's very easy to
parse data files and work with data in
our centric scripting language all we
need to do there is create a simple
repeat loop with a simple construct like
repeat with each line of file my data
file then inside that repeat loop we
have a number of interactive command
similar to the first script we did we're
just hitting command and to create a new
card we're typing in the first name from
the data file than the end then the last
name and so on entering entering all
these two of user interface so we've
gone ahead and entered all five of our
records for this data test and now like
I said we want to go ahead and validate
the information that that eggplant
entered so we'll go ahead and do that by
just stepping in here and using the find
feature of the address book again very
similar just interactive commands with
the remote system so I'll get go ahead
and let that continue running a really
important point I'd like to make here is
that this is all going directly through
the user interface just the way users
would our eggplants back to our eggplant
squared bug the explants squared bug was
in it was a bug with the interface of
our application it was a problem with
some thread timing we had which is one
of the reasons it only showed up
occasionally and testing it from doing
our unit tests and our model tests
didn't expose this bug but running it
through eggplant through yet through the
interface did this becomes increasingly
you know more important with the faster
machines like the g 5s and even more
important with multiprocessor machines
that are really likely to expose
threading problems that you might have
so that's a basic example of how you
would use eggplant with some data to do
some data-driven testing and very the
input and results I clarify one
that's a slide again thanks outside and
take a little look at performance
testing now performance testing of
course is designed to measure the speed
of your application how fast it's
running and quite often developers when
they think about performance testing are
going to be concerned with the speed of
their code and you know the cool new
algorithm that they've devised to ring
the most out of their g5 processor but
there's another type of performance is
important also and that's the
responsiveness of the application as a
user is using it and this is very
important of course because you want
your users to be happy with your
application and not frustrated those
things are proceeding so for example you
might have a search feature in your
application you could test that go with
an eggplant script that clicks on the
search button and then wait for up to 30
seconds perhaps or whatever the next
thing is to show up on the screen to
indicate that the searches has finished
and this is fine but 30 seconds was
going to be awful long time for your
users to wait if it's actually taking
that long and this script will only fail
if the in fact it takes longer than 30
seconds by adding the lines shown in
blue here you could check to see how
long it really takes that search to come
back and if it's greater than 5 seconds
and you could log a warning message to
indicate that there's a problem you may
want to look into in your application so
of course in the case where you're
testing performance it's it's really
important to be able to gather
consistent and repeatable timing data
that you can compare from one build of
your application to the next to see if
you're in fact slowing down or have made
some improvements their final type of
testing we'd like to talk about today is
the integration testing the goal here is
that to really ensure a quality
experience for your users your
application may work just fine for your
use cases and you know your
the tests and all that but chances are
nowadays your application doesn't live
on its own it's going to be interacting
with other applications it has to be
able to go out and live in the real
world so it's really important that you
test it in that kind of an environment
you may also wants to test a complete
process end-to-end if your application
for example produces data that that it
can export that can then be read into an
Excel spreadsheet perhaps or quicken or
something like that you're going to want
to test not only that your your program
runs but that's the data that it
produces can be read in those other
applications and looks the way it should
you also may have different
configurations that that you want to
test perhaps your program has some
plugins or other modules that are
optional and you're going to want to
test the various combinations of things
and perhaps to run all of your tests and
each of these different configurations
and finally if you have a cross-platform
application obviously going to want to
test it on different platforms so that
it runs on perhaps you have a web-based
application there you need to not only
test on different platforms but in each
of the different browsers that you
support on those platforms even if you
don't have a cross-platform application
you may want to test your program
running on on different types of
hardware you know different to different
macs and so this is a place where where
a remote testing approach like eggplant
offers can come in handy perhaps you
have a g5 that's in a different
department at your company and you'd
like to be able to run your tests
against that when you reach this stage
you may have hardware at the vendors
site even that you'd like to run your
tests on or perhaps at the Apple ADC
compatibility lab in cupertino and that
can be done remotely obviously again the
automation provides consistent and
repeatable results if you're running
your tests automatically you can
you can do much more testing than you
would if you only had manual testers one
of the big one of the big wins obviously
was automation always is that you're
eliminating a lot of your manual testing
time and giving a much more efficient
execution one of our customers develops
tape backup software that runs on unix
systems and works a lot of different
tape drives from different manufacturers
every time they get a new new new model
of tape drive from one of the vendors or
want to revise their application they
have to validate that everything is
still working on all of these different
devices and on the different operating
systems that they support and this when
they had a manual process to do this
took them about two weeks to go through
all of their validation tests with
eggplants they were able to write a
script that runs in eight hours and by
being able to to run their tests over
night like this of course they not only
stables and a huge amount of time and
effort but we're actually able to modify
their their approach to developing their
software because they you can you can
imagine with a two-week testing cycle at
the end of every build you're not going
to be able to innovate quite as quickly
so Thompson's going to show us a kind of
an interesting integration test now on
the general machine thanks so for this
next script let's say that my company
wants to offer a free trial license of
its software 12 WWDC attendees we want
to allow people to you know come to a
website fill out a web form get a trial
license key download the software put in
that license key run the application and
test interactions as if that our
software has with the operating system
we can validate this entire process
using something like eggplant so let's
go ahead and just look at this
integration script at a real high level
here
again we're going to reuse our launch
application command to go ahead and
launch up Safari and then we're just
going to go through that entire process
we're going to go test the down the free
trial download form by going to URL and
filling out the form and then grabbing
the license key off the site then then
we want to go ahead and download the
application using Safari make sure that
downloads and the disk image unpacks and
installed correctly then we want to go
ahead and move on and launch our
eggplant application and put in the
license key that we got from the website
then you could imagine we would actually
run a whole battery whole series of
functional use case tests like we like
we did in the in the first demonstration
but in particular we want to do things
like test its interactions all of
eggplants documentation is written as
PDF so we want to go ahead and make sure
it's if they all open up properly and in
in the preview application and it can be
read and seen and finally we want to go
ahead and clean up and quit out of that
and go ahead and even test the
uninstallation process so we can go
ahead and do all of that with eggplant
so I'm going to get started here
ok
and just run the script and we can just
sort of watch it go through its paces
again driving driving various components
all on that remote system an integration
test like this could it also be a multi
multi system test we could initiate one
process on a machine quick close this
connection open a connection to another
machine and start processing there dear
eggplants to automatically agree so as a
licensing agreement panel that comes up
here we're launching the application and
it says it needs a license so we'll plug
in the one that we pulled off the
website again we could run a bunch of
eggplant tests now or we can just go and
check and make sure that the
documentation is coming up correctly so
there it is the bookmarks and everything
so having moved moves through all that
we'll go ahead and just clean up clear
out the license and quit and close
eggplant so there we've just there were
seeing an example of an of an intense
process test of all the steps necessary
for it for a full integration thanks
Jonathan that was great I think you can
see the power of automation there in
about a minute and a quarter we were
able to do all of that which is
something that you'd like to be able to
validate after you're ready to ship your
application to be sure that it's all
ready to go so some of the information
that you might be looking to get as a
result of your testing of course the
first thing is that you you're looking
for bugs you want to find any bugs that
you might have in your application and
be able to see what those are project
managers usually are also looking to
gather some statistics about how many of
your functional tests are passing or
failing at any given point in time and
hopefully reducing the failures
throughout the development cycle if
you're doing some performance testing if
that's so important for your application
of course you're going to want to gather
some some timing metrics there and so
clearly that's an important thing
another really important piece though is
to provide some feedback to developers
and I don't know how many of you who are
developers may have had situations where
testers tell you all this is the thing
crashed here or something didn't work
and but but they're not very specific
always about exactly how that occurred
or on the other side of course the
testers are sometimes frustrated that
they tell the developers that something
doesn't work and they're not always
believed so being able to provide clear
communication of exactly what the steps
words that led up to a crash or a bug
can be really really valuable and that's
one of the key things that automation
can provide by having your test
automated and logging the information as
it goes it can show you exactly what the
steps were and of course you can
reproduce that for the developers and of
course the automation also is really
critical for being able to reliably
compare one run to another and see the
progress that you're making during the
development cycle Jonathan's going to
give us a quick view now of some results
in eggplant thanks eggplant
automatically is going to record results
any time we run a script so let's go
ahead and just look back at the results
are recorded for that last integration
demo that I ran for you so we just come
into eggplant and go over to the results
tab here you can see it automatically
records the time each time the script is
run and then it keeps statistics on any
errors or warnings or exceptions that
were raised if we actually click on that
run a little bigger you can actually see
that eggplant is actually logging every
interaction that it that it performed to
the remote system in terms of clicking
what time exempt from the action what
action it performed and even we're on
the screen that that was found and we
can step step through all of this and so
you've got a you've got a very clear
sense of exactly what was happening
during the test even down to the exact
second that it happened and we can go
ahead and double-click on any of these
lines to jump back to the script on
and see what what command caused that
interaction of course something that's
really important to know about testing
is how do errors get get reported how do
you tell when something went wrong I'm
going to go ahead and force an air
conditioned here in this integration
demo by having us go to a web page that
doesn't exist and then just go ahead and
run the script with that so it won't
take too long before eggplant tries to
move through its script but lo and
behold is the page isn't there you know
the webmaster did something bad so
eggplant comes up and says that a script
failed and now if we go back and look at
our results we see that the results for
that execution is in red and again we
can follow in follow through here
everything that eggplant was doing
including what it was trying to do when
it eventually failed it was trying to
find the name field to fill out the form
on the website any time the script fails
eggplant automatically grabs a full-size
screenshot of exactly what the screen
looked like when it failed so you can
easily communicate to the developers
exactly what went wrong and exactly what
it was what it had done beforehand so
that's just a simple look at the the
kind of logging results that are
important for for testing thanks Donovan
for money and I might also notice that
all of eggplants reporting is done
through standards text files that you
can easily import into other reporting
tools that you generate whatever kind of
reports you might need a little look now
it says some additional uses of
information beyond the just testing for
example if you you may have developed a
web-based application and you carefully
tested it you know that it all worked
it's now out in the field it's deployed
running 24 by 7 and you might like to
know that that it's still continuing to
run so one possibility would be to
create an eggplant script that goes out
so maybe once every hour or every five
minutes whatever makes sense for you and
to run some of your simple tests or it
just looks to see that the web page is
still there and
benefit if there's a problem that you
could you know write the script to
actually send you an email message let's
say to say oh something's not quite
there and didn't didn't work of course
if you have a critical application and
an email may not be the best answer so
you might want to have your script do
something a little more interesting
there's a there's a number of web
services out there where I veena is one
that we've worked with that you can
actually write a script that will then
call you up on the telephone and
interact with you know press one for if
you want to abort the thing or whatever
and gets those results back and you can
read that in your script and take action
based on that of course you may not
really want your computer calling you up
at four o'clock in the morning to tell
you that the website is down so why not
have have a script actually do something
about it instead of just telling you you
could conceivably write a script that
when it detected an error condition
would then go and log into the server
directly and restart apache or whatever
might be needed one of our customers was
interested in doing some validation of
third-party software there were big
organisation with hundreds of Windows
machine and whenever they get a new path
to windows they're interested in knowing
before they deployed that patch you know
to all their windows boxes they'd like
to know that all of the applications
that are critical to their users are
still working so they are instant using
eggplant to to do this kind of
validation of other software running on
the new version of windows before they
deployed that throughout the
organization system administrators of
course do a lot of repetitive tasks in
the maintaining systems throughout your
company if you've typically they write a
lot of scripts shell scripts in pearl or
some other scripting language which
works great for a lot of things but
again there are some things that that
can't be done that way maybe they are
only available through a user interface
we've had heard of people who needed to
update their virus software on all of
their windows machines
and that's with something that there was
a process which had a user interface it
couldn't be scripted directly so might
be very nice for a system administrator
to have a script that they should run
that would connect to each of those
machines in turn rather than having to
walk around to them one of our customers
fujifilm has taken a formerly manual
process of validating x-ray film and
automated that this is a process that is
a subject to audit I the FDA and very
critical that this be done on a regular
basis there's a variety of different
kinds of automation that that are
possible hardware and BIOS level testing
is not something maybe that would
immediately pop to mind were for
something that's a typically thought of
as driving software but in fact most
hardware of course has some sort of an
interface and subscribe an interface
then eggplant can interact with it and
drive it we even have one customer who
is doing BIOS level testing there's a
hardware kvm switch that supports VNC
and they're able to use that to have
eggplant drive the machine even at boot
time we also have come up with an idea
of creating movies so we built in a
feature into eggplants that allow you to
capture a QuickTime movie of exactly
what's going on on the remote screen
either when you're interacting with it
live or under script control since you
are already developed a test now to walk
your applications through the various
use cases of how users are going to
interact with your system might be kind
of nice to be able to capture that as a
movie and present that as part of your
documentation and going to have a quick
demo now moving just really quickly for
our final demonstration here's a little
script that I've written that actually
records a QuickTime movie of that
integration demo that we've been working
with the first thing is it does it just
sets up some timing parameters and this
just slows down the script to make the
movie easier for a user to follow
exactly what's going on
and then we just say start movie and go
ahead and call the script that we've
already been working with I'm not going
to run it in and have you watch the demo
again but i'll go ahead and show you the
results of having created this movie
just before
so again it's it's a process we've
already gone through for testing and
validating this but but if we want to go
ahead and create a simple interactive
movie to show users how to do how to do
something within our application it's
real easy using using eggplant to just
to just throw a little wrapper around it
like this and and create a little quick
30 seconds to minutes minute loan
QuickTime movie that you can put up on
your website one of the nice things
about creating a movie using a script of
course is that if something changes in
your application or you decide there's a
little different way you'd like to
present that process your users you can
easily just modify your script and
recreate the movie it's a very quick
process a lot easier than then editing a
movie file so this is some up we've
talked today about a number of different
types of testing and the kinds of value
that automation can bring to those unit
testing which is very important to
validate your code as the at the low
level functional testing to test your
use cases how the user is going to be
interacting with it stress testing is
very important to bring out any bugs
that may be hiding in the application
and performance testing of course to
make sure that if the running as an
unacceptable speed integration testing
is often overlooked it's a one of those
things though that it really impacts the
experience that your users have with
your application when they when they get
it we also talked a little bit about
some of the kinds of results and
information that you might be looking
for from your testing process and have a
quick look at some different uses of
automation beyond testing so I'll to
spend some information that you you can
use to go home and improve your own
testing processes and turn out some
great applications for Mac OS 10
it's our contact information here and we
also have a boost downstairs in the
exhibit area would be very happy to talk
to you later today or tomorrow there's
also a free download of eggplants that's
available to attendees of WWDC this year
there's a the URL is included in the
information that's available as a part
of this session or you can come come
talk to us in the booth and like 20 yes
there's a also some more information
here about additional information that's
available through the apple website