WWDC2004 Session 301

Transcript

Kind: captions Language: en good morning everybody welcome to session 301 automated testing on Mac OS 10 I'm John Mont Brian from apple developer relations and I'm pleased to introduce your host John Comiskey will be speaking to you he's a senior engineer from applescript engineering and welcome Jon Missy thank you good morning thank you for being here on John Chris gamma engineering the applescript group I'm going to talk to you today for about 15 minutes about using Apple script to test your application and then some gentlemen from Redmond from red stone software will be talking about eggplant much more high-powered way of testing your application so this session will cover a ways to use apple script to test your application thorough testing is crucial you want to make sure that your program works correctly before you ship it to your customers and the only way to do that is with with thorough testing and making sure everything works exactly right we like to encourage people to use apple script to test their applications because it has a lot of advantages it's very flexible you can test what you want to test you write the test yourself you control them it's extensible every time you add something new to your program you can add new Apple scripts that test those features it can be very complete you can probe every single corner of your application press it into all of its boundary cases this way it improves your accuracy you can keep track of what the results of your tests were from one execution to the next and make sure that you're getting the right answer you can repeat those tests time after time each time you change your application you can repeat all your tests and prove to your boss that you haven't broken anything and it's controllable you had you say what's going to be tested you can have a small test that you run every single time that you build the application a larger test that you run just before you turn it over to your testing organization and then they can have a massive suite of tests like I said that probe every single corner of your app education and the best thing is that it all goes a lot faster than if you're trying to do it by hand so we can cover lots of things today you have to plan for script ability it just doesn't happen all by itself you're going to want to instrument your code so that when you're driving it with scripts you know what's happening you can follow the flow and make sure that that's what you wanted to have happen you want to ensure that you have code coverage like I said probe into every single corner of your application make sure that it all works not just the most frequently used stuff last year in session 311 we talked a lot about using Apple GUI scripting to test your application we say a little bit more about that today and then the most important part is to sustain the effort if it's nice to have a good set of tests but it's very important to keep them up-to-date as your program changes and then like I said fellas from eggplant we're from redstone we'll be talking to you about eggplant last year we had a very long session about how to design a dictionary that's section 414 if you have the dvds you can watch it there if not it's available on the developer website we also promised you some guidelines for putting together your dictionary and Chris Neville did an excellent job of that and those are now available also on the developer website so you should definitely look at the scripting interface guidelines even if your programs already scriptable you should look at the guidelines because it might it might help you out even if all you do is go through and change some of the comments your dictionary might be easier to use if you're using cocoa to do your script ability it's necessary to align the objects in your dictionary with the cocoa objects inside your program even if you're not using cocoa you're going to have to have some kind of mapping from your dictionary to your coat in cocoa it's very direct in one-to-one you're going to want to streamline this though from the users point of view you don't necessarily want to show them all the ugly guts of your program you want to name things what your pro what your users call them and make them interact the way your users are familiar with using them through the gooey if you absolutely have to you can put in a private test suite that doesn't shift to your customers which will allow you to get to some of the lower level things that you know you need to test but are not likely to be useful to the end users and you're going to want to cover all the functionality or program you're going to want to make sure that there's some way to test and get out everything your program does that might be a pretty daunting task at first so we encourage you to go ahead and say is this operation and each release add a little bit more script ability than the time before and that way you'll get there at least in a couple of steps to know what your code is doing inside it's got to be instrumented and you can do this lots of different ways if your company are standardizes on a particular kind of profiling software then of course you should use that but it's not there's other ways to do it first thing even want to do is build a skeleton of your code if your application already exists you're going to skip this part but if you're writing a new application you're going to need cocoa objects for each of your each of the objects in your Apple script dictionary you're going to have to have accessors for all the properties of those objects and you're going to have to have accessors for all the elements that can be inside of those objects if you have commands in your dictionary you're going to need methods to handle those as well if the command is directed at a particular object such as send mail then the method goes on the mail object if if it's directed towards the application itself such as sleep then that message is going to be coded slightly differently and it's going to attach to the application rather than one of your objects the you're going to need to add some kind of instrumentation to your code what I do is really very simple I want to create a record of where my programs been I want to know every routine that I've called maybe what the input looked like and maybe whether it fast or failed so you're going to want to create a record of where you've been you're going to want to log any relevant error conditions that occur if the files supposed to open it doesn't you're going to want to log that if you've got problem with leaks in your program you can track allocation and de-allocation and make sure that everything balances and like I said it doesn't have to be fancy this is what i use it's just a couple of macros and if you set that switch from 0 to 1 all of these macros expand when you run an apple script you'll get a very extensive list of what your programs done and then if you set the switch back to zero and recompile it all just goes away so it's not part of the program that you ship when you look at these at these logs you're going to do a couple of things you're going to have to understand why your code did what it did and justify it you don't want to be running off into the weeds and doing a lot of confrontation that's not necessary if you generate an error condition you're going on and investigate why you might be testing an error condition so you're doing it on purpose or you might be trying to like I said open a file and for some reason it's not there you're going to want to balance all of your allocation and de-allocation and make sure that you're not leaking and you're going to want to retain a record of all of this so that when you repeat the test later you can make sure that it did the same thing the same way or it didn't do the same thing the same way and that's the fix to the problem and this is an example of what I see when when I run one of my programs with the logging turned on I've got my initials in there because it gives me a nice string to search for that's not likely to occur anywhere else it lets the people i work with know who it was that put this log message in there in case they want to know what it's about and this here is fetching the startup disk from the system events program it calls the startup disk access or method and then it creates a disk object a disk appens to be a node and no it happens to be an item an item happens to be an element and you can see all of those things happening and then at the end of the program it's going to dispense with all of these objects okay you're going to want to ensure code coverage it doesn't do you any good to test part of your program your users are going to find the parts that don't work so you want to set everything but it's like I said it can be a very daunting task to do this right away especially with a large existing program so go ahead and stay is the implementation do something useful and meaningful in your next release and then add to it each time you rev your program and there are ways to exercise code without necessarily exposing it in your applescript dictionary like I said before you don't want to expose all the inner workings you want to want to keep some of that private but there are certain operations that you can do that will cause those inner workings to be exercised you want to make sure that you make that part of your test and if there's no other way to do it you can put a private test suite in your program and just dispense with it just before you ship it we would rather you leave your scripting in place for the end users to use as well but if there's something that's just too nasty you can hide it from them you're going to want to test all the corners of your program you want to test trivial things things that you don't expect to work edge conditions really large numbers really small numbers you're going to want to force all the errors what happens when you tell your program to open a file that doesn't exist and you want you want to make sure that the user the end user is going to receive a meaningful response that token something's going along and you need to change your script and one thing that applescript can't do particularly well is get at the glue code that you write to bind your view and your model objects together applescript drives your model objects more or less directly if you've got a lot of glue code that stages things back and forth between your model objects and your view objects you're going to need a way to test that one way is to write less code if you use something like cocoa bindings you'll write a lot less glue code and you'll have to do a lot less test and you don't have to test the code that you don't write Apple's GUI scripting we talked about last year it can it can be used to do a lot of things mostly it can access things that are not scriptable to begin with and allow you to get past blockages in your workflows there are several things that does particularly well it does exercise all of your glue code since it does come in through the view it's going to exercise your blue coat and if that's what you're trying to do GUI scripting can be a good way to do that it recreates end user scenarios exactly if you've got a report from the field that if I select this menu and then this menu and then press this button something bad happens then you can recruit you can recreate that scenario exactly in your test and see what it is that the users talking about and then repeat that test over and over again if you're doing a phased implementation of Apple script in your program you're still going to want to catch the rest of it and you can use GUI scripting to do that you don't have to write any code for GUI scripting to work you just have to turn on accessibility and if you're testing your program specifically for 508 compliance there's a tool that's going to be introduced here this week for testing accessibility compliance you can also use GUI scripting to do that yourself starting right away you're going to need to use GUI scripting if your workflow depends on other applications that you don't control perhaps your application interacts with an application written by another company and that application either isn't scriptable or some key part of it that you need isn't scriptable you can use GUI scripting to get past that this is what GUI scripting was invented for for unblocking workflows for scripting the unscripted ball and it may be what your customers are using if your applications not fully scriptable yet you're your end users may have worked around that by writing GUI scripts of their own if they've integrated those into a mission-critical workflows you're going to have to make sure you don't break those things before you ship a revision of your code it gooey scripting has got some limitations though it shouldn't be used as a substitute for your own script ability it's easy to get started on GUI scripting but it's hard to maintain one of the one of the reasons is a lot of applications the view layer has a complicated containment hierarchy there's controls inside of controls inside of controls and if you ever revise your application relay out any of the screens then you're going to have to also revised all of your GUI scripts as well and whatever it is you're trying to do is going to be bounded by the GUI you might not be able to generate the stress conditions that you'd like to just through the GUI and if that's the case you're going to want to use object model scripting to do that the most important part i said is sustaining the effort it's easy to get started add a little bit of script ability or program be very happy with it use it to do some important testing and then set it aside you'll revise your program every year every couple of months when you add that functionality you're going to want to add testing for it and the way to add vesting is to add script ability at the same time whenever you do revise your program you're going to want to run all of your prior tests everything that you've ever done make sure that you haven't regressed anything make sure that you haven't broken anything so you should hang on to all of your tests and you should hang on to the results of all of your tests so you can run them over again and make sure they do the same thing each time and when you add bug fixes to your program or you get reports from the field from end users you're going to want to write test cases that cover that make sure that the end user has what he needs you going to put those in your in your testing suite as well and repeat them every single time you revise your program so these are the things that I talking about today got a plan for script ability it doesn't just happen we're going to instrument your code so you know what happened and why you've got a probe into all of the corners of your code so that your only cyllage you're testing all of it instead of just part of it you want to use GUI scripting where it's appropriate but you're going to want to use in your own model object scripting to the greatest extent possible and you want to keep that over and over again don't give up and I like to thank you and turn this over to Doug Simon's from redstone and he's going to talk about eggplant thanks John good to see everyone this morning I'm Doug Simon's one of the principal developers at the red stone software and as John said we'll be talking about our eggplant to testing tool we also want to help we also want to give you a little bit of an overview of some different approaches you might want to consider in testing your application for each type of testing we're talking about so we'll look at what are the goals of testing and what value automation brings to that process and we'll try and give you some concrete examples of each of these to give you some idea of how you want to spend your testing efforts so as I said we'll talk about eggplant eggplant is a an interface level testing tool and we'll give you some idea of how that works because we using that in some of the examples that we're presenting some of the different kinds of testing that we're going to talk about this morning our unit testing which is to test your code at a low level look at functional testing which is a testing how a user might interact with your application and through youth cases stress testing you want to test your application in depth and really push it to its limits in a number of different ways performance testing to measure how fast your application is running and integration testing to test your application and a broader picture of the context where we'll be running give you a little bit i do some of the types of information and results you might be looking to guests from your testing process and finally we'll take a look at some additional uses of automation beyond testing let me tell you a little bit about how eggplant works is that as an interface level testing tool it's in fact allows allows you to drive your application exactly as a user would so how does that work well it's a two computer system eggplant runs on your Mac os10 machine and connects over a tcp/ip network to any computer running a VNC server how many of you have used VNC or heard of it yeah quite a few VNC is a neat program that it's an open source software that allows you to access a computer remotely people use it to access their home computer from work or vice versa and eggplant has a VNC client built into it which allows us to access any computer running a VNC server which is pretty much any computer out there the VNC process gives eggplant of view of the screen of the other computer and allows you to control the keyboard and the mouse to that eggplant adds scripting which provides automation of course we have a scripting language called sent stock which is a very easy to understand English like language that gives you all the control that you need to develop some sophisticated tests and so the end result is that eggplant can drive any software it doesn't matter what language it's written in since we're driving directly through the user interface and can be running on any operating system let's talk a little bit about the unit testing now the goal here is to verify the correctness of your application at the code level ordinarily unit tests are written to test an individual function or method within your application and in recent years a number of agile development methodologies have become popular such as extreme programming any of you extreme programmers or agile developers not too many in this crowd but in the one common components too many advil approaches to a software development is a test driven development and the idea of test-driven development is too for developers to write tests to verify you know their code and quite often as the tests are written before the code and then they you can develop the code to satisfy those tests there are some good open source free tools for doing this kind of testing or the first one was j unit which is used for doing unit testing of java applications and since then there have been a lot of other tools of a similar sort there's one called o SI units for example to do unit testing of objective-c code that we use for our unit testing applescript the course can also test your code at a fairly low level interacting either with the individual modules or testing the model of your application and John did a good job of covering a lot of the benefits that automation brings in terms of repeatability and young consistency one of the other things that's a great advantage course of automation is that so you can build these unit tests into your build process providing a continual feedback to the developers as they're developing the application and coming you know revising everything so that can be a big benefit functional testing is a little bit higher level testing now we're looking at how users interact with the system and the goal here is to verify the requirements of your application most application are develops beginning with some requirements but to tell you what it is that the application is supposed to be able to do and frequently these requirements are written up in the form of use cases which are individual scenarios of how a user would walk through some particular process within the application a lot of functional testing is done manually and frequently this is done by developing test cases that are modeled after those use cases again just so the manual tester can walk the application through some particular sequence and make sure that it's performing the way that it should using an interface level tool such as eggplant you can automate those kinds of tests and this allows your regression test to be run again after every build for your functional test and very much the same way that your unit tests are run I'd like to invite my colleague Jonathan Gillespie up he's going to be doing the demos today and we're going to show you how we would use a eggplant to do some functional testing thanks Doug good morning as our first demonstration I'm going to be showing you an example as doug said of a simple functional test script but for those of you who haven't seen our product eggplant before I also want to just give you an idea of how easy it is to start writing test scripts using that so as Doug mentioned expat works by connecting to a remote machine so let's go ahead and start by connecting to my test system over here and you can see you can see here that I'm controlling the remote machine directly sending it keyboard and mouse events just like this so if we wanted to create let's say we had a simple use case that we wanted to create a functional test form for my demonstration my first demonstration this morning I'm going to go ahead and just do a little simple use case using Apple's iTunes application so here you can see we've just got a real simple use case we want to go ahead and launch i-tunes we want to search the iTunes library for a particular song we want to play that song and when that song finished we want to go ahead and quit itunes just a typical use case scenario so let's see how we'd go about scripting that using eggplant the first thing we need to do is create a test suite a suite in eggplant is just a collection of scripts and other resources necessary to run a text there's a sweet and let's go ahead and create our first simple functional test now I could just start writing a test script right here using eggplant scripting language sense talk but instead let me take advantage of eggplants script generation mode that makes it real easy to write a test so I do that by switching from live mode into capture mode now I'm no longer interacting with the remote machine directly instead I have a selection rectangle that I'm using to identify elements of the remote machine that I want to work with graphically so the first the first step in our use case was we wanted to launch i-tunes so we're going to go ahead and do that the exact same way a user would by clicking on the itunes application down here in the dock so I select the items application I tell eggplant using the toolbar that I want to click on that image it can't brings up a save image dialog and I go ahead and name the image and save it and when that happens when I do that three things happen the first is that eggplant stores that image over here in our test suite e itunes icon the second thing that it does is it goes ahead let me make that a little bigger so you can all see it second thing is it does is that actually writes a comment that script command into our script and the third thing is it does is it actually execute to that line click the itunes icon and when that line is executed what eggplant does is actually searches through the remote screen to find that to find that image and when it finds it it clicks on it you can see here is that it's brought up the remote screen or brought up itunes on the remote system so the next step in our youth cases want to search the library for a particular song as a user we do that by clicking and entering something up here in the search field and we identify the search field by this search label right here because we don't want to click directly on the search label so eggplant allows us to set what we call a hot spot so that we can interact it's two positions relative to an image so I'll just move that hot spot directly above the search label and again tell eggplant we want to click there in the search field you can see we have it in you can see we have an insertion point now and we can send keystroke commands to the remote machine just as easily so let's go ahead and see if there are any eggplant songs over here in my itunes app and sure enough there's one so the next step in our use case is to go ahead and play that song so we just do that again the same way a user would by clicking on the play button [Music] you said a lot now oh man that a staccato tell hungry haha and Rachel [Music] so we wanted to continue on with our youth case we want to make sure to wait for that song to end itunes goes ahead and changes the play button to a pause button while the song is playing so what we want to do is wait for that pause button to return back to a play button that's how we know the song has finished again just like a user does so X plankton waits for certain elements to appear on the remote screen using the wait for command the way for command allows us to specify a maximum period of time for a particular to em for a particular image to show up in this example we'll go ahead and just give it 90 a maximum of 90 seconds if the image doesn't appear in that time it's going to it's going to quit it raise an exception and fail the script mmhmm yeah I'm just naming these images as I record the script yes actually if you look over here in our test script we can actually see all these images being recorded I could have actually reused the play image from before and that would have been a better way to go so then the last step in our script is to go ahead and quit the iTunes application and we'll do that here through the menu so here we've written our first simple functional test script using eggplant and I can go ahead and run it here now one two three four five Oh to hell Alex are you said a lot now oh man that ate Chicago still hungry and Rachel so that's our first demonstration on functional testing using eggplant thanks Jonathan we have slides again okay let's talk a little bit now about the stress testing functional testing and unit testing are great to ensure that your application is is working and doing what it's supposed to do what it's designed to do but you also we're going to want to push your application a little harder and see if you know as John mentioned you know push put in that some unexpected conditions some things that are likely to cause errors touch some of the boundary conditions there in your application you don't really expect your user to enter negative numbers perhaps for a duration of time or something but that you know they will eventually and so you want to be sure to test some of those exceptional situations so the goal here is is not just to verify that your application is doing what it's designed to do but that it's not doing what it's designed not to do and so you want to want to push it a bit there and so the goal here is really to find bugs and track them down to their scores if you can automation it plays a really critical role here it's a always of course it's very nice to be able to automate things to save some time and effort but when it comes to to this kind of stress testing you really want to be able to do things repeatedly and scripts are do that a lot better than people do so you want to want to be able to iterate over large sets of data and to try your application with different combinations of input and also to be able to run tests repeatedly to reproduce any intermittent crashes you might have how many of you ever had a bugs or reports of bugs that were very hard to reproduce they happen and in fact we had one at eggplant a few months ago and people ask us sometimes whether we use eggplants eggplant and the designer of fact we do we have a test we call our EP squared test or eggplant squared in which eggplant is driving another machine which is also running eggplant to create and run from tests on a third machine and when we had this occasional crash showing up we were able to run the eggplant squared test repeatedly overnight it takes about two three or four minutes for each time through the test and not something that a person would want to sit there and do 100 times over but by doing this we were able to find that this particular crash occurred once or twice every 100 run and we were able to track down where it was occurring and isolated and fix the problem so I'd like to ask Jonathan to give us a little demo of a data-driven test thanks again as Doug and John Comiskey beforehand pointed out many times when you're looking to stress test your application you want to try providing large amounts of varied data and sometimes you want to try doing the exact same actions over and over again so for this for this demonstration I've created a simple little data file here and I'm going to use this data file to drive the address book application to just create some create some records in address book and and then look up the phone numbers once we've done that so let me bring up this script here I'll try and make that a little bit larger there we go so I'm actually gonna walk through the script using a new feature of our upcoming release of eggplant which is actually an interactive debugger so when I run the script it's going to pause here after the pod script command the first line of the in the script is to call another script here you see that eggplant is fully modular we can call other scripts passing parameters this particular script actually will will launch an application on the remote system using the finder let's step through that or actually i'll continue through that we can watch it so here we are in the address book application we have a it's very easy to parse data files and work with data in our centric scripting language all we need to do there is create a simple repeat loop with a simple construct like repeat with each line of file my data file then inside that repeat loop we have a number of interactive command similar to the first script we did we're just hitting command and to create a new card we're typing in the first name from the data file than the end then the last name and so on entering entering all these two of user interface so we've gone ahead and entered all five of our records for this data test and now like I said we want to go ahead and validate the information that that eggplant entered so we'll go ahead and do that by just stepping in here and using the find feature of the address book again very similar just interactive commands with the remote system so I'll get go ahead and let that continue running a really important point I'd like to make here is that this is all going directly through the user interface just the way users would our eggplants back to our eggplant squared bug the explants squared bug was in it was a bug with the interface of our application it was a problem with some thread timing we had which is one of the reasons it only showed up occasionally and testing it from doing our unit tests and our model tests didn't expose this bug but running it through eggplant through yet through the interface did this becomes increasingly you know more important with the faster machines like the g 5s and even more important with multiprocessor machines that are really likely to expose threading problems that you might have so that's a basic example of how you would use eggplant with some data to do some data-driven testing and very the input and results I clarify one that's a slide again thanks outside and take a little look at performance testing now performance testing of course is designed to measure the speed of your application how fast it's running and quite often developers when they think about performance testing are going to be concerned with the speed of their code and you know the cool new algorithm that they've devised to ring the most out of their g5 processor but there's another type of performance is important also and that's the responsiveness of the application as a user is using it and this is very important of course because you want your users to be happy with your application and not frustrated those things are proceeding so for example you might have a search feature in your application you could test that go with an eggplant script that clicks on the search button and then wait for up to 30 seconds perhaps or whatever the next thing is to show up on the screen to indicate that the searches has finished and this is fine but 30 seconds was going to be awful long time for your users to wait if it's actually taking that long and this script will only fail if the in fact it takes longer than 30 seconds by adding the lines shown in blue here you could check to see how long it really takes that search to come back and if it's greater than 5 seconds and you could log a warning message to indicate that there's a problem you may want to look into in your application so of course in the case where you're testing performance it's it's really important to be able to gather consistent and repeatable timing data that you can compare from one build of your application to the next to see if you're in fact slowing down or have made some improvements their final type of testing we'd like to talk about today is the integration testing the goal here is that to really ensure a quality experience for your users your application may work just fine for your use cases and you know your the tests and all that but chances are nowadays your application doesn't live on its own it's going to be interacting with other applications it has to be able to go out and live in the real world so it's really important that you test it in that kind of an environment you may also wants to test a complete process end-to-end if your application for example produces data that that it can export that can then be read into an Excel spreadsheet perhaps or quicken or something like that you're going to want to test not only that your your program runs but that's the data that it produces can be read in those other applications and looks the way it should you also may have different configurations that that you want to test perhaps your program has some plugins or other modules that are optional and you're going to want to test the various combinations of things and perhaps to run all of your tests and each of these different configurations and finally if you have a cross-platform application obviously going to want to test it on different platforms so that it runs on perhaps you have a web-based application there you need to not only test on different platforms but in each of the different browsers that you support on those platforms even if you don't have a cross-platform application you may want to test your program running on on different types of hardware you know different to different macs and so this is a place where where a remote testing approach like eggplant offers can come in handy perhaps you have a g5 that's in a different department at your company and you'd like to be able to run your tests against that when you reach this stage you may have hardware at the vendors site even that you'd like to run your tests on or perhaps at the Apple ADC compatibility lab in cupertino and that can be done remotely obviously again the automation provides consistent and repeatable results if you're running your tests automatically you can you can do much more testing than you would if you only had manual testers one of the big one of the big wins obviously was automation always is that you're eliminating a lot of your manual testing time and giving a much more efficient execution one of our customers develops tape backup software that runs on unix systems and works a lot of different tape drives from different manufacturers every time they get a new new new model of tape drive from one of the vendors or want to revise their application they have to validate that everything is still working on all of these different devices and on the different operating systems that they support and this when they had a manual process to do this took them about two weeks to go through all of their validation tests with eggplants they were able to write a script that runs in eight hours and by being able to to run their tests over night like this of course they not only stables and a huge amount of time and effort but we're actually able to modify their their approach to developing their software because they you can you can imagine with a two-week testing cycle at the end of every build you're not going to be able to innovate quite as quickly so Thompson's going to show us a kind of an interesting integration test now on the general machine thanks so for this next script let's say that my company wants to offer a free trial license of its software 12 WWDC attendees we want to allow people to you know come to a website fill out a web form get a trial license key download the software put in that license key run the application and test interactions as if that our software has with the operating system we can validate this entire process using something like eggplant so let's go ahead and just look at this integration script at a real high level here again we're going to reuse our launch application command to go ahead and launch up Safari and then we're just going to go through that entire process we're going to go test the down the free trial download form by going to URL and filling out the form and then grabbing the license key off the site then then we want to go ahead and download the application using Safari make sure that downloads and the disk image unpacks and installed correctly then we want to go ahead and move on and launch our eggplant application and put in the license key that we got from the website then you could imagine we would actually run a whole battery whole series of functional use case tests like we like we did in the in the first demonstration but in particular we want to do things like test its interactions all of eggplants documentation is written as PDF so we want to go ahead and make sure it's if they all open up properly and in in the preview application and it can be read and seen and finally we want to go ahead and clean up and quit out of that and go ahead and even test the uninstallation process so we can go ahead and do all of that with eggplant so I'm going to get started here ok and just run the script and we can just sort of watch it go through its paces again driving driving various components all on that remote system an integration test like this could it also be a multi multi system test we could initiate one process on a machine quick close this connection open a connection to another machine and start processing there dear eggplants to automatically agree so as a licensing agreement panel that comes up here we're launching the application and it says it needs a license so we'll plug in the one that we pulled off the website again we could run a bunch of eggplant tests now or we can just go and check and make sure that the documentation is coming up correctly so there it is the bookmarks and everything so having moved moves through all that we'll go ahead and just clean up clear out the license and quit and close eggplant so there we've just there were seeing an example of an of an intense process test of all the steps necessary for it for a full integration thanks Jonathan that was great I think you can see the power of automation there in about a minute and a quarter we were able to do all of that which is something that you'd like to be able to validate after you're ready to ship your application to be sure that it's all ready to go so some of the information that you might be looking to get as a result of your testing of course the first thing is that you you're looking for bugs you want to find any bugs that you might have in your application and be able to see what those are project managers usually are also looking to gather some statistics about how many of your functional tests are passing or failing at any given point in time and hopefully reducing the failures throughout the development cycle if you're doing some performance testing if that's so important for your application of course you're going to want to gather some some timing metrics there and so clearly that's an important thing another really important piece though is to provide some feedback to developers and I don't know how many of you who are developers may have had situations where testers tell you all this is the thing crashed here or something didn't work and but but they're not very specific always about exactly how that occurred or on the other side of course the testers are sometimes frustrated that they tell the developers that something doesn't work and they're not always believed so being able to provide clear communication of exactly what the steps words that led up to a crash or a bug can be really really valuable and that's one of the key things that automation can provide by having your test automated and logging the information as it goes it can show you exactly what the steps were and of course you can reproduce that for the developers and of course the automation also is really critical for being able to reliably compare one run to another and see the progress that you're making during the development cycle Jonathan's going to give us a quick view now of some results in eggplant thanks eggplant automatically is going to record results any time we run a script so let's go ahead and just look back at the results are recorded for that last integration demo that I ran for you so we just come into eggplant and go over to the results tab here you can see it automatically records the time each time the script is run and then it keeps statistics on any errors or warnings or exceptions that were raised if we actually click on that run a little bigger you can actually see that eggplant is actually logging every interaction that it that it performed to the remote system in terms of clicking what time exempt from the action what action it performed and even we're on the screen that that was found and we can step step through all of this and so you've got a you've got a very clear sense of exactly what was happening during the test even down to the exact second that it happened and we can go ahead and double-click on any of these lines to jump back to the script on and see what what command caused that interaction of course something that's really important to know about testing is how do errors get get reported how do you tell when something went wrong I'm going to go ahead and force an air conditioned here in this integration demo by having us go to a web page that doesn't exist and then just go ahead and run the script with that so it won't take too long before eggplant tries to move through its script but lo and behold is the page isn't there you know the webmaster did something bad so eggplant comes up and says that a script failed and now if we go back and look at our results we see that the results for that execution is in red and again we can follow in follow through here everything that eggplant was doing including what it was trying to do when it eventually failed it was trying to find the name field to fill out the form on the website any time the script fails eggplant automatically grabs a full-size screenshot of exactly what the screen looked like when it failed so you can easily communicate to the developers exactly what went wrong and exactly what it was what it had done beforehand so that's just a simple look at the the kind of logging results that are important for for testing thanks Donovan for money and I might also notice that all of eggplants reporting is done through standards text files that you can easily import into other reporting tools that you generate whatever kind of reports you might need a little look now it says some additional uses of information beyond the just testing for example if you you may have developed a web-based application and you carefully tested it you know that it all worked it's now out in the field it's deployed running 24 by 7 and you might like to know that that it's still continuing to run so one possibility would be to create an eggplant script that goes out so maybe once every hour or every five minutes whatever makes sense for you and to run some of your simple tests or it just looks to see that the web page is still there and benefit if there's a problem that you could you know write the script to actually send you an email message let's say to say oh something's not quite there and didn't didn't work of course if you have a critical application and an email may not be the best answer so you might want to have your script do something a little more interesting there's a there's a number of web services out there where I veena is one that we've worked with that you can actually write a script that will then call you up on the telephone and interact with you know press one for if you want to abort the thing or whatever and gets those results back and you can read that in your script and take action based on that of course you may not really want your computer calling you up at four o'clock in the morning to tell you that the website is down so why not have have a script actually do something about it instead of just telling you you could conceivably write a script that when it detected an error condition would then go and log into the server directly and restart apache or whatever might be needed one of our customers was interested in doing some validation of third-party software there were big organisation with hundreds of Windows machine and whenever they get a new path to windows they're interested in knowing before they deployed that patch you know to all their windows boxes they'd like to know that all of the applications that are critical to their users are still working so they are instant using eggplant to to do this kind of validation of other software running on the new version of windows before they deployed that throughout the organization system administrators of course do a lot of repetitive tasks in the maintaining systems throughout your company if you've typically they write a lot of scripts shell scripts in pearl or some other scripting language which works great for a lot of things but again there are some things that that can't be done that way maybe they are only available through a user interface we've had heard of people who needed to update their virus software on all of their windows machines and that's with something that there was a process which had a user interface it couldn't be scripted directly so might be very nice for a system administrator to have a script that they should run that would connect to each of those machines in turn rather than having to walk around to them one of our customers fujifilm has taken a formerly manual process of validating x-ray film and automated that this is a process that is a subject to audit I the FDA and very critical that this be done on a regular basis there's a variety of different kinds of automation that that are possible hardware and BIOS level testing is not something maybe that would immediately pop to mind were for something that's a typically thought of as driving software but in fact most hardware of course has some sort of an interface and subscribe an interface then eggplant can interact with it and drive it we even have one customer who is doing BIOS level testing there's a hardware kvm switch that supports VNC and they're able to use that to have eggplant drive the machine even at boot time we also have come up with an idea of creating movies so we built in a feature into eggplants that allow you to capture a QuickTime movie of exactly what's going on on the remote screen either when you're interacting with it live or under script control since you are already developed a test now to walk your applications through the various use cases of how users are going to interact with your system might be kind of nice to be able to capture that as a movie and present that as part of your documentation and going to have a quick demo now moving just really quickly for our final demonstration here's a little script that I've written that actually records a QuickTime movie of that integration demo that we've been working with the first thing is it does it just sets up some timing parameters and this just slows down the script to make the movie easier for a user to follow exactly what's going on and then we just say start movie and go ahead and call the script that we've already been working with I'm not going to run it in and have you watch the demo again but i'll go ahead and show you the results of having created this movie just before so again it's it's a process we've already gone through for testing and validating this but but if we want to go ahead and create a simple interactive movie to show users how to do how to do something within our application it's real easy using using eggplant to just to just throw a little wrapper around it like this and and create a little quick 30 seconds to minutes minute loan QuickTime movie that you can put up on your website one of the nice things about creating a movie using a script of course is that if something changes in your application or you decide there's a little different way you'd like to present that process your users you can easily just modify your script and recreate the movie it's a very quick process a lot easier than then editing a movie file so this is some up we've talked today about a number of different types of testing and the kinds of value that automation can bring to those unit testing which is very important to validate your code as the at the low level functional testing to test your use cases how the user is going to be interacting with it stress testing is very important to bring out any bugs that may be hiding in the application and performance testing of course to make sure that if the running as an unacceptable speed integration testing is often overlooked it's a one of those things though that it really impacts the experience that your users have with your application when they when they get it we also talked a little bit about some of the kinds of results and information that you might be looking for from your testing process and have a quick look at some different uses of automation beyond testing so I'll to spend some information that you you can use to go home and improve your own testing processes and turn out some great applications for Mac OS 10 it's our contact information here and we also have a boost downstairs in the exhibit area would be very happy to talk to you later today or tomorrow there's also a free download of eggplants that's available to attendees of WWDC this year there's a the URL is included in the information that's available as a part of this session or you can come come talk to us in the booth and like 20 yes there's a also some more information here about additional information that's available through the apple website