---
title: WWDC2000 Session 414
framework: wwdc
role: article
path: wwdc/wwdc2000-414
---

# WWDC2000 Session 414

## Transcript

Kind: captions Language: en we have with us Eric Bailey to discuss performance metrics trying to squeeze even more information out of your application and we'll try and squeeze some more information into your head as well enjoy the session okay so I can get this to go forward here there we go I'm Eric Bailey and Internet services at Apple where we deploy applications like the Apple Store and AI tools I'll be joined shortly by Stan German who works in the web objects tools group it's going to talk to you about some of the new features in web objects 4 or 5 which are going to help you measure the performance of your well objects application the sessions have probably gone in the wrong order we wanted to talk about performance first how do you measure the performance of your application then when you find the problems go and optimize it which i think was this morning and then deploy it which was yesterday so today we're going to talk about performance metrics I'm going to talk about how to measure the performance of your web objects application and improve it we're going to focus in on your application not all the other parts of your environment and we'll discover what factors will influence the overall performance of that environment and help you pinpoint the code you can go and improve so you've probably seen a picture like this before which talks about the overall architecture of a very typical web objects application where here we see that HTTP request lifecycle comes in gets served by the web server into a web objects application the green area a lot of the times hitting database information flowing back to what objects back out through the web server into your client browser the point of this slide is this look is to show that there are a lot of factors that influence the overall performance of your application it's not just inside the code that you've written in the action Asian business logic area of your well objects application other factors do come into play and it's important to note what all those are and a lot of web objects deployments have a lot of other boxes in them and that you may be accessing LDAP you may be relying on files on the disk statting those reading those in writing those out all those can influence the overall performance of your application so today we're going to try to answer a few questions how do I measure performance what does it mean and am I getting good results we're going to go through whoa snap switch is a very useful tool for getting started and then move on to into event logging which is a new feature in web log fix for five it will help you get into finer grained detail what areas of your application need your attention there we go Thanks so the web server at the top layer of this really acts as a load balancer in the web objects deployment environment it's basic job is to send the forward the HTTP requests onto an instance of your application and in so doing it's basically an i/o intensive task you get down into web objects and then you may be making a request upon the database all the factors on the left are very important to note especially database and other sessions have talked about improving your overall database performance optimizing EOF making your queries more efficient and although a lot of us take it for granted there's a lot of i/o network traffic going on there's latency all these play into the overall performance like I said and you want to not take these for granted once you take some measurements within web objects in your database at HTTP server layer and at the client end you'll get a better feel for whether this is important or not usually it's not a problem in a lot of deployments but then the last Lefter you've measured some traffic and felt the load on your application you'll figure out if you need to talk to your sis ID mins about tuning your routers or configuring something differently so the first tool that's built into web objects we're going to talk about as well stats there's been in web objects for a couple of releases and it's a good starting point to figure out if your application is is on the right foot it's not going to get very fine-grained like event logging will but early on in the development cycle you should take a look at what stats and figure out if your pages are coming back your transaction times are acceptable we'll talk about what's acceptable in a minute and what might not be acceptable well stats like I said it's not very fine-grain it measures transactions time at a high level basically an HTTP request comes into the webserver forwards it on to your instance and as soon as it gets there start the timer and as you can see here the database is downstream of that request so therefore all of the time taken to do a sequel query is going to be recorded in that timer once we actually are finished sending the response out of your instance back to the adapter web objects based adapter embedded in the server that timer will stop so what steps is in the well extensions framework and it's always on as pretty low overhead well it's not always on you can turn it off there's low statistics store class and we'll application which you can use to modify well stats customize it to your needs a little bit and you could turn it off if you wanted to this is a screenshot of the direct action called low stats an example URL above of how you would access it in the well info center application as you'll see in the screenshot we have 17 overall transactions in the top left corner of the table this is not many transactions and I don't I don't expect it you should read anything it about your web objects applications performance with that few transactions you want to get it warmed up touch all the areas of your application the first time you load a page web objects has to read that component off the disk initialize a few classes internally and you really you really need hundreds and even thousands of transactions through your application before you get a sense of the performance but here we we hit well info center a few times and 17 to be exact and we had an average transaction time of 1.0 3 6 seconds now that shouldn't alarm us because like I said you want more transactions but that's not great I know that whoa InfoCenter though does some indexing that startup and we would expect this to a certain degree the next number over this interesting is the average idle time I had this application running for a while before I took the screenshot and here the idle time is hours really but this is the amount of time between requests so how long your application sits idle this is a good number to use to get a sense and deployment it may have been mentioned whether or not you have enough instances or too many I know that on I tools we we look at this number a fair amount to see if we need to add more instances you want to have it be probably I don't know it's it's it's a hard number to figure out exactly what you want it to be but 10 20 seconds is good higher than that is is even better if you want to if you expect bursty traffic you're going to want to want to see the average idle time in kind of low or normal traffic modes be fairly high in the twenty to thirty second range so that you have enough instances that can handle a burst of traffic moving down the page if you scroll down you'll see a couple of tables that detail their component actions and the direct actions so the component action statistics table on the Left we see the name of the component being returned this is important to note that you know in the in the graph and the diagram that I drew before web objects application does take values from request invokes an action and returns a response the component that is being used to return the response is listed in the left column so although we see in the menu component that the average was very fast eighteen one thousandth of a second that was returning the menu page so it also involves whatever was the what was going on in your application when you took values invoked an action and returned that component so for instance the main component is very slow one point three oh six seconds now again we don't have enough data to be sure and the thing that I would point out is noticed that the minimum is 16 1000th which would indicate that it did fine on a few requests and the maximum is seven point one to seven so like I said I know well InfoCenter loads some indexes and figures out some things when it launches to better help you find documentation so it has some warmup time so we'd really want to hit this application a thousand two thousand times and we'll talk about how to do that a little bit later so that was wool stats and now I'm going to hand it over to Stan German who's going to talk about event logging which will help you get into more finer grained detail about where your application needs tuning all right hello I'm going to tell you something about will even blogging which is a new feature it will set it in 4.5 Louis tez has been sitting around for a while but this is a new feature and most people don't even know it's there so I'm going to tell you about it it's way more fine-grained and low stats it doesn't do just components you can use it for logging any kinds of intervals it allows you to identify the bottleneck of your app well that's what performance tools are all about and what's interesting is building to any four point five four point five complication that you are running yeah patience okay that works so what you're going to learn here what is an event what's the definition of an event what are the building events with objects comes with a lot of building events so that you don't need to do anything in your applications how do you enable it how do you read the data that it shows to you and if you're really into it how do you define your own custom events for tracking of your own application specific things okay I'm going to give up on this what are the prerequisites to use a new event logging well on any platform with objects four-point-five will have this enabled you can connect to it with any simple web browser and you don't need to do anything you don't need to link against the special framework or library just when the app looks slow you just connect to it and find out what's going on right now what is an event an event has basically a starting point and an end point so within the framework we do most of those strategic places database fetching component rendering we already have the start point there then we do our own stuff and we have this stop point there and those events can be nested let's say we have a page and the page is being rendered so the page of subcomponents so we have the start of cage and then well start of component one and then the component is rendering itself and maybe needs to do a database access so we have the database access in there boy I went logging is very very efficient there is again no special framework or library that you would need to link against all event logging is programmed in line so we don't do any objective-c posing or anything like that and all the analysis that you will see in the demo is done at analysis time so while we collecting the events it's pretty efficient and we have memory overflow protections so that's when you for instance forget it turned on its religious Ronald's out of memory and kill your server that actually is serving customers he had you might be thinking well performance and really efficient that what what word is that on a 3 and Omega host g3 we can log 300,000 events per second sustained so that's something and again sustain that can go on for hours and we won't die on that there's no disk i/o involved I know that it's pretty these days disk is very slow compared to what it used to be relative terms and also the way it's done it's pure C methods we have some special handling how to do it straight in see instead of Objective C and Static Java methods for maximum performance some more numbers we have a custom memory manager and garbage collector and so let's say what's the worst thing that can happen to you you turn on the event logging and you just forget it turned on and so the worst thing that can happen to you is that you'll waste 4 megabytes of memory per execution thread and our own memory manager will recycle this memory as it as time goes by so the most the oldest events will be trashed and the newest ones will be used and all of those levels can be said so if you don't want to waste 4 megabytes just on 1 or whatever you can set that in per default robustness as we all know nobody's perfect and our applications do tend to do bad things raise exceptions and we handle exceptions gracefully for instance you start an event and it's nested nicely deeply and within the database execution you hit exception well what's going to happen the exception handler is way way below and all those nicely opened events they will not be closed but the exception handling mechanism will still catch it and if the system finds out that for some strange reason something's going wrong the memories going Way or something it would just turn itself off and that's on by default you would really have to tell it not to turn itself off and continue logging the system is very complete we have strategic places where events are built-in there's literally doesn't tell them I didn't really want to list them all but we have 4e adapter regardless which adapter you have editing context different component events again many and for all pages and photo bindings we'll see that in a demo later on the user interface for this tool is accessed through to direct action webpages through any browsers that's the will event setup page that is used to set up the event logging and the even display which is actually used for the data analysis somebody cell phone is working I had no signal here but events are grouped into groups that kind of make sense say the database context event is a group of actually two events objects with a specification and Save Changes it doesn't really make too much sense to log one without the other so they're grouped together and this way we have about a dozen of groups with a total of you will see the setup page again like three to five events per group the event setup page again I'll go more in detail into that in the demo is used to turn on or off blogging you can also use it to reset logging to clean the cache and it will show a dynamic page of all events that are relevant for application for instance when you are using it Oracle adapter it will not show the Sybase event it will show only the Oracle event and so on so if you have custom classes and you have you even maybe your custom events those will be listed and if you don't they won't show up so it's all dynamic the event display page is again accessed writable even displayed direct action and it allows you to view the data to collect the data in five different ways depending on what particular thing you're looking for in the demo I will have talked about three of those most useful ones so this is a more detailed example of what we might encounter in a real application execution this is the same slide as before and say you have a main page and it does something and then within itself it builds a dynamic nested component and that component conditionally executes the database fetch maybe it might not need it but maybe it might compared to that we have another page it's the same page but at that point the conditional did not fire and so there is no database access needed so again for comparison we have those two and all of the events logging so too as to make it easier for you to read the data on-screen is aggregated when you have here this is the way it's going to be displayed on the browser we see that the main page could execute a total of two times the method component itself got also executed two times but the database fetch was executed only once and so we see that for instance that total time spent in the main page was 70 milliseconds over two instances and the nested component it was 60 milliseconds in those two instances and the database fetch lasted for 50 milliseconds at one time so what can we read out of this we see that the main page that for 10 milliseconds something different than dealing with it's nested component and that divided into event so 5 milliseconds each time and of course the lion share of for whole page goes to the database access which in one same single axis we used up pretty much all of the time so this is one way to interpret the data also you need to understand it the times you're looking for what's not there so like again between the main page and the mystical then there's ten milliseconds somewhere and since you have the source code for you can exactly see which part it is there will be there will discuss three different ways to display the data the most the simplest one for when you're just starting to performance enhance your application will be view by page and then by component you have a page that has multiple embedded components and so you'll see the most expensive page on top then you will see that you open up the link and there will be more events in there sorted by component and you can drill down into those components into database access and whatnot there's another view view by a it's very similar to the one view by page and then component except it will show all the components flattened so in a really sophisticated page you might find yourself looking at a really long list and it might be a little bit harder to find out which component what database access happened but it's a more complete information and you can still drill down into those supplements and finally the unsorted view is also aggregated so the same aggregation that I outlined initially would take place but it shows you everything flattened out throughout your application so you see really the worst thing right on top and the second worst thing and I think this would be the demo so yeah excellent what we have here is a the think movie is very popular with us on demos I'm sure you see it before that's what we ship its in as an example purely written in Java and I wish if the source for that so you know we have the home page with the search page master index and what I'm doing now I just you know click around let's search for s so made clear a stand by me and we have the simple I'm not really going to talk about yet what they want to see it's like let's say it's slow and so we want to turn on event logging so we go to the whoa you and so age I hope I'm not the only one who thinks that the keyboard is not the best on the planet so here we have the list of relevant events for this application we have the database context event editing context event application event Association event that's the same as like a wool binding it's probably better name for that component and component reference event so I'm just going to say select all and well let's go back to the application and I'm going to play around I just boys like to play and Star Wars okay master index let's just click around make the application do something and let's go here scroll around a little bit this will not be the most amazing event lock again because we're not showing we're not really exercising the app I'll show you a page star Wahb with lots of really cute geeky pictures this is the web objects builder team as it used to be back then one and here we also have the EU F team look that's a really special M on both teams here and so let's not leave the walls came out all right look at this this is a very beautiful picture okay and now let's see how the application did well Eve and this play I proudly type with two fingers well this one is a little bit too big does anybody know how to make the phone smaller okay well I have a button on to love it okay it looks awkward but we will go first to the show by page then component view mode and here we see that all the pages that our application has touched during this brief exercise we log a total of thirteen hundred seventy four events 91 events of that was an EF and 183 in web objects so we see again I said we can log 3 in 2000 events per second and we see that there's not that many events here so you really don't get any performance hit and when you leave it turned off you don't get any performance hit whatsoever so let's see what what's happening in the display video page we see that it has received 643 calls and it took a turn 33 milliseconds in total while the display set page got about as many calls but to go only 42 milliseconds so we know already worse the problem is in this page alright and now it's sorted by component this is a pretty lame example because this page is only one component itself so let's assume there would be no more component and here we see oh pretty much all of the time 814 milliseconds out of the 833 were used in pulling the data for the actor image so well it was kind of to be expected because these are data blocks but it shows you here and I can again drill down and I see that actually of all that time most of it was in your binding most of that was actually in the objects which write specification call within EOF so you might want to look maybe your page specification is not the most optimal one etc but if the binding still took some other time almost 100 milliseconds and you might want to look into that the other display mode that I mentioned is the one group by page only and this will be only very minor change you see there is no component in their display video page but if it were a real application with more components it would actually be nice to have them sorted by component and otherwise it can still drill down and you will see the axis which objective sight spec and the last one that I mentioned in the slides before would be the one aggregated hierarchically events which is just without sorting without any sort of hierarchy and we see immediately that the data pool from the database is listed topmost didn't need to even drill down anywhere however as you see the list is getting longer and longer and if we had more events it would be a little bit harder to read and as I mentioned before with the aggregation of events when we have a similar call graph that we actually coalesce that all that you see there's also the raw mode that I usually don't want to click on because it would list this one five and twenty four times on the webpage so this tool actually does a lot of work for you just by looking for the identical call graph and putting it together so so much for the demo can we switch back to slides now what can you do what if you need to customize it what we've shown here this is the stock your event will event logging and it does already quite some stuff for you but if you need to customize it the events are very very general they're based to basic super classes the will event and the EEO even depending what you want to measure it just came sub class that our events are pretty much everywhere where it makes sense as you can see and now let's say that you want to measure something very very specific let's say that it was no database search that took so long but for some really strange reason you're in it code is taking so much time and in your init code you're doing only three four things and you want to find out what's really so bad so what you can do is this is a generic piece of code or okay awake not in it you want to log your own stuff so what you do there's a global variable that's being set because we set again very efficient so do just a boolean task if your event logging is enabled then only in that case will incur that little bit of overhead we'll create a new event which is a static method new event of class just a normal event class and you can pass any string as a comment to it as you saw in the table before in the demo there's always like comments that you can display and then you mark the start of the event we also have atomic events that just say like hello I was here and they don't have any duration but we will not really go into those then you have your own code there that you want to profile and then finally when you're done if event is not equal null or nail in my favorite language then you mark the end of event and the system will automatically logic it's possible that within your profile code here you would have some other start and end event in there and again if you raise an exception our own exception garbage collector in a sense will take care of that an alternative that's a this was the cheap solution now let's go a little bit into the more sophisticated one you can provide your own event classes say you want to log some really specific things about in which context or you call you want not just like I was cold and I took fifteen milliseconds but more I was called from this component or I was called from the other component and you need to store some data in your instance variables so you just create an empty class and it's good enough and it will be automatically added to the runtime and the system now knows that it exists and then you just call it oh there's a description file that's a file my component event dot description and there's only one mandatory field that's the e event group name which is my whatever event and then do this do that I will go back to this screen we see here your event center mark start of event event awake now that's the key that you would be mad that will be the do this or do that and that's just for you as a programmer a really short description and what's going to be shown to the user in the whole table view of Steve and display will be the long or both description like operation number one or another operation to relock so this is the basic key value mapping only the group name is mandatory when you don't supply the other fields it will just take whatever you pass in as a string and you place it in the project resource suitcase and it will automatically be taken care of I think that's about it for event logging so I'll give back to Eric probably he knows how to operate this the button that you showed me did not work - right so vent logging is really neat and I'm looking forward to upgrading some of our applications from 402 four or five just get that feature and help us get into tuning our applications at that high level of detail and one of the things that you that Stan and I both commented on is that you need to do this under stress sitting there and having one person hit the application and logging these events isn't enough it doesn't give you enough detail doesn't tell you what's happening when your application is running so the next topic we're going to talk about is a tool that's built into web objects refer to it as a playback and recording recording and playback it allows you to generate load on your web objects applications and then the numbers that you've seen in well stats and the event logging are going to turn into something that you can interpret and it'll be useful in helping you forecast things like how many instances you're going to need how much hardware you're going to need and where to go from there on your deployment a couple of things I wanted to go back and touch upon when you're doing this kind of testing and you saw that this measures using the real time you want to make sure that what the system that you're testing on doesn't have a lot of other things running on it if you have other servers running you have other processes running they're going to take time on the CPU and take up resources that are going to skew these results okay so playback on recording as I mentioned is a load testing feature built into web objects I'm going to talk in a minute about what it's what it's as I mentioned it's load testing they'll talk in a minute about what not to use it for because it's intended for some very specific things and I've heard and seen people misuse it basically what it does is record sessions you use a browser you click through your application get the coverage of the pages that you want and it records that session saves out the raw HTTP requests and response to disk and then you can use those as a later time to play them back and simulate traffic so I emphasized load right that playback is intended for prototyping load and it's once you do this you get a good feel for how your application is performing when it's in a deployment environment one client in one server doesn't really cut it so once you start using the playback tool or some other tool then you can access your application with the browser and see how it reacts and with one instance on your development box then you can take this on to you to your QA or staging environment and see how it behaves there on the class of hardware that you're going to deploy on and get a feel for how it's going to work there so I'm saying it's free well yeah it's free if you have the product there are other tools I'll mention in a few minutes that you could use that aren't free or that aren't bundled with little objects that do the same kind of thing and do other things as well but playback manager is a great start like whoa stats is for getting the feel for our applications performing the last point I want to make is that we've used it very successfully in Internet services and the Apple Store is an example of a high volume internet site that we use the playback tool over and over fine tuner application continue to reuse it and ended up with a very stable application once we deployed it so it's not a coverage tool it's not for functional testing it's not scriptable once you record that session that session is going to get played back and you can't change that you'd have to record many sessions to get coverage in this depending upon your application can be very complicated so that's why I emphasize load before because if you're going to use it to try to get to coverage you're going to want to use some other tool or create a custom script another limitation is that it can't do very good validation on the results of your pages so if you're sending back dynamic data coming from your database you're going to have some challenges making it validate that that's okay all it really does is compare page links and says you have this page is the same as this one and actually continue on to the next HTTP requests in the session and it it can get tricky to manage a high volume of virtual clients with the tool if you want to simulate tens of thousands of users it's not going to scale up to your needs very easily the playback manager can do this it takes it'll take some of your time and investment to get into that and set that up it also take some hardware to do this you're going to need more machines to simulate that kind of traffic so the first thing you have to do is make a recording so here's an example you just start think movies the application of stand use earlier with this argument and give it a path to tell it where to record those sessions where it's going to save those requests and responses start your application launch your browser click your application and when you're done stop your application that's all you need to do in the session will be several files will be stored in a directory they'll be a directory created inside slash temp like for example sync movies dot RTC recording and there will be all the HTTP requests and responses in there as I mentioned there's a couple of options for playback there's a command-line interface which I'll show you a screenshot of and there's playback manager which allows you to launch some java remote demons that can it can fan out the load across to all those and simulate more traffic in a lot of cases the command line tool is going to do just fine to do some of the things to use event logging and well stats to generate hundreds or thousands of transactions just with one or two clients just open a couple of terminal windows get those going and you'll be able to discover a lot of the things that slow your application down so let's take a look at the screen shot of that command-line interface so here I took a screen shot of running this recording I did on hello world hello world's a very simple application and puts up a form and it you type in a text and it sends the next request and displays a page that says hello whatever name you typed in now another important thing about playback to mention when you record this script it's recording the rise to DP so if you entered in in my case Eric for the hello world example every time it did that it would send back the same text so another limitation there is it's not good for dynamic data there's some other tools that can take data and insert those into the forms in your web objects application and simulate more variety in your in coverage in your form posting I wanted to mention another thing on here you see a number on the far left which is the total number of requests that the tool has sent there's a the time it took to receive that response back and that's the total amount of time measured from the clients perspective now earlier we saw low stats and we looked at event logging and we got numbers that were measured lower down in the stack right so in this case we're looking at seven one thousandth of a second for request number one if we look at we did the same measurement on well stats we should see a smaller number because there is IO going on the rise to DPS any being sent back and forth from the browser that's going to take at least some time so you can use these numbers to make sure that the numbers you're seeing on the server side makes sense and then that overall picture that I showed at the beginning you're taking timings at the very top inside web objects as soon as the requests get in gets in and is being sent back and then you use event logging to go deeper in and analyze the smaller code fragments that you've written in the middle you'll see average which is a moving average and this is a good number to match up with one of those numbers have begun top of low stats which tells you what your overall transaction time is and then it shows you bytes and I set the different I set a parameter you'll see on the top line to be very large basically like I said it does matching based on the page lengths and if your page lengths that it gets back don't match then it considers that they failed transaction and you'll see that all these passed and you can have it ignore failures and keep going or you can have it stopped when it when it gets to a page that doesn't match if your recording goes through your application doesn't get into a lot of dynamic data you don't have pages that are grow based on how much data is in your database you can set that percentage down a lot lower and get more meaningful results and actually use it for some validation but that leads us into what other options are there there are some third-party tools we've used a few that will do more for you than playback can I know of a couple and another some others out there there's mercury has one called lode runner runs on NT RSW has one called the test suite which runs on NT as well and used a little bit of each of those to do some testing and they're a lot better for coverage and that you can script so you can ask your you can write a script with one of these tools let's say for example an eye cards application on our website you can have it go in and try to send every one of those cards and we can get better coverage to make sure all the images load it's not doing what playback and recording is doing where it's just sending the HTML it actually fetches all the images as well so we can make sure that all of our images appear we don't have broken links etc with a tool like this and then you can write a script to do the coverage so go touch every category and I cards go try every card make sure all the images load etc etc this obviously will help you with regression testing this your application is not changing too much or you've made a change that should not affect the user interface you could use a tool like this to make sure that you have no new bugs and they're more flexible in the way that they do validation so if you have a string that appears on a page that's an error server error error page not found something like that you can ask these tools hey if you ever see this string report an error I want to know about it and you can have it if you have dynamic data and your page gets much longer you may have some text at the bottom of that page that signifies that this was this was a valid retrieval of data this page is fine you can say I don't care whatever else is on the page if you see this string it's a it's a valid response and we should continue on with the test the other thing that you can do with these once you've set them up is put a box just outside your deployment environment outside your firewall and have it just sit there and execute a script every hour - or maybe every couple minutes make sure that your application is responding and doing the things that you expect it to do which you know turns it from a testing and development tool into a deployment tool where you can check for response times make sure your application isn't down and it's just you know unlimited with the scripting capabilities you could add to to do some checking of your site so I want to wrap up with other factors of influence performance and talk about these a little bit we're talking back them again I showed the architecture diagram we were focused in on the green box what you can do in web objects and as I said before when you do a test like this make sure that you're testing something similar to what you're going to be deploying with ultimately if you can test in your deployment environment that's the best thing you can do if you have the same class of hardware that you're going to use in your deployment environment use that if you're going to be if your database is going to have a load on it from other sources not just web objects seeming like that if you can the more realistic you can make this environment for testing the better results you're going to get from event logging and low statistics and the big picture you know hardware the CPU the CPU that you use the amount of RAM makes a big difference and there's tools on all the platforms we deploy on you can use to monitor how busy is the CPU am i utilizing this system very efficiently is it maxed out as a CPU bound or is the is a memory swapping out to disk check the you know in Solaris or something as simple as vmstat you can use to do that and there's been some other sessions that talked about database optimizations advanced DOF you can do a lot of things there to improve your overall database performance obvious things turning indexing on and if you can influence the schema you know you could you can make some adjustments there that are going to affect the overall performance I'm not going to even go down too far into the single threaded versus multi-threaded topic I'll just tell you a little bit of what we found is that single threaded apps do just find and the the maintenance cost is very low you don't you don't get into more complexity and a lot of the apps in fact all the apps we deploy our single threaded and they perform just fine letting the the operating system do the switching raus multiple instances is sometimes just as good is not worrying about all the issues that go along with multi-threaded and having said that you can obviously run into situations where you have long-running requests and multi-threaded apps may be your answer and as a stand pointed out with the event logging features you can write custom events my picture only showed a database you could have other things LDAP you may be relying on reading and writing files a lot other factors write custom events wrap those bits of code with with custom events and figure out you know how those impact your overall system the more information you have you'll be able to influence the other parts of that picture that I drew earlier maybe your problem is not in web objects but it's in your database or or somewhere else and the more data you have you can go and optimize your code and once that's done it's somebody else's job so hopefully we've given you a better idea for how to measure the performance of your WebEx application how to dig deeper get some numbers I'll invite Stan back up here and we can take your questions [Applause] you