WWDC2004 Session 218

Transcript

Kind: captions Language: en good afternoon is everybody ready to go home okay well and make yourselves comfortable so hello yet again in case you don't know I'm Matt your aunt's i work in DTS and I can't stay off the stage I can't help it I've been on stage every day this week I think that might be a record actually so um I kind of adopted this talk at the last minute some of you may have been expecting to see George Hemsky who's our resident graphics guru but he had a small incident come up tuesday night so so Gerard won't be joining us I'm going to do the best I can to sound as authoritative as he would on this topic so what are we to be talking about we're going to talk about the changes that we've made in Java 14 to update 1 you've got Developer Preview three available to you as of this week I'm talking about the architectural changes we made specifically to on-screen Java 2d drawing and we're going to talk about improvements not only in correctness and hopefully eliminating a lot of the bugs that some of you have reported but as well as performance improvements that we've seen as a result of these changes we talk very briefly about courts 2d and it's and its relationship of Java and how that story has changed with the announcements that have come in with Tiger this week and we're going to talk a little bit about the third party aap is that are new to mac OS 10 as of the last time we talked here last year a specifically Java advanced imaging Java 3d and jo dl which we which we affectionately know it's joggle so Java 2d on Mac OS 10 like I said earlier we have a bunch of improvements and some new features that we've made available to you with gp3 among them is a graphics context management and I'm going to be saying context a lot in this talk and and within the context of this presentation by context I mean that just a standard Java graphics object java.awt graphics object and talk about state management the state of a Java graphics object and how we've worked around a lot of problems that we have with that in the past and talk about how these changes affect and improve multi-threaded drawing and update one we're going to talk about the intelligent screen update mechanism that we've introduced with BP three I believe that's new with TP three okay and we're going to talk about some improvements to full screen mode drawing and XOR drawing as well so let's go right into it and talk about context management like I said it's a new mechanism and update one and it's completely independent of the cocoa drawing context that we use beneath as you know the AWT in 142 in Java 14 is built on top of the cocoa app kit now in the past we used to share our graphics context with cocoa and created a number of problems or issues that we needed to be delicate with and the most problematic one was that sometimes cocoa would prevent Java from drawing all together so if there was some painting things that needed to be sent through the contest Coca would som sometimes deny us the shared context and it would cause us to either skip frames or paint methods all together or at least delay the visual appearance of painting on the screen the new system improves performance in addition to correctness and the perceived the perceived quickness by the user before not only could the context have been denied by Coco but it could have been modified by Coco as well and we didn't necessarily know what we were getting back and we needed time to reconcile those states and make sure that the state that you expected your Java graphics2d object to be in was the same as the states that we were using behind the scenes so now because the contexts are independent we always know the state of your Java context and we can start drawing that much sooner and when I say state changes I mean things like color clip anti-aliasing fonts all the set methods that you would call on a graphics 2d object all of those things we know at all times now what those states are from what you're setting in your Java code before we had to make sure whatever we got back from Coco was synced with what your Java code was expecting and now that we have an independent contest we no longer have to worry about that and that obviously is less execution time and more time devoted to painting operations here's a visualization of what I'm talking about it might sound a little confusing what you see here is that we're having a little fun with you see Java and cocoa basically fighting over the same context and that checkered blue and orange there is basically kind of illustrating the unknown states at a cot that a given shared context would be in in the original 14 to release and whenever Java got it we needed to make sure it wasn't in cocoa state we need to make sure it was in our state so now in 14 to update 1 because we have our own context exclusively available the job available to you we no longer have to worry about any of that reconciliation cocoa can do what it wants and Java can do what it wants and this particularly is good for multi-threaded drawing the improved continence management that I was just talking about basically allows us to draw correctly from multiple threads we don't have to worry about locking or contention or anything like that it allows us to cash graphics2d objects now that we don't have to worry about state validity or anything like that you can safely cash a graphics 2d objects where in the past it was really really not a good idea and not only just makes things more correct but it makes it very very much faster and this is a huge benefit particularly to old applets from the 10 and 11 days that maybe had been doing a lot of multi-threaded work from inside the browser and here's another illustration of that you can see this huge heavyweight master lock that's basically blocking or barring the entrance into all these context from all the Express and in the past you had cocoa the main cocoa drawing thread and all of your Java threads if you add multi-threaded drawing trying to go through the same locks through the same area now it's what update one it's just the Java threats because we have our own context and because we have our management down much better it's a lot less work to go through from multiple threads and draw to the screen and I'd like to invite Caroline up to show us just what we're talking about alright thanks Matt so this demo is going to show you the huge improvements we've made in multi-threaded drawing support in 142 update 1 first I'll show you a demo of 142 doing some what they try to drawing or trying to the first square is actually the normal retaining mechanism not multi-threaded where you draw you make your paint method and the events get posted to the AWT thread and the second square let me open this up again so you can see it again the second square is using a separate thread and calling get graphics on each repaint the third and fourth squares are each using to drawing threads but they're not working very well so if we go to 142 update one on demo to on demo to please you can see that it looks a lot better it's considerably faster in all cases so these bottom two squares the third one is using two threads once drawing in black and once drawing in yellow and it's using one cache graphics object to draw into and the fourth square the green and black one there's a green thread and a black thread basically each drawing up green or black lines and are drawn into suit two separate cash graphics objects so if you need to use multi-threaded drawing it's now a real option for you in using 142 update one all right I'll hand it back to Matt thank you Karen you sure those were both running the same hardware okay you're sure okay so we just showed you that multi-threaded drawing works a lot better an update one but we want to mention a few things just for general advice even though multi-threaded wrong should work correctly and and perform well as you saw up on the screen we still recommend that you draw on the AWT thread using the standard paint mechanism and repaint mechanisms whenever you can whenever it's reasonable cash graphics objects are also much safer but we really prefer you still use the graphics request method whenever you can and make sure you have a valid graphics object the real time to consider multi-threaded drawing is if you have an event heavy application if you have a lot of user interaction and you find these user events starting to basically clog up the event threat and really start to affect your painting performance and you know it's kind of the old aided you know never pre optimized try to use the normal pain mechanism first and if you have a really heavy interactive application and you notice that your painting is suffering based on user interaction then then would be the time to start thinking about using multiple threads cash in context things like that and all the things that we've been talking about up until now the multi-threaded drawing the context management is this is all referring to on-screen drawing doing off-screen drawing into things like a buffered image from a separate thread is absolutely fine same as it always was and so if you have kind of a scene graphing situation where you're painting into an image off screen and then throwing that onto the screen at the last minute you can do that as you always have before it should be just as correct and as fast as it always was so the next thing we want to talk about is intelligent screen updates and this is kind of a marketing term you may know it more commonly as flushing so the thing that we were talking about when we were coming up with this mechanism is that hardware is getting faster software optimizations are always happening but displays are still fix that a risk as are a relatively pretty slow rate your application and usually draw it you know 100 100 100 frames per second or more and you know the display is chugging along at you know six 70 hurt and so it seems like a waste of time trying to push all this stuff up to this up to the screen when it's not going to refresh any way physically so what update one does now is it only updates the screen when it's necessary when we know it's possible you know you can see the thing on the screen here says 100 Java updates is more than 60 screen updates there's no point naturally performing that flush operation if you're not going to see it now this might sound a little scary but we've done a lot of work to make sure it's correct and we're not skipping any paints or any dropping any frames or anything like that there shouldn't be any visible difference and well there should be something of a visible difference and that's that drawing should actually be faster as a result much much to the tune of the the state management in the state reconciliation I was talking about earlier we're spending less time waiting for screen update so we're wasting less time doing that when it's redundant anyway and that gives us more time to process your next paint event also full screen mode this is a place where you know we've gotten a lot of i'll call it feedback on we've reimplemented it to you know primarily for correctness and in doing so we also did a lot in the speed department and actually it wasn't that complicated a riorca texture we basically changed from a blip offer strategy which is basically copying the bits directly from the back buffer into the front buffer which which is a particularly expensive operation we change that into a flip up for strategy where you have a front and back buffer and then just flip those from one of the other back on the screen the former front buffer is now your back buffer where you can do your next frame for withdrawing and I think Caroline is going to show us what that looks like alright so I've got one more another before-and-after demo here we have a full screen application running in 142 and what this is trying to do is it's trying to maintain 30 frames per second but since we're only getting ten frames per second there's only going to be one ball the more it continues to add balls and tell until it reaches 30 frames per second so one ball but if we move to 14 to update one on demo two please you can see it keeps on putting involved until it reaches 30 frames per second how high are we going to get wow this is a pretty fabulous improvement full screen is now a really good option for you if you if you want to use it on Mac os10 all right back to men thank you very much Caroline I don't know if you had seen had any of you use puzzle pirates before when you had seen Scots deploying demo earlier puzzle pirates did not used to do too well in full screen and obviously Scott was demoing it in gp3 the other day but yeah one ball at ten frames per second 600 balls at 30 that's 1800 x I think and that's all software changes so drug did a good job there so we also want to talk about X or drawing last but not least well maybe it is least we did a lot of work to improve the correctness of X are drawing and this has been a perpetual thorn on our side and the main reason being is that quartz which is what our java 2d drawing mechanisms built on top of doesn't support x or natively and if you've been to any of the court sessions you may have heard this from the courts engineers themselves so we've had to implement it for you on our own and the correctness should be there now but obviously there's going to be a cost because we've implemented it all on our own so the performance may not be what you expect and it probably never will be because courts does not simply doesn't support it natively and so what that means is that you should probably if you feel like you need to use X or drawings for some reason you should probably talk to us to see what you really need to do and see if we can offer some kind of alternatives for you things like alpha composites if you need to do blending operations and the other thing that bearing and Tom mentioned already on Wednesday is that if you're doing X or to erase things you know you're doing a double draw to erase some text for example anti-alias text is not going to be erased by X or drawing because of the anti-aliasing operations those are not taken into account when you do an XOR operation so even if that if that was your real reason for using X or it's not going to work anyway and if you need to do an erase operation the best thing to do is to keep your static scenes that be before the before modification seen in something like an off-screen bufferedimage and then push that to the screen when you need to do your race or your undo so what are all the performance gains as a result of all these things we're talking about well I mentioned the context management changes we're no longer waiting for Coco's permission to draw we don't have to do any guesswork on the state from when Coca may or may not have modified it and we've got safe context cashing for multithreaded wrong if you want to do that the intelligent update mechanisms it means we spend less time waiting for the screen and more time actually executing your pink coat and the full screen drawing change from a flip them from a lit buffer to a flip fluffer tongue twister that changes really as you saw quite an improvement in performance and here's just a quick graph of some of our internal benchmarks as well as the standard ones like the Java to deed MO and the swing mark benchmark in aqua you know we've seen anywhere between twenty and fifty percent improvement just with these software changes in DP three and I think Caroline is going to give us one more demo of the overall changes all right this is the last before and after demo this is really similar to what you may have seen in the Java State of the Union except in Java State of the Union we compared our selves to windows and here since we've been talking about the graphics changes we're comparing ourselves to ourselves so this is a java 2d demo running in 142 and the important point to notice is that we're getting around 115 frames per second alright so in 142 update one on demo to getting more than twice back around 200-250 sometimes so these are significant improvements I think you're really going to like them all right back over to miss Thank You Caroline so I think you can see doing a lot of work to speed things up for you and also at you know it's hard to demonstrate correctness the multi-threaded drawing demo did a good job at that but hopefully you guys will see this in your applications and we'd like to hear your feedback if it is or if it is not either way you want to make sure we're doing the right thing before we release this update so let's talk a little bit about tiger I don't have a demo for you but as you probably heard courts 2d is going to be open to yell so is it hardware accelerated using OpenGL in 10.4 and because we're built on top of course to do our 2d drawing this is just going to be faster java graphics for free without any work that you need to do or even months of the java team needs to do and i have again on there because some of you from the 13 days may remember that we had an OpenGL pipeline of our own in Java and you might wonder why that went away well that's because the gentleman who were who was working on that for Java is now working on court and if you're wondering why hardware acceleration went away from Java it's because it went to the whole system instead and thank you for waiting and you should have it very soon now and the difference between the tiger hardware acceleration and what you may have been used to in 13 is that it really does look like quartz in 13 the hardware acceleration looked like OpenGL and then Tiger it's going to look like quartz it's going to have the anti-aliasing and all the things that you expect from ports drawing so we've been talking about what we've done i wanted to talk a little bit about what you guys can do this is kind of a repeat of things that we said last year but they're still very important tips so i wanted to put them on here the big thing with images in particular is that you need to make sure that you are using an image format that's compatible to the system and when i say when you're using images i'm talking about creating new images that you're going to draw into the paint onto the screen later using methods like toolkit create image graphics configuration create compatible image that lines actually a lot longer you need to do about three or four other static getters before you get to a graphics configuration but static images meaning things that you've loaded from a JPEG file using image IO dot read new imageicon so on and so forth those things will be fine we will optimize though for you once they're decoded as far as new images that you're going to do drawing operations in to make sure you use these compatible in JP is things like particularly the index color formats that are very popular on windows because they're low memory those that image format is not compatible on Mac OS 10 and we're going to need to do pixel conversion for every operation that you do and that brings me into that third bullet there that says if you must use a non-native image please avoid using doing direct pixel manipulation because every time you have to go get the data buffer and get the raster will need to convert those pixels from whatever the store from whatever the image format is to something that courts can actually read we'll need to do that every single time if you use general Java drawing methods like you know drawstring draw line we can fake that and not worry about doing the conversion every single time and obviously that on the fly conversion is going to be expensive so if you can please it will please use the compatible image formats and try to avoid using something non-native so let's talk about some of the new things it's not exactly new anymore we announced that I believe in December or so but we haven't talked about on stage yet so I wanted to mention the Java AI is now available on Mac OS 10 you saw a demo of this Monday night with the Mars rover the gentleman from the the JPL labs in in Pasadena and you know basically bringing high-performance image processing a number of new formats that are not available on the standard java.awt you can you can process him you can do image processing we're a network we did implement the native media lib code from Sun and brought it over to Mac OS 10 so you can get the performance that you would expect from j'ai and it's going to be included with tire it's currently a download for you and more importantly for your users if you're writing in j'ai for Panther it's available as a download the entire group will be pre-installed and you won't have to worry about it and to the same the same tune Java 3d is also was made available at the same time it's a higher level 3d model it's not really a direct 3d programming API it's intended for higher level seen graphing and things like that it is platform independent so you can write applets and Apple occasions in 3d we implemented the we implemented Java 3d with opengl and core audio for the audio and like to have AI it'll be available as in Tiger and currently available for panther as a separate download and there's one other thing to talk about which is Jo GL I'd like to bring Ken Russell up to talk to you about that good afternoon so my name is Ken Russell I'm at Sun Microsystems and one of the authors of the Geo GL package that you may or may not have used an Gerard and I worked a bit on the jo GL ports OS 10 so what is this package its binding a Java binding for the OpenGL 3d API that works on as many platforms as we could make it work on in particular OS 10 works on linux that works on solaris works on yeah windows and the intent here is to let you write portable fast 3d applications in java the the binding is open source it's part of sun's java gaming initiative you can get the source code in java net it's one of the core gaming api projects he's got bindings to jo al the open audio library we've got an input device a binding called j input which lets you get inputs from you know joysticks and keypads and stuff like that some of the features of this opengl binding versus some of the others that are out there are first of all that it supports all the stuff that's in opengl one dot five so you use this you get all the latest stuff all the way to sender extensions you can do vertex programs fragment programs or shader objects all this stuff it's got portable support for p buffers so you can do hardware so accelerated off-screen rendering in a portable fashion get the results on screen very quickly and where necessary you can actually dive down to the vendor specific extensions so it's like if you actually had to use the wiggle api's for something on Windows you can sort of test to see what platform you're on and then dive down into those AP is if necessary we are working on standardizing this binding and the real reason for this is so that we can get some amount of sort of compatibility guarantee for you the developer on all the platforms that you're going to be deploying on and the another intent here is to try to get the mobile device space and the desktop device space where j2me and se have a bit of a rift we're trying to make the OpenGL bindings for these two platforms as similar as possible so that you'll be able to effectively write a desktop app and maybe when the cell phone market and hardware acceleration down there catches up you'll actually be able to deploy basically the same app on a mobile device which would be really really cool so as I mentioned before Jo GL supports multiple platforms and I'd like to point out here are two things first of all even if you're not using the awt and that sort of framework and you're using the cocoa Java bindings for Mac OS 10 jo GL actually supports that configuration explicitly so you can instantiate a little mac OS 10 GL impul object and then treat it in exactly the same way as you would in a and a WT based joga lap and it works and we've got unfortunately we don't have a demo today showing this off but there has been demos developed in the past which which have have stressed this out another thing to mention is again the full screen support that was just shown that's working great with Jo GL & OS 10 so you can write full screen fully hardware accelerated games and apps and we're going to actually show one today that's pretty cool and I see a point here on the slide that I did not know which is that this works well with the new OpenGL profiler on oh wait is this shark vs ok so well it's working with the OpenGL profiler I was 10 so it looks like you can get fairly detailed information about what's going on in your apps they're all are also a couple of pipelines available for the Jo GL binding and these let you get debugging and tracing information and this is sort of a Java specific feature but it's it's extremely useful when you're trying to debug something that's going wrong in your app it'll basically stopped at the point of failure at the actual OpenGL call that made the error state in the state machine and give you a back trace which is pretty helpful so I'd like to show you is a really cool demo done by a couple of crazy guys over in Germany call by tonic software I assume that this is a play on words of Bionic vs Bionic but so if we could go to the the demo2 machine oh wait aha rongsheng okay but we're going to see here is a full screen app full screen little game that you guys may have seen before this is place to do okay now what are you looking at here this is not a new tension written in Java hello Susan devil said looking at here or not we knew and you let me kill some guys here maybe they'll they'll shut up mute on the keyboard huh ah misty okay thank you alright so what we're looking at here is it's not a pure Java new sort of engine that is written to use the quake to file format this is a port to the java programming language of the original see source for quake to woo a lot of that can we kill the bad guys yeah I got to take care of some business first okay all right yeah okay so this is a port a straight transliteration of the sea sources for quake two that were available by the GPL on its website to the java programming language all right they use jo GL at the bottom layer to get to the graphics subsystem and they did whatever data structure conversion was necessary from c to java to be able to use the pack file format the original data files from the game in this port this is running at over eighty five percent of the speed of the original see sources all right at this point there is no reason to not write your next game in the Java language this binary runs on multiple platforms it runs on OS 10 it runs on linux it runs on Windows it runs on solaris okay no porting necessary and now you can do even more cool stuff because you're in Java you can ship bytecodes around the net and it's all safe you can load new stuff in new functionality download it to the client you don't have to worry about you know oh gee did I just open up the door for the next after virus for example because you got a security manager in place and the axis is all safe to the underlying hardware features so this is actually an open source project you can go to the by tonic software website and the demo is called Jake 24 you know quake 2 to Java so what you can do is just google for Jake to and say I think the first hit their their websites out there you can get the binaries you can get the source code and I would encourage you to send them feedback because I think that this is just awesome stuff they did what a lot of us have been talking about doing for a long time which is G what would happen if we had the huge commercial apps side by side C and Java how fast would it be this is showing that it's pretty darn fast I think that's pretty much it ok well thanks again I kind of hijacked Ken with that opengl profiler bullet at the last minute we decided he was going to say this slide so yeah he didn't know about that i was looking at talking to some of the OpenGL guys as well as Gerard about the fact that you can use the OpenGL profiler to profile a joggle application now it's not going to give you what I mean it it is a sea-based profiler but because joggle or Jo GL is a rapper it's pretty straightforward as to what it's talking about and you get a list of functions named ABC functions but you know the names are very similar and it's usually pretty easy to tell what's going on in your joggle app versus what the OpenGL profiler is reporting and again we demo dick too that's the by tonic de as in Germany link if you want to check it out and yeah it's pretty cool we then we've been playing it quite a bit so in conclusion we're going to let you guys out early today because it's friday but we're working on it we've heard a lot of your feedback and we know that we've we've had a lot of work to do and we've done a lot of it we've done a lot to improve correctness and drawings we've done lots of prefer to improve drawing performance we've done a lot with system integration especially been working very closely with the courts team to make sure that everything in Tiger is working smoothly not only for courts but for java as well and we've got these new AP is available to you java 3d java AI and of course Jo GL we want to make sure that you do use your images wisely use this use the Java API is to create a compatible format and try not to manipulate pixels directly if you if you don't know what kind of format you're using or what you need to use and check out joggle and write your games in Java you really can do it now there's really nothing stopping you so for more information obviously we gave you the by tonic link earlier we've got that we've got a dmg of course with every session there's the reference library is available to you for all the Java items we should have a tech note coming out for image performance tips and the kind of things that I talked very much in brief today and there's a great article which is not new at all about painting and awt and swing and what the expected behavior should be on a standard java to platform and these are the kind of rules that that were guidelines that you should keep in mind when you start to do a lot of heavy drawing in Java this is a really priceless article and it's very similar to some of the stuff I talked about today it took a lot of my information from this from that article the Jo GL homepage is off java.net and of course you can go to Java suncom for anything else you're interested in that's not listed