WWDC2003 Session 421

Transcript

Kind: captions Language: en thank you welcome to sort of our second accessibility session obviously last year at last year's wvc we had sort of the inaugural session on accessibility in macula Sten and the reason why this takes place for the developer conferences because Apple is is building in accessibility into the operating system that's pretty much going to be the theme we're going to be covering throughout today's presentation accessibility is a pretty big field we highly have an hour to go over it so we have a pretty tight agenda and I sort of broken the session off into some major part first we give you a real brief overview of accessibility why it's important certain documents and standards you need to be aware of when addressing accessibility then you spend a little bit of time talking about apples approach to accessibility because previously with mac OS 9 we built some utilities and functionality that are available in the OS and it was third party developers who really had to be the foot soldiers for accessibility to create assistive technology solutions which are software and hardware solutions to adapt the functionality of a personal computer and software to the needs of the user with a disability and the you know interestingly enough that Apple is actually beginning to build in a lot of some infrastructure into the OS to support this community and then we're gonna actually sort of talk about universal access because we also want to have certain features and functionality available in Mac OS 10 know essentially out of the box that a user with a disability can can rely on and then we'll talk about sort of the infrastructure we're putting in the operating system to support this is the technology community again those developers to create the solutions that adapt common software and common hardware to the needs of persons with disability then I talked briefly about supporting the accessibility API because our approach that we're trying to do is to make it as lightweight as possible for the application developer although in certain cases there are things that an application developer can do they can interfere with making their app automatically accessible and we'll cover those then we'll talk about more directly to the assistive technology community I've already seen many of you I familiar faces in the NFC in the audience we're going to talk about how do I bring a piece of assistance LG to the platform which is again Apple is doing some infrastructure will work here to support accessibility and where does my solution may exist on Mac OS 9 or or in another platform how does that fit into our technology stack and we're also going to basically cover two points there one is how do I choose the right starting point when bringing that's just a technology solution over to Mac OS 10 and also which key technologies to be aware of to look at leveraging so Apple approaches accessibility and sort of a cross-functional way and this is sort of our mantra our world view of accessibility we consider you know we don't break the world into users without disabilities and users with disabilities it's a continuum of users and we want to support users with with all abilities and so it's important that you understand the way we approach this is we want to make sure that we can deliver an operating system that has a great experience and this is sort of you know our world view and this is definitely how the cross functional team which involves people from the relationship side at Apple involves technology evangelist like me it obviously involved several different groups and engineering who work on accessibility is a problem on Mac OS 10 and the key thing is that we believe accessibility is important and if you look at the statistics are these are the US government figures we're technically 54 million people in the US have some form of disability that's a huge number okay now obviously there's a there's a large continuum of disabilities and not all of them require assistive technology to overcome but I think the key thing is that if you look out statistically there's a large number of users out there that can benefit from having your application work with another party's assistive technology that works in conjunction with our infrastructure and the operating system more interestingly since the sort of the sentence of section 5 late and the issue of accessibility you have accessibility becoming a sales requirement and education the federal and local government spaces we'll get into what section 5 way it is I think on the next slide but the key thing is if your product sells your application cells into certain markets you're going to get asked about accessibility what is your section 508 compliance it's going to be the question you're going to ask I get asked and you know it's something you want to be able to answer correctly because if you don't have an accessibility story for your application you not might not be lacks with those markets then lastly it's really the right thing to do because i think that's difficult for people who have not worked with the disabled community or the assistive technology developers may not have a full understanding of how simple changes in your application can have significant effects in terms of opening the usage of that application to a person with a disability it's very interesting added one of my first contacts with assistive technology was I was at a conference and I was waiting to go off an elevator and I heard from behind me the mac and talk probe voice asked me to push the second floor button on the elevator and it turned out to be a person who used an iBook that had been converted to do you know work with them as a piece of assistive technology is essentially acted as that person's void now I use speech I often have the text-to-speech capability immaculate in read me emailed but I didn't really fundamentally understand that this technology it was a bonus technology for me was a pivotable technology to a person with a disability enabled them to communicate and so you really have to thank little things you can do inside your application can have significant reciprocal benefits to the disabled community and it's very much worth your time to engage that community to engage is's assistive technology developers to learn about this because it really change your mind on on how you approach accessibility so just being some checkbox on some sales form is something that's really the right thing to do quick note about section 508 in many ways section 508 has been driving this section five wages of federal requirement and essentially it's a it's a specification that the government's published that sits down a minimum accessibility standard for a variety of new devices and now new devices are things like operating systems applications electronic devices such as computers and I it's essentially sort of the same body of legislation and law that mandates curb cuts and variceal handrails and things like that and it just took a little while for those standards to be applied to items such as personal computers sector five late obviously was you know reauthorized or basically it's today it's coming down the pipe in 1998 the commute various communities are affected by this got together and started you know organizations working on defining what those requirements were and in June twenty-first of 2001 these standards actually became formal so there's actually a document that you can go get that tells you exactly what things your application or your operating system or your electronic device has to do to be accessible now one unfortunate situation that arise with a hard document first particularly a federal document you know outlining accessibility is in many ways it's not complete in many cases people consider that it is complete okay and this can create some issues it is a great starting place section 508 is a good starting place to learn about accessibility to figure out how to make your product accessible but it does not define everything that you could potentially need to do to open your products up to users with disabilities so to find out more about section 508 you can go to the access board gov and have a website specifically put together for section 508 accessibility now last year we covered this in a little detail as far as how to interpret it we have more content for you this year on the operating system side we don't necessarily want to tell you how to reinterpret it but one big tip that I had when learning how to interpret this document for Apple was essentially you know reading the faq they very recently put up a very complete faq site that has a lot of commentary and questions that's been posed by developers to is the you know tues access board to say hey how do you interpret this particular subsection and also read the commentary documents out section 508 was basically being formulated and coming into being lots of different companies asked questions and those were recorded and those have been distilled down into the commentary sort of audit trail as the requirement grew and there's fantastic information to look in there to see how one required might actually impact your application because odds are someone's already asked the question the key thing is when you first look at section 508 you're going to say I have to adopt all this in my application I have to fundamentally we are attacked re architect my application to support accessibility and you might think oh there's no way I'm going to do this this is just too heavy weight I just can't afford it or I'm just not architected for it but the reality is there's an interesting sort of scenario liya that section 508 lays out where there's really three communities that work together to create a fully accessible solution and it's not left up to any one vendor any one part of the chain of hardware vendor operating system you know company individual you know is V to essentially create the whole solution all themselves and so in reality you're going to look towards companies like Apple and Mac os10 the shoulder some of the burden of making applications accessible we're going to rely on you to adopt our accessibility API is that we'll talk about later and make your apps engage our platform accessibility features and then obviously there's a lot of fit and finish work that needs to be done by the experts in the field we're going to understand how to adapt a particularly need of adaptation for for example a screen reader for someone who has a site disability or for example someone who has a emotion disability and hath use an alternate input device like a system puff switch and or an eye mouse or head mouth so that community also gets engaged so between the three of us we sort of shoulder the burden and when you look at it from that perspective and you leverage all the core expertise it's actually a much more manageable problem and the key thing is that Apple can actually help you make the connection obviously we build an operating system we build hardware we work with software vendors to just have products that maybe not maybe they may not have an expert on accessibility we also work with the assistive technology community which in most cases arleta facto experts on accessibility so we can help you make those connections so feel free to contact us with questions about you know how does section 508 impact my application who don't need to talk to you about alternative input devices for example and we can help you out so let's talk about our approach to accessibility because again it's it's new when Justin when you introduce Mac os10 a lot of people are concerned they they didn't see much about support for assistive technology and accessibility built into the OS but in Jaguar last year we actually announced a whole bunch of additions to the u.s. to really support you know making applications accessible and support our users with disabilities we actually when looking at this problem because it is a big problem we actually factored out in two parts one is something called user level accessibility this is our built-in functionality those universal access features I'll talk about in a moment and this is when we looked at the things that we have to do to enable users with disabilities to engage our platform they were certain subset of them that it made sense for Apple to do and so we actually have built those and bake those into the operating system and I'll show you those in a moment and then it's a second you know in the second hand we realize there's a whole lot of adaptation that needs to be done to engage the whole community and we're not expertise in every you know every form of adaptation so we wanted to actually build infrastructure in the u.s. to enable the assistive technology community to engage the platform and so that's what the developer level accessibility is and that's really characterized in arm a close to an accessibility API so universal access some of you may have already seen this icon it's in the press panel of Mac OS 10 and this is really the hub if you will for our assistive technology features that are built in the user level accessibility that I spoke about so I can should we come over here too I think it's a demo machine three and give you a real quick tour of the universal access press panel and show you what we have built into Mac os10 some of this you may already seen in in Jaguar but we've actually made some enhancements to it very much in the case of user feedback what people wanted to see so I come here and I see right here of my universal access press panel I click on it and it's going to bring up essentially I said the hub of our user level accessibility features we sort of break it down into categories of disabilities so for example we have seeing hearing keyboard and mouse and if we take a look at the first which is seeing this is actually where I'm a graphics guy at heart so this is where I think the coolest feature is we actually have full support for screen enlargement inside the operating system and this enlargement is done in a very technically advanced way because the way our windowing system works and the fact that we use technology such as OpenGL to accelerate our display so we're actually able to do some pretty fantastic things here for example I can zoom in and I have full speed zoom it's very high quality and I can zoom in and out very fluidly and the other interesting point is that one of the problems with enlargers that people have been familiar with in the past is in many cases these only worked with windows and text documents so if there's other rich media that happened to be being used at the time they really couldn't participate because of the bottle necking and slow down then having to do all this copy pixel copying and replication would burden the system we wanted to make sure that we could architect the system that really didn't suffer from that so we actually created the system to where it's able to do things like for example and large even multimedia so it's a QuickTime pop-up felt the loop playback and now I'm able to freely zoom up and experience multimedia without having to have any conflicts with a screen and larger and I can actually enlarge to extremely large proportions and I now have a 35 foot tall pointer over there so it's really powerful feature now one thing that we also got his feedback was that people wanted more control over the zoom so we actually made it pretty flexible in here we're able to set your zoom how how far you want to zoom in zoom out if you wanted to show you where it's going to zoom into a lot of options like that we also have the ability to turn on and off the smoothing so if I am enlarge that I can turn on and off the anti-aliasing because Mac os10 is fully anti-aliased and in some ways if you have issues with contrast the anti-aliasing can you know make the type sometimes difficult to read this was sort of a first solution to be able to address that problem but in Panther we've actually gone one further actually zoom out a bit we'll see we have a new option and that is going to be the enhanced contrast and this is a really cool feature because we have a lot of feedback that the anti-aliasing of soft edges gays people with low vision problems and second is the pinstriping the pinstriping to be very difficult if you have issues of perceiving contrast so we actually sat down whatever you know crack graphics engineers who thought about it and came up with some very funky math to actually be able to do this effect where and let me actually go one further in and see the pinstriping I'm actually able to take it and move my screen and just change the the slider here and it's going to be able to make the pinstriping drop out and it's going to increase the contrast of the windows additionally it's going to harden up the edges of the type so even though we have anti-aliasing the type can get the we can harden up the edges of the type so if we look at it the type itself is going to become bolder and bolder and this is again and request it was put in specifically for features by by our users and it's system a system wide effect so and it's also variable so you can dynamically to knit we also have you know full support for a flashing alert for people who have hearing disabilities we have sticky keys and slow keys which are also very alternate input devices if you will for people with dexterity issues you may not be able to manipulate the keyboard is a you know for example they can only hold one key down at once like for example we turn sticky keys on and this is also pretty nice so they did a lot of fit and finish on this where I can hit the shift key individually and we'll see over here so I get a shift key and it sort of cached or held for me to hit another key and then it would like for example I've text that it open it actually typed that letter so it means that the user does not have to worry about hitting multiple combinations of control key shift keys to interact they can do single key presses actually cash and demonstrate you know up and the up on the screen and a semi-transparent way the keys that are currently the modifier keys that are currently cash they're going to affect whenever a main keys hit you also have mouse keys to enable the numeric keypad to out like a like a mouse we also have some other options so for example if i cruise over to something that's it's sort of interesting a lot of the features that we've built in in the universal access press pain we found a lot of people you know like using them regardless that they had a disability or not for example the screen and large turned out to be very larger turns out to be very popular in last year's wwc for people show code snippets we also have some features that work the other way so for example if i go into a keyboard and mouse and see new option called keyboard shortcuts and essentially this enables users to define hotkeys now anyone can use this but for user with a disability who might need to assign a hotkey one key press to do a lot of different operations this is very very powerful and then also we're going to have a demonstration a little bit later of another very important piece of assistive technology that's built into Mac OS 10 which is our speech which keeps on getting better and better so that was a quick tour of the universal access press panel and let me go back to slides so let's start to drill down on the developer sighs I talked about the user side the experience when the user get the machine and has the ability to essentially engage it even if they have a disability but now let's sort of work our way down into the infrastructure to talk about what application developers need to do and also talk about what assistive technology developers need to do we're gonna have to do a little bit of storytelling here because we had to really take a look at accessibility how it's handled on other platforms and also in Mac OS 9 and we built our infrastructure for doing accessibility and Mac OS 10 and one of the key things that sort of created the scenarios we built entirely new lat nua left Mac OS 10 is entirely from the ground up new operating system and that actually had for two challenges one is we had a lot of infrastructure to build second is for our developer community it was a little bit of a curveball particularly for the assistive technology community because their landmarks in terms of how they affected accessibility on the platform totally changed so one of the key things is that I think the biggest thing that really affected them was the protected memory model now obviously everyone was overjoyed to get protected memory on Mac OS 10 but it was horrifying if you were assisted technology developer because that meant that your app assistive technology application could not tell what another application was doing because there was a hard wall there was protected memory in between them and also some chord changes in the way we handle I o Mac loston has a very abstract and modular an object-oriented IO architecture and another thing that's actually an interesting challenge is our visual environment is really complex the ability to seamlessly composite 2d 3d multimedia to do the dynamic enlargement it's a very different world up in the frame buffer than it is and it was on other platforms and in the past but in many cases some of these challenges actually turn out to be benefit and we'll talk about those letters would go on but the key language we took this opportunity by the fact that we built a new operating system and we had to come with innovative solutions for the assistive technology community and developers you need to make their applications accessible we decided to say hey we're in here we're building infrastructure we might as well formally include that infrastructure and make it to where developers can rely on key parts of the system to do things on their behalf so the first thing we wanted to look at is how it was done the wrong way so we actually looked the way it was done on Mac OS 9 and it wasn't done the wrong way because you know anyone was making a choice to do it the wrong way it was really because it was the way the accessibility and assistive technology organically grew up on that platform because Mac OS 9 architectural II was a short stack of hardware with an irregular layer of system software and then the application sat on top of that it was all running in one memory partition so it was pretty easy for the assistive technology vendor to basically put themselves up on top and if they needed to change the behavior of running application or adapt the functionality the operating system they could just sort of have at it and go around and change things so in many cases assistive technology would essentially eject is between the application system software obviously assistive technology developers not the only people do it we had all these issues with extensions on Mac OS 9 and that also was used the same techniques but in many cases assistive technology vendors had to be more aggressive in some cases that even change the behavior of an application insert their code into a target application to modify the behavior or they would do things like you know get in between the hardware and start pulling events you know directly before the system software could get them and in many cases they would actually do trickery with the way the hardware presented itself to the Macintosh so you actually had a scenario where assistive technology devices and assistive solutions were in a very fragile sort of you know do-it-yourself environment and we really realized that this was the wrong way to do things we did not want to replicate this behavior on Mac OS 10 so that meant that Apple had to get involved in create this infrastructure and so we realized in our new operating system we have a common layer that's called the application frameworks layer in between all these different ways to develop from oculus 10 each application framework there's actually common infrastructure that runs between a lot of them so we said this is a nice choke point if you will to start building a fix accessibility into the operating system to support assistive technology and application developers we had to sort of factor the problem because each when the interesting about Mac OS 10 is it's got you know for discreet ways to develop for it from unix developer i can use darwin and bsd on the java developer have full java implementation if a developer as core competency on the macintosh from you know Mac OS 89 whatever I have those api's they've been carved know about significant bodies api's in carbon they've been brought forward and they're available in the platform and i also have cocoa which is a fantastic way to do object oriented development all these frameworks are very different from one another and making them all accessible was very tricky so we actually originally started with just carbon in cocoa so last year we were focused on just making carbon and cocoa applications accessible in the platform but for Panther we've actually extended that out to include Java as well so we're able to leverage the common infrastructure to essentially have all the applications you know sitting on top of the application framework layer they're using this poor infrastructure and so the idea here is to take the assistive technology solution which before had to do some very interesting things with the system and sort of make it just another application to where it was an application that just ran using the same infrastructure and was able to use that to do what it needed to do to have access into other applications we had to formalize some pathways with zach's access to occur so that's what the mac OS and accessibility API its api api level support for accessibility it's accessible to the standard c interface so it's accessible from carbon and cocoa it allows applications to discover and control the UI of other running processes ok because that when you look at it we looked at the whole field of accessibility that's one of the most common thread that a piece of assistive technology needs to be able to adapt the functionality of an application that may not even be aware it's being adapted so we give pretty low level control for one application to peer across the protected memory and look into another application and determine the you ike state and to be able to expect that state it supports some existing api's it was actually had in a system for a while that were more general purpose ap is for synthesizing events keyboard mouse event and so it can also be using injunction with those to create some very powerful solutions but more importantly this API said has really become a common ground for Apple and the assistive technology community to work on accessibility on the platform in fact a lot of the feedback we've gotten over the past year in terms of things you'd like to see in the way we handle text and text representation has been incorporated in the Panther and then obviously we introduced the core set of this and Mac OS 10 dot two and we've done some enhancements particularly in the area of text handling in cancer so the key problem we had to solve was how to let one application know what's going on in another application and the main problem was that if we look at a running application a piece of assistive technology and the fact that Mac os10 supports radically different application environment it have a pretty tall task because if you think of a Coco application way it thanks for example it's windows you know kind of control that it every an object that every every application is going to have a window it has a very different language from away a carbon application thanks of a window which is very different from the way a java application may think of a window and obviously java's its own own centrally virtual machine so it's even more complicated in the Java case so what we really didn't want to create a scenario is where the assistive technology vendor had to code up a solution for each individual framework because I would work for anybody that means either sister technology vendor area three times the amount of work or they would just choose one and say I have a piece of assistive technology that you only work with a Coco app but what if a user really wanted to have their carbon app adopted so we needed to wait in and solve this we need to take this sort of Tower of Babel in terms of controls and you I representation and try to fix this so we've kind of come up with a lingua franca called a UI element where what we did is we essentially modified back in Jaguar parts of the system so they could express you I in a concept called a UI element an abstraction and they can all use this abstraction pass it over the assistive technology application that can then parse this abstraction and figure out what's going on in any target a poor gardless of framework so what does UI element again it's a frame we're going to pendant abstraction for user and user interface and it will actually have a little diagram of how a dialogue works but you can think of essentially an application is appearing of a hierarchy of these UI element abstractions because a UI element can actually have attributes in terms of you know the title of the window or the value of a particular control or slider for example a UIL McCann also have children attribute so for example we have a window and it contains within it a child with Mike which might be a button and obviously certain controls or intercept actions user interface events and so you can actually have UI element can have an action as well such as a prep action or an increment action it's a whole suite of those so let's actually sort of take a look because I think it's better to sort of describe how this really works in reality and say show how the UI elements and the accessibility API work to overcome the problem of one application a piece of assistive technology need to know what's going on in the user interface of another application so here we have a theoretical piece of assistive technology and we have another application that's just you know my app that's running obviously they're separated by protected memory model learn different processes so we need to somehow to do to start a dialogue to determine the user interface state so the first thing I do is get the process ID and I was a couple different api's to do this but once I have the process ID that's going to I'm gonna have to basically asked for you I LM at the initial UI element for this process ID so i will call a x UI element create application path in the process ID and i'm going to get back a UI element that's going to contain some information the top level information on that application and it's generally going to have a role in this case it's an application so the object i vas is going to say hey I'm an application it's going to have a title hey on my app I'm not frontmost in this case because that's the interesting thing previously a lot of trickery had to be done by always targeting events at the frontmost application to control you I because that's going to be the application that receives you I but the interesting thing here is you know having a dialogue with UI elements is not dependent on the application being always frontmost and then also we're going to get information back that this application has a bunch of windows and obviously if we're trying to parse what's going to application we're going to need look at the windows so the next thing we're going to do is take the UI elements and essentially ask the application say x UI element copy attribute value and say I'm interested in the windows of that application and it's going to return obviously the window and applications and in theory this application has a window called my document it tells us with the position it also can tell us that that window has some children and those are probably controlled so I'm going to keep on going down this path and continuing to ask these questions through the accessibility API to find out more and more about this application to sort of explore and parse DUI so i'll say ax UI element copy attribute value again but asked for the children value by asking KX child children attribute and that's going to give me back the children the children that are in that the previous window that we're in and so I'm going to get back today this window contains a button and it's labeled print and it's enabled because that we get state information back from using the accessibility API and here's its position on the screen well obviously I might want to be trying to make this application print so I know that hey it's a print button it's a button I can have to send some sort of event to effect that operation so I'm going to basically say hey and say aax UI element copy actions is going to tell me the actions that are available to that particular element I'm interested in and in this case I see that I have the X button this is print it sports true that it's a that's enabled and it accepts event called a xpress so now i know i can actually send a button press event to that control and then the a the actual application will receive that event just as if the user actually did it themselves so that's an example to dialogue that you can have with accessibility API s and there's also a second part which is a notification obviously it would not be a really good idea for us to make assistive technology developers continuously parts of running you I of every application continuously having to basically you know always sample an application to determine that the UI state has changed so once you've actually gotten the you know the lay of the land inside an application you might only be interested in certain controls and you can basically say hey there's a control over here you can notify me if something happens to that control so obviously the key ones are like if windows get created move or destroyed if the application itself is brought to the front or sent to the back even things like if UI element that I've been interested in the past gets destroyed it goes away I obviously need to know that if I'm trying to control that that that you I or even parse or adapt that you I and also value change which is really useful for controls like sliders that have values that can change on the basis of user interaction it's like what I did like to do is invite Mike umber to the stage and he's going to take us and that is his actual job title right there and and he's can actually show us the UI element inspector UIL inspector is actually sample code that you guys can pull down and take a look at and it's basically a way that you can use the accessibility API s and Roman action okay hello my name is mike amber and i'm going to start off with a demo of UI elements vector that's going to sort of give a live illustration of what travis has been talking about and then I have a few slides only six so don't worry and I'm going to cover some of the things we've added for panther in the accessibility API s and then finally I'm going to end with another demo of a fun little app so you ilm inspectors available at sample code from DTS it has been for a while so some of you may already be familiar with it and I'm going to start by launching it and what this program does is every time the mouse moves it checks what UI element the cursor is over so as I move over different things you can see that the contents of this window change I'm going to increase the font size so people can read it a little better and and now I'm going to launch TextEdit we're going to do some demonstrations of what it can do with text edit so here I'm over an untitled window and text edit and we can see that we get the the list of attributes that the window supports things like the role the role description the title at the top of the window is showing the position of the window in the hierarchy so you can see at the top of the hierarchy is the application itself and then this window is one of its children now one of the sort of challenging things with the UI of this program is since whenever I move the mouse it changes what I'm looking at if I want to operate on that thing that's a problem so use a hotkey and you can lock in on the particular UI element that you're looking at and everything turns to red to indicate that and then this little interaction window appears and I can't increase the font size in it so I have to apologize but I'll describe in detail what I'm doing so we can see where we are now locked into the window and one of the things you can do is use this pop-up menu and get a list of all the attributes in the window which we've already seen above and we can see some of them have w's next to them and that means they're writable so i'm going to pick the position attribute we can see its position is given an XY coordinates and I can change the y coordinate and hit set value and the window move another fun one is the minimized attribute and it's a boolean it's now at zero and I could change it to one set it and the window disappears and then I'll bring it back by setting it back to zero and this illustrates the point that sometimes people new to the api's forget is that actions aren't the only way to interact with UI element a large amount of your interaction is going to be accomplished through setting the values of writable attributes ok so now i'm going to bring up text edit preferences pane and I'm going to illustrate a couple more features of UI element inspector i'm going to focus in on this text that labels this box and there's this little checkbox in the interaction window that lets you highlight the thing that you're locked on to and so this is useful for going around the app and just seeing what you can hit so I'll demonstrate here we've hit a another button here we can hit a pop up and I'll go back to that label and then also in the interaction window is a pop-up named go to and you can actually walk the hierarchy so if you use your mouse to get close to where you want to be and then we can go up to the parent we see the highlight changes to reflect that that title is actually a child of that box and then we can go up another level in the box as a child of the window and I'm going to show one more thing so you might be asking the question can you ILM inspector inspect itself and the answer is yes it can and I'll in the demo by having it close itself so we can hear I'll perform an action which is a xpress on the close button of the window and we're done okay [Applause] alright so as we just showed we have this model of attributes and values and you can ask a UI element you can pass it int name as an attribute you can get back to value and this really suffice is for most purposes and this is and this was really all we had in in Jaguar but we found that sometimes just passing the name as an attribute isn't quite enough so in Panther we added this concept of parameterised attributes and for instance you might want to know what character the mouse is over and in that case you need to pass in the position of the mouse or you might want to know the bounding rectangle for some text and in that case you need to pass in the range of characters of the texture interested in so we added this concept of parameterised attributes it's conceptually similar to attributes but now when you ask for the value you pass in a name and you also pass in a parameter and the type of the parameter varies and I'll go over some examples of that in the upcoming slide so to accomplish this we added two new AP is in Panther the first one is ax UI element copy parameterised attribute names you can see we kept brevity in mind when naming these api and the second one is ax UI element copy parameterised attribute value so the first one is very similar to an existing API that returns you a list of all the attributes this one returns you a separate list of the parameterised attributes and it's a separate list for compatibility reasons we didn't want existing code bumping in to parameterize the attributes and trying to get their value and not passing a parameter so since they're separate list that can't happen and these and the second call is what you're going to use once you've got the name of the parameter ID attribute you're going to use you get to pass in the UI element the attribute the appropriate parameter and then you give back your value and for now we only support reading not writing of the parameterised attribute values but if the need arises we might consider extending that in the future so no presentation would be complete without a giant table of very small text so I apologize if you're having trouble reading it but i'll i'll go over some of the items in detail this is a list of the parameterised attribute that we have currently implemented in panther and they all have to do with text so there are a lot of requests from developers to find out more detailed information about text out with stylist in how it's formatted on the screen and this is where we found we really need we really needed a parameter along with an attribute name in order to answer these requests so the first one is KX line for index parameterised attribute so this one is addressing the issue of the text layout if you pass the index into a text string then we're going to tell you what line is on and then the second the the second attribute up there does the opposite if you give me a line number I'm going to tell you the range of text it's on that line then there's a string for range which will given a range of text allow you to just get that much of it before you could ask for a text object text in its entirety this lets you get it a chunk at a time then there's one that'll tell you if you specify character position what range of texture over and you have to keep in mind that you know the mouse might appear to be over a single character but in fact because of surrogate pairs or multi-byte glyph you might be over several characters so we return the whole range then there's range for index and this again you have to do with multi by characters if you specify an index in the in the text will give you the full range of characters that comprise that glyph so there's call similar to this like range of compose characters in the kit and this is sort of the accessibility equivalent to that then we have some that have to do with layout given a there's the KX range for KX bounds for range parameterised attribute so you give a range of text and we're going to give you back a bounding rectangle and then there's the KX RTF for range parameterised attribute so again you can give us a range of text and we're going to hand you back the RTF representation of the text and from there you can extract the font the size the color just about everything you could want to know and then finally there's this KX style range for index parameterised attribute I'm sure you guys are going to enjoy typing these names in just as much as I am speaking them right now and this last one lets you go through your text and style run so if you're interested in when things change you can you can use this and find out you know the extent of the current style run and then move on to the next one there's a couple of other attributes we added for text accessibility in Panther and these aren't parameterised attributes the first one is just to tell you how many characters are in a in a text object and the second one is to tell you what range those characters are currently visible and the next two are a little more esoteric these have to do with text that flows if you have for instance a two-column document that might be two separate texts UI elements and the text flows between them so the first one the shared text UI element attribute tells you what's your portion of the text that you're displaying because in general most of the attributes if you use ask for the text they do to treat this treat you as if you have all the text because that's most most the time what you want and it's most convenient but if you really need to get to find out what exactly only this UI element has you can use this attribute to find it out and then the last attribute ax shared character range attribute actually I got these backwards the second one is the one that will tell you your share the first one is going to return you an array of other UI elements where your text is shared with so you know it would just return you the other columnist your text to split between two columns now here's a few other untaxed related attributes who's added for Panther first we had a lot of requests for people wanting to be able to be able to identify the default button so we have a we have a parameter on windows that will tell you the default button and we also have one that will identify the cancel button we also added a couple of notifications so when drawers appear or when sheets slide down you can get a notification when they're destroyed you get the normal UI element destroyed notification so we didn't have to add anything there we also added some new sub rolls for windows again this is a we have a lot of requests from developers wanting to be able to distinguish a floating window possibly ignore it from a dialog from a regular window and so we added a variety of sub rolls that will identify dialogues floating windows and and then there's an all-purpose unknown if someone puts up a custom window we can't tell what it is and so I'm going to finish up with a demo of a little athlete call monkey see monkey do so I'm going to start by launching text edit and this is a typical document you might work with on any day with a variety of tastefully formatted text different styles and different colors and next I'm going to launch I'm going to move this out of the way and then I'm going to launch monkey see monkey do and this app its name monkey see monkey do because basically that's what it does it uses only the accessibility API and it tries to replicate the user interface of another app so I'm going to from the list it presents me i'm going to pick text edit and it's computing its own you know it's creating its own window that's supposed to look as close as it can get to the original app and again all excusing is the accessibility API to get the information it needed and it's actually completed a while ago I'll just have to move it off so again this illustrates that all the rich text information is available to you through the accessibility API so I'm going to demonstrate some some other features of monkey see monkey do first I'll go to TextEdit I'm going to close this window and you'll notice monkey-see monkey-do removed its copy that was a example of getting a notification of the UI element destroyed and now I'm going to bring up preferences now this is actually a little tricky because when text etics deactivated its preferences panel disappear so that adds a little bit of a wrinkle to this but monkey see monkey do is already done it's best to replicate the Preferences panel and you might notice you might have noticed the change if you were paying attention this one doesn't quite look right and that's because there's no way through the accessibility API to discover whether you're using regular size controls tiny controls or you know small control so it take you could take a guess based on the size and here it's not guessing quite right so I'm going to pull aside monkey see monkey do version of the preferences and again text that it's not in the foreground so that's why you don't see the other window but as soon as I bring it back you'll see it and using notifications there's some more connections we can show for instance we can pick something from the pop-up menu of course I picked one it was concealed so you can barely see it at the below the upper window i'll pick i'll go back to automatic and you can see a change next I'll click a radio button and you can see it changes between the two and i can do illustrate some typing we can change the the window with to a hundred and when i hit enter it will propagate down so so again this this application is just sort of a illustration of what's possible to do with the accessibility API ok back to trap let's go back to slide so now when I talk about is if you're an application developer and touch briefly on how do I engage the accessibility API all the sort of infrastructure we've been talking about the first half the presentation and there's go through it really quickly because we did quite a bit of this last year in terms of going in great detail so there should be safe sessions on ADC TV and the dvd-rom on carbon accessibility there's also one in cocoa text I mean that cocoa controls and also the accessibility over D session have a lot of this content in greater detail but the key thing is obviously there's multiple types of applications that work on different frameworks on Mac OS 10 and the accessibility API can work with carbon applications but there are some caveats essentially we've been telling developers and carbon applications to use standard controls for a long time for other reasons besides accessibility if you were one of those developers who took her advice you're in pretty good shape because we were able to make a lot of standard controls accessible in Jaguar and Panther however there's a long sordid past of carbon developers doing their own controls if you do this in your application you will have to actually do some work to support the accessibility API in your application the other thing is important as your app should support carbon events as well and that also is one of the ways accessibility API work the other thing that is a sort of an area of difficulty and it's something that we don't have total control over as much as we'd like to is that in carbon there's also a history of people doing their own text engines or you know you name it license.txt engines they roll their own there's several different ones available in the osm LTE there's text at it and it's too big of a problem for us to solve on our own so carbon developers want to use the parameterised attributes are going to have to add some support into their text engine for supporting these api's something we'd love to fix if we could but it's too big a problem for just apple to tackle we're gonna have more information published on this topic on techno 2092 and that is going to be forthcoming and we'll have lots of detail on what carbon developers need to do to properly support the accessibility API good news on the other frameworks Coco it's a great story there the cocoa team actually modified app kit to support the accessibility API last year and they continued obviously increase that support and because the way Coco works essentially you're always inheriting from parent classes applications that may not have been a war aware of accessibility and say for example Mac OS 10 dot one when those applications are running 10 dot 2 or in Panther those applications will be accessible there's still the possibility for Coco apps tap custom controls there are ways to make those custom controls accessible Java applications an interesting story here is you don't need a Java bridge accessibility bridge like you do another platform well the java accessibility bridges bridges out the accessibility from the platform native side where the assistive technology has to run over through to the sandbox we don't need that on Mac os10 it was you know Java and Panther you just basically we've done the work for you and awt and swing applications that use standard controls and those environments are going to be accessible as well let's talk about brief it about bringing your system technology to Mac OS 10 this is a more for the assistive technology developers who are looking to engage our platform there's a lot of ways you can go wrong because it's sort of like a restaurant with a huge menu and we want you to choose certain poisons and putting a certain items we have like the chef's special to do the right technologies and so I'm going to give you some tips right here one of the things that we have been dealing with assistive technology community for three years for apple and it's a continual ported confusion as to where you should start how should you begin coding your sister technology application and it sort of become clear to us and looking at all the options repeatedly with a lot of developers that if you're in a situation we're trying to bring a new assistive technology application up on that goes 10 you want to do it in cocoa it's just movies faster more efficient for you get a lot of platform benefits by using cocoa and you also get a lot of stability because the way our texture works where cocoa can actually innovate and change and you'll inherit those advantages and the rapid application development aspect of cocoa has a lot of assistive technology developers a small shop where and this is going to be tremendous productivity gain for you guys so we hardly write and you really recommend you take a look at Coco that you're making assistive technology device such as an alternate input device ahead mouth you know I tracker a switch I think that one of the problems that we see a lot in this community's you guys use a lot of legacy i/o cereal you know ps2 mice the reality is you know Apple sort of you know really helped catalyze USB when we introduced the imac a long time ago yes these ready so you know if you have a ps2 assistive technology device you're going to want to really think how do I move it to modern serial i/o and most of your case it's going to be USB but there's also interesting possibility with things like firewire where an eye tracker instead of being this large multi-thousand dollar box and specialized custom image processor done in Hardware could be a relatively inexpensive firewire camera with you know using the you know the real-time capabilities and the high-speed throughput a firewire able to actually pump it to data into software that's running on Mac OS 10 which is using maybe our platform tune DSP and math libraries to do the actual calculations to track the eye so then you can take by using the proper i/o and clever software you can essentially factor your problem from a very expensive hardware implementation down to potentially very inexpensive and affordable hard hardware the other interesting thing is when doing any of the new serial i/o please follow standards it doesn't do any good to do the i/o and then not follow the standards and this is agent Lee happens where people deliver an excellent piece of assistive technology but they'll make it work like a keyboard so when the user hits the switch input device it just says letter a letter a letter a and then it becomes a complicated software problem to determine okay where'd that letter A come from it comes from the assistive technology device versus did it come from the keyboard and software layers of software have to be put in the foot and putting the place to the code that where if you follow for example USB HID spec which is all sources of input devices you could say this is a button and that's much clearer than having a switch emulate a piece of a keyboard there's also interesting for a wireless opportunity such Bluetooth and also force feedback we support for to be back in Mac OS 10 and i think that's to be an interesting to mention to provide feedback back to the user when using an alternate input device and we have some speech technology that's available in the operating system a really good speech technology we've been working on it for a number a number of years and that's available for assistive technology developers to do text to speech it's also available for to do speech recognition and I'd like to invite Kim Silverman up this stage and we're going to quickly do a configuration adjustment and this was sort of an impromptu ad today and do a quick demonstration of some of the excellent speech capabilities built in tobacco sent by Trevor wall shows is setting up there are a whole lot of speech features built into the operating system we don't have time to show you even a small proportion of them we're just going to step through particularly what's relevant to the accessibility API thank you so there are speech synthesis and speech recognition capabilities that use that let's talk about the speech synthesis first excuse my back I'll be looking at the screen here we're going to go to the system preferences and the speech preferences we're in the spoken user interface tab I can turn these things on the first thing I'll turn on is selected text when the key is pressed it's going to ask me for hotkey i'll use f8 and that means in any application if i have some text selected I can speak it just by pressing f8 so we've got some news which I just brought up before the session today I'll press f8 and we ought to hear it security talks between and clearly in Palestinian leaders are making good progress so there's a selective text when the key is pressed and this is one of the disability features that like many become popular amongst people who don't have disabilities as well it means if your type a document for example and you want to hear how it sounds you want to prove it you just select it and hit your hotkey and it's read back that is useful for detecting spelling mistakes because spell checkers can only detect Tytos that don't make words if you ask any cognitive psychologist you'll point you to a lot of data showing that most of the typos that we make make other real words and therefore spell checkers can't catch them then we have this item here text under the mouth this again uses the accessibility API and when I turn it on you'll hear it may speak already the name of that checkbox thanks don't even know and as I move this around and speak other controls speech recognition I'm not doing anything to launch this it's happening spontaneously it's all boys so can lose your interface ok we'll turn that guy off otherwise your drivers crazy now let's go through speech recognition if ever you try out the speech recognition the first thing you should do is give it a chance to normalize to the way your voice sounds in the acoustic environment in which you're using it we have built-in automatic tracking and adapting to the acoustic environments but you can speed it up I'll show you how and I'm going to do it now because I'm getting echoes from the back of the wall which are quite a long delay and it's going to learn about that under we built the speech recognition to work with the microphone that's built into most of our machines I mac ibook powerbook Emacs tower machines don't have a built-in microphone you can get a number of third-party USD head mounted microphones I'm using this one which was produced by our buddies over at vxi those guys really know what they're doing there are other brands you can use as well so you go to a speech recognition preferences to the listening tab and click on volume now as I'm speaking I can adjust the volume so that my speech Peaks just go into the red and then my task is to read down these commands and it's listening to me now and as it hears me say one of these and recognizes them it will flash what time is it quit this application open a document show me what to say make this page speakable move page down hide this application switch to find them so ostensibly we just let you set the volume on the larger phone but what you don't know is we just set nearly a thousand parameters they're sending our spectral properties of this microphone and me and this acoustic environment now we're ready to use the accessibility features I go to spoken user interface what's no I don't speech recognition command and here there are a number of sets of commands that can be spoken from the speakable items application oh I should turn it on first we go to unearth ternal on here this brings up speakable item and there's our old familiar command window and zooming under that for a moment well there it is I like that and I don't normally do this but I will turn on speak confirmation which means it will speak back to me anything that our hears me say what time is it when a giant head its 1136 ok this is speaker independent I've not trained this to my voice other than the normalization that you just saw me do and that all of you will do when you try this in your hotel room tonight won't you yes all right down here turned off by default is front window command and menu bar commend both of these users accessibility API that will turn this guy on and now I can speak through any front window commands in any applications using real control I'm showing you this because you're going to keep using real Apple controls and you default voice in silence boys spoken user interface spoken who's the interface we have not programmed anything specimens of these preferences so we can go outside of our preferences that will still work show all cheer all let's go and do another one I'll just pick one at random let's do a doc and magnification magnifications so they would even see that the speech was developed as a client for the accessibility API Travis mentioned that it gives you and our common ground we use speech as a client so that we could make sure the accessibility API worked before we gave it to you it also means that if you use standard controlled you get a lot of accessibility in your application just for free and three accessibility API you can add even more I think that's enough for that thank you thank you go back a slide please so that's pretty cool stuff we got great speech technology Mac OS 10 and it builds on top of the accessibility API to do some very interesting things so let me talk about the last point I got to kind of hustle through it Makka soon has a unique visual pipeline we have a compositing windowing system and also windowing system it's very aware of the idea of time to layer things on top of one another and this is sort of unique in the industry yeah I don't want to go to a technical description of how the actual pixels flow to the you know to the frame buffer but the key thing is that there's one process it always knows what's going on the frame buffer and that process as lowest level primitives as a concept of a window and this means assistive technology can do some very interesting things and also solve some problems for assistive technology one thing is that the windowing system does support the ordering so it's very possible for assistive technology applications to that need to be always on top to have to do that it's just basically setting in the proper layer and the z-order stack and then applications can't put their content on top of and piece of assistive technology content that needs to be always on the screen and then also the compositing gives all sorts of opportunities for the visual presentation of assistive technology because in many ways there's always a fight between Skrillex green real estate between the assistive technology and the real estate that the application that's being adapted assumes is that it's using so I actually want to show you a really quick demo of a piece of sample code that actually sort of shows an interesting way to maybe solve this problem so let me go to yes that's right there my machine it's going to bring an application that engineer wrote that's showing into something called a overlay window and this is a just a simple application that I had one of our DTS engineers do works with Coco come up with it's going to essentially demonstrate a prototype if you will of what's called a scanning application assistive technology which is soft it usually works in conjunction with a switch that the user will hit the switch and basically a pallet will pop up and then a selection will start iterating over that pallet they hit the switch again and then they can select an item of maybe actions that are encoded in that pallet and the interesting thing is you know this is showing the idea of assistive technology being able to move out of the way of applications because it's able to use the macula strands ability to composite and do a transparency and blending to where this could easily be made as the user needs that it could fade forward and begin its scan and then it could fade away when the user needs to maybe view a web page or something underneath it and also one last point is this little sample is about 200 lines of coke 0 code i believe i remember my numbers correctly and the interesting thing is that you know one of the things that typically people need to do in scanning applications and support a large number of tiles or different little iconic representations of the actions and the interesting about the power of cocoa and mac OS 10 is the fact that i'm able to put a variety of different file formats into this my little scanning application here and the system of the stall displays them without me having to know anything as a developer how to decode them this will even work for PDF files so i could use a i was a standard software manufacturer that did an assistive technology scanner i could then use an ilitch tool like illustrator a free hand to create my high-quality artwork which would then be dynamically scaled down and have the best possible vector fit for that piece of assistive technology this is a piece of sample code available off on the dts website so let's go back to slide when last point I want to make is that there's an issue that is sort of happening right now and between assistive technology and the propensity for applications that begin to grab the full screen to take over the display it's a functionality we offer in Mac OS 10 a lot of games use it and it creates a problem that let's say for example someone wanted to use an alternate input device which required a piece of assistive technology software running in the computer to be able to display its content on top of maybe the game that's running that would be difficult to do the game says the screen is mine so here's sort of a recipe or recommendation if you're a game developer or you have an application that does grab the full screen now you can use the z-order capability of mac lessons compositing window model to essentially put your window at a particular level allows us the technology to work it's pretty basic and so taking over the screen you just simply hide the menu bar you then hide the dock then what you do is you bring up a full sky full size window without any sort of borders or controls around it and you put it in particular level kind of screen size of a level plus 1 and this there's another layer that's above this essentially allows assistive technology to appear above your your application this can be like 2d 3d it can be multimedia so in theory you could have a piece of assistive technology with a semi-transparent menu sitting on top of maybe a game a 3d game if it was designed that way and the window layers are defined in CG window level H which is in the courts 2d headers so in summary we basically covered a lot of ground here we've talked a little about accessibility from the aspect of why you should do it we've talked about what technology Apple's built into the operating system to accommodate accessibility both from a user and developer perspective we've shown you some new api's that we put in Panther to really benefit assistive technology developers who need to get at text attributes and then additionally I've made some recommendations for both application developers an assistive technology developers how they can better leverage the technology portfolio on Mac OS 10