WWDC2003 Session 421

Transcript

Kind: captions
Language: en
thank you welcome to sort of our second
accessibility session obviously last
year at last year's wvc we had sort of
the inaugural session on accessibility
in macula Sten and the reason why this
takes place for the developer
conferences because Apple is is building
in accessibility into the operating
system that's pretty much going to be
the theme we're going to be covering
throughout today's presentation
accessibility is a pretty big field we
highly have an hour to go over it so we
have a pretty tight agenda and I sort of
broken the session off into some major
part first we give you a real brief
overview of accessibility why it's
important certain documents and
standards you need to be aware of when
addressing accessibility then you spend
a little bit of time talking about
apples approach to accessibility because
previously with mac OS 9 we built some
utilities and functionality that are
available in the OS and it was third
party developers who really had to be
the foot soldiers for accessibility to
create assistive technology solutions
which are software and hardware
solutions to adapt the functionality of
a personal computer and software to the
needs of the user with a disability and
the you know interestingly enough that
Apple is actually beginning to build in
a lot of some infrastructure into the OS
to support this community and then we're
gonna actually sort of talk about
universal access because we also want to
have certain features and functionality
available in Mac OS 10 know essentially
out of the box that a user with a
disability can can rely on and then
we'll talk about sort of the
infrastructure we're putting in the
operating system to support this is the
technology community again those
developers to create the solutions that
adapt common software and common
hardware to the needs of persons with
disability then I talked briefly about
supporting the accessibility API because
our approach that we're trying to do is
to make it as lightweight as possible
for the application developer although
in certain cases there are things that
an application developer can do they can
interfere with making their app
automatically accessible and we'll cover
those then we'll talk about more
directly to the assistive technology
community I've already seen many of you
I familiar faces in the NFC in the
audience we're going to talk about how
do I bring a piece of assistance
LG to the platform which is again Apple
is doing some infrastructure will work
here to support accessibility and where
does my solution may exist on Mac OS 9
or or in another platform how does that
fit into our technology stack and we're
also going to basically cover two points
there one is how do I choose the right
starting point when bringing that's just
a technology solution over to Mac OS 10
and also which key technologies to be
aware of to look at leveraging so Apple
approaches accessibility and sort of a
cross-functional way and this is sort of
our mantra our world view of
accessibility we consider you know we
don't break the world into users without
disabilities and users with disabilities
it's a continuum of users and we want to
support users with with all abilities
and so it's important that you
understand the way we approach this is
we want to make sure that we can deliver
an operating system that has a great
experience and this is sort of you know
our world view and this is definitely
how the cross functional team which
involves people from the relationship
side at Apple involves technology
evangelist like me it obviously involved
several different groups and engineering
who work on accessibility is a problem
on Mac OS 10 and the key thing is that
we believe accessibility is important
and if you look at the statistics are
these are the US government figures
we're technically 54 million people in
the US have some form of disability
that's a huge number okay now obviously
there's a there's a large continuum of
disabilities and not all of them require
assistive technology to overcome but I
think the key thing is that if you look
out statistically there's a large number
of users out there that can benefit from
having your application work with
another party's assistive technology
that works in conjunction with our
infrastructure and the operating system
more interestingly since the sort of the
sentence of section 5 late and the issue
of accessibility you have accessibility
becoming a sales requirement and
education the federal and local
government spaces we'll get into what
section 5 way it is I think on the next
slide but the key thing is if your
product sells your application cells
into certain markets you're going to get
asked about accessibility what is your
section 508 compliance it's going to be
the question you're going to ask I get
asked and you know it's something you
want to be able to answer correctly
because if you
don't have an accessibility story for
your application you not might not be
lacks with those markets then lastly
it's really the right thing to do
because i think that's difficult for
people who have not worked with the
disabled community or the assistive
technology developers may not have a
full understanding of how simple changes
in your application can have significant
effects in terms of opening the usage of
that application to a person with a
disability it's very interesting added
one of my first contacts with assistive
technology was I was at a conference and
I was waiting to go off an elevator and
I heard from behind me the mac and talk
probe voice asked me to push the second
floor button on the elevator and it
turned out to be a person who used an
iBook that had been converted to do you
know work with them as a piece of
assistive technology is essentially
acted as that person's void now I use
speech I often have the text-to-speech
capability immaculate in read me emailed
but I didn't really fundamentally
understand that this technology it was a
bonus technology for me was a pivotable
technology to a person with a disability
enabled them to communicate and so you
really have to thank little things you
can do inside your application can have
significant reciprocal benefits to the
disabled community and it's very much
worth your time to engage that community
to engage is's assistive technology
developers to learn about this because
it really change your mind on on how you
approach accessibility so just being
some checkbox on some sales form is
something that's really the right thing
to do quick note about section 508 in
many ways section 508 has been driving
this section five wages of federal
requirement and essentially it's a it's
a specification that the government's
published that sits down a minimum
accessibility standard for a variety of
new devices and now new devices are
things like operating systems
applications electronic devices such as
computers and I it's essentially sort of
the same body of legislation and law
that mandates curb cuts and variceal
handrails and things like that and it
just took a little while for those
standards to be applied to items such as
personal computers
sector five late obviously was you know
reauthorized or basically it's today
it's coming down the pipe in 1998 the
commute various communities are affected
by this got together and started you
know organizations working on defining
what those requirements were and in June
twenty-first of 2001 these standards
actually became formal so there's
actually a document that you can go get
that tells you exactly what things your
application or your operating system or
your electronic device has to do to be
accessible now one unfortunate situation
that arise with a hard document first
particularly a federal document you know
outlining accessibility is in many ways
it's not complete in many cases people
consider that it is complete okay and
this can create some issues it is a
great starting place section 508 is a
good starting place to learn about
accessibility to figure out how to make
your product accessible but it does not
define everything that you could
potentially need to do to open your
products up to users with disabilities
so to find out more about section 508
you can go to the access board gov and
have a website specifically put together
for section 508 accessibility now last
year we covered this in a little detail
as far as how to interpret it we have
more content for you this year on the
operating system side we don't
necessarily want to tell you how to
reinterpret it but one big tip that I
had when learning how to interpret this
document for Apple was essentially you
know reading the faq they very recently
put up a very complete faq site that has
a lot of commentary and questions that's
been posed by developers to is the you
know tues access board to say hey how do
you interpret this particular subsection
and also read the commentary documents
out section 508 was basically being
formulated and coming into being lots of
different companies asked questions and
those were recorded and those have been
distilled down into the commentary sort
of audit trail as the requirement grew
and there's fantastic information to
look in there to see how one required
might actually impact your application
because odds are someone's already asked
the question the key thing is when you
first look at section 508 you're going
to say I have to adopt all this in my
application I have to fundamentally we
are attacked re architect my application
to support accessibility and you might
think oh there's no way I'm going to do
this this is just too heavy weight I
just can't afford it or I'm just not
architected for it but the reality is
there's an interesting sort of scenario
liya that section 508 lays out where
there's really three communities that
work together to create a fully
accessible solution and it's not left up
to any one vendor any one part of the
chain of hardware vendor operating
system you know company individual you
know is V to essentially create the
whole solution all themselves and so in
reality you're going to look towards
companies like Apple and Mac os10 the
shoulder some of the burden of making
applications accessible we're going to
rely on you to adopt our accessibility
API is that we'll talk about later and
make your apps engage our platform
accessibility features and then
obviously there's a lot of fit and
finish work that needs to be done by the
experts in the field we're going to
understand how to adapt a particularly
need of adaptation for for example a
screen reader for someone who has a site
disability or for example someone who
has a emotion disability and hath use an
alternate input device like a system
puff switch and or an eye mouse or head
mouth so that community also gets
engaged so between the three of us we
sort of shoulder the burden and when you
look at it from that perspective and you
leverage all the core expertise it's
actually a much more manageable problem
and the key thing is that Apple can
actually help you make the connection
obviously we build an operating system
we build hardware we work with software
vendors to just have products that maybe
not maybe they may not have an expert on
accessibility we also work with the
assistive technology community which in
most cases arleta facto experts on
accessibility so we can help you make
those connections so feel free to
contact us with questions about you know
how does section 508 impact my
application who don't need to talk to
you about alternative input devices for
example and we can help you out so let's
talk about our approach to accessibility
because again it's it's new when Justin
when you introduce Mac os10 a lot of
people are concerned they
they didn't see much about support for
assistive technology and accessibility
built into the OS but in Jaguar last
year we actually announced a whole bunch
of additions to the u.s. to really
support you know making applications
accessible and support our users with
disabilities we actually when looking at
this problem because it is a big problem
we actually factored out in two parts
one is something called user level
accessibility this is our built-in
functionality those universal access
features I'll talk about in a moment and
this is when we looked at the things
that we have to do to enable users with
disabilities to engage our platform they
were certain subset of them that it made
sense for Apple to do and so we actually
have built those and bake those into the
operating system and I'll show you those
in a moment and then it's a second you
know in the second hand we realize
there's a whole lot of adaptation that
needs to be done to engage the whole
community and we're not expertise in
every you know every form of adaptation
so we wanted to actually build
infrastructure in the u.s. to enable the
assistive technology community to engage
the platform and so that's what the
developer level accessibility is and
that's really characterized in arm a
close to an accessibility API so
universal access some of you may have
already seen this icon it's in the press
panel of Mac OS 10 and this is really
the hub if you will for our assistive
technology features that are built in
the user level accessibility that I
spoke about so I can should we come over
here too I think it's a demo machine
three and give you a real quick tour of
the universal access press panel and
show you what we have built into Mac
os10 some of this you may already seen
in in Jaguar but we've actually made
some enhancements to it very much in the
case of user feedback what people wanted
to see so I come here and I see right
here of my universal access press panel
I click on it and it's going to bring up
essentially I said the hub of our user
level accessibility features we sort of
break it down into categories of
disabilities so for example we have
seeing hearing keyboard and mouse and if
we take a look at the first
which is seeing this is actually where
I'm a graphics guy at heart so this is
where I think the coolest feature is we
actually have full support for screen
enlargement inside the operating system
and this enlargement is done in a very
technically advanced way because the way
our windowing system works and the fact
that we use technology such as OpenGL to
accelerate our display so we're actually
able to do some pretty fantastic things
here for example I can zoom in and I
have full speed zoom it's very high
quality and I can zoom in and out very
fluidly and the other interesting point
is that one of the problems with
enlargers that people have been familiar
with in the past is in many cases these
only worked with windows and text
documents so if there's other rich media
that happened to be being used at the
time they really couldn't participate
because of the bottle necking and slow
down then having to do all this copy
pixel copying and replication would
burden the system we wanted to make sure
that we could architect the system that
really didn't suffer from that so we
actually created the system to where
it's able to do things like for example
and large even multimedia so it's a
QuickTime pop-up felt the loop playback
and now I'm able to freely zoom up and
experience multimedia without having to
have any conflicts with a screen and
larger and I can actually enlarge to
extremely large proportions and I now
have a 35 foot tall pointer over there
so it's really powerful feature now one
thing that we also got his feedback was
that people wanted more control over the
zoom so we actually made it pretty
flexible in here we're able to set your
zoom how how far you want to zoom in
zoom out if you wanted to show you where
it's going to zoom into a lot of options
like that we also have the ability to
turn on and off the smoothing so if I am
enlarge that I can turn on and off the
anti-aliasing because Mac os10 is fully
anti-aliased and in some ways if you
have issues with contrast
the anti-aliasing can you know make the
type sometimes difficult to read this
was sort of a first solution to be able
to address that problem but in Panther
we've actually gone one further actually
zoom out a bit we'll see we have a new
option and that is going to be the
enhanced contrast and this is a really
cool feature because we have a lot of
feedback that the anti-aliasing of soft
edges gays people with low vision
problems and second is the pinstriping
the pinstriping to be very difficult if
you have issues of perceiving contrast
so we actually sat down whatever you
know crack graphics engineers who
thought about it and came up with some
very funky math to actually be able to
do this effect where and let me actually
go one further in and see the
pinstriping I'm actually able to take it
and move my screen and just change the
the slider here and it's going to be
able to make the pinstriping drop out
and it's going to increase the contrast
of the windows additionally it's going
to harden up the edges of the type so
even though we have anti-aliasing the
type can get the we can harden up the
edges of the type so if we look at it
the type itself is going to become
bolder and bolder and this is again and
request it was put in specifically for
features by by our users and it's system
a system wide effect so and it's also
variable so you can dynamically to knit
we also have you know full support for a
flashing alert for people who have
hearing disabilities we have sticky keys
and slow keys which are also very
alternate input devices if you will for
people with dexterity issues you may not
be able to manipulate the keyboard is a
you know for example they can only hold
one key down at once like for example we
turn sticky keys on and this is also
pretty nice so they did a lot of fit and
finish on this where I can hit the shift
key individually and we'll see over here
so I get a shift key and it sort of
cached or held for me to hit another key
and then it would like for example I've
text that it open it actually typed that
letter so it means that the user does
not have to worry about hitting multiple
combinations of control key shift keys
to interact they can do single key
presses
actually cash and demonstrate you know
up and the up on the screen and a
semi-transparent way the keys that are
currently the modifier keys that are
currently cash they're going to affect
whenever a main keys hit you also have
mouse keys to enable the numeric keypad
to out like a like a mouse we also have
some other options so for example if i
cruise over to something that's it's
sort of interesting a lot of the
features that we've built in in the
universal access press pain we found a
lot of people you know like using them
regardless that they had a disability or
not for example the screen and large
turned out to be very larger turns out
to be very popular in last year's wwc
for people show code snippets we also
have some features that work the other
way so for example if i go into a
keyboard and mouse and see new option
called keyboard shortcuts and
essentially this enables users to define
hotkeys now anyone can use this but for
user with a disability who might need to
assign a hotkey one key press to do a
lot of different operations this is very
very powerful and then also we're going
to have a demonstration a little bit
later of another very important piece of
assistive technology that's built into
Mac OS 10 which is our speech which
keeps on getting better and better so
that was a quick tour of the universal
access press panel and let me go back to
slides so let's start to drill down on
the developer sighs I talked about the
user side the experience when the user
get the machine and has the ability to
essentially engage it even if they have
a disability but now let's sort of work
our way down into the infrastructure to
talk about what application developers
need to do and also talk about what
assistive technology developers need to
do we're gonna have to do a little bit
of storytelling here because we had to
really take a look at accessibility how
it's handled on other platforms and also
in Mac OS 9 and we built our
infrastructure for doing accessibility
and Mac OS 10 and one of the key things
that sort of created the scenarios we
built entirely new lat nua left Mac OS
10 is entirely from the ground up new
operating system and that actually had
for two challenges one is we had a lot
of infrastructure to build second
is for our developer community it was a
little bit of a curveball particularly
for the assistive technology community
because their landmarks in terms of how
they affected accessibility on the
platform totally changed so one of the
key things is that I think the biggest
thing that really affected them was the
protected memory model now obviously
everyone was overjoyed to get protected
memory on Mac OS 10 but it was
horrifying if you were assisted
technology developer because that meant
that your app assistive technology
application could not tell what another
application was doing because there was
a hard wall there was protected memory
in between them and also some chord
changes in the way we handle I o Mac
loston has a very abstract and modular
an object-oriented IO architecture and
another thing that's actually an
interesting challenge is our visual
environment is really complex the
ability to seamlessly composite 2d 3d
multimedia to do the dynamic enlargement
it's a very different world up in the
frame buffer than it is and it was on
other platforms and in the past but in
many cases some of these challenges
actually turn out to be benefit and
we'll talk about those letters would go
on but the key language we took this
opportunity by the fact that we built a
new operating system and we had to come
with innovative solutions for the
assistive technology community and
developers you need to make their
applications accessible we decided to
say hey we're in here we're building
infrastructure we might as well formally
include that infrastructure and make it
to where developers can rely on key
parts of the system to do things on
their behalf so the first thing we
wanted to look at is how it was done the
wrong way so we actually looked the way
it was done on Mac OS 9 and it wasn't
done the wrong way because you know
anyone was making a choice to do it the
wrong way it was really because it was
the way the accessibility and assistive
technology organically grew up on that
platform because Mac OS 9 architectural
II was a short stack of hardware with an
irregular layer of system software and
then the application sat on top of that
it was all running in one memory
partition so it was pretty easy for the
assistive technology vendor to basically
put themselves up on top and if they
needed to change the behavior of running
application or adapt the functionality
the operating system they could just
sort of have at it and go around and
change things so in many cases assistive
technology would essentially eject
is between the application system
software obviously assistive technology
developers not the only people do it we
had all these issues with extensions on
Mac OS 9 and that also was used the same
techniques but in many cases assistive
technology vendors had to be more
aggressive in some cases that even
change the behavior of an application
insert their code into a target
application to modify the behavior or
they would do things like you know get
in between the hardware and start
pulling events you know directly before
the system software could get them and
in many cases they would actually do
trickery with the way the hardware
presented itself to the Macintosh so you
actually had a scenario where assistive
technology devices and assistive
solutions were in a very fragile sort of
you know do-it-yourself environment and
we really realized that this was the
wrong way to do things we did not want
to replicate this behavior on Mac OS 10
so that meant that Apple had to get
involved in create this infrastructure
and so we realized in our new operating
system we have a common layer that's
called the application frameworks layer
in between all these different ways to
develop from oculus 10 each application
framework there's actually common
infrastructure that runs between a lot
of them so we said this is a nice choke
point if you will to start building a
fix accessibility into the operating
system to support assistive technology
and application developers we had to
sort of factor the problem because each
when the interesting about Mac OS 10 is
it's got you know for discreet ways to
develop for it from unix developer i can
use darwin and bsd on the java developer
have full java implementation if a
developer as core competency on the
macintosh from you know Mac OS 89
whatever I have those api's they've been
carved know about significant bodies
api's in carbon they've been brought
forward and they're available in the
platform and i also have cocoa which is
a fantastic way to do object oriented
development all these frameworks are
very different from one another and
making them all accessible was very
tricky so we actually originally started
with just carbon in cocoa so last year
we were focused on just making carbon
and cocoa applications accessible in the
platform but for Panther we've actually
extended that out to include Java as
well
so we're able to leverage the common
infrastructure to essentially have all
the applications you know sitting on top
of the application framework layer
they're using this poor infrastructure
and so the idea here is to take the
assistive technology solution which
before had to do some very interesting
things with the system and sort of make
it just another application to where it
was an application that just ran using
the same infrastructure and was able to
use that to do what it needed to do to
have access into other applications we
had to formalize some pathways with
zach's access to occur so that's what
the mac OS and accessibility API its api
api level support for accessibility it's
accessible to the standard c interface
so it's accessible from carbon and cocoa
it allows applications to discover and
control the UI of other running
processes ok because that when you look
at it we looked at the whole field of
accessibility that's one of the most
common thread that a piece of assistive
technology needs to be able to adapt the
functionality of an application that may
not even be aware it's being adapted so
we give pretty low level control for one
application to peer across the protected
memory and look into another application
and determine the you ike state and to
be able to expect that state it supports
some existing api's it was actually had
in a system for a while that were more
general purpose ap is for synthesizing
events keyboard mouse event and so it
can also be using injunction with those
to create some very powerful solutions
but more importantly this API said has
really become a common ground for Apple
and the assistive technology community
to work on accessibility on the platform
in fact a lot of the feedback we've
gotten over the past year in terms of
things you'd like to see in the way we
handle text and text representation has
been incorporated in the Panther and
then obviously we introduced the core
set of this and Mac OS 10 dot two and
we've done some enhancements
particularly in the area of text
handling in cancer so the key problem we
had to solve was how to let one
application know what's going on in
another application and the main problem
was that if we look at a running
application a piece of assistive
technology and the fact that Mac os10
supports radically different application
environment it have a pretty tall task
because if you think of a Coco
application
way it thanks for example it's windows
you know kind of control that it every
an object that every every application
is going to have a window it has a very
different language from away a carbon
application thanks of a window which is
very different from the way a java
application may think of a window and
obviously java's its own own centrally
virtual machine so it's even more
complicated in the Java case so what we
really didn't want to create a scenario
is where the assistive technology vendor
had to code up a solution for each
individual framework because I would
work for anybody that means either
sister technology vendor area three
times the amount of work or they would
just choose one and say I have a piece
of assistive technology that you only
work with a Coco app but what if a user
really wanted to have their carbon app
adopted so we needed to wait in and
solve this we need to take this sort of
Tower of Babel in terms of controls and
you I representation and try to fix this
so we've kind of come up with a lingua
franca called a UI element where what we
did is we essentially modified back in
Jaguar parts of the system so they could
express you I in a concept called a UI
element an abstraction and they can all
use this abstraction pass it over the
assistive technology application that
can then parse this abstraction and
figure out what's going on in any target
a poor gardless of framework so what
does UI element again it's a frame we're
going to pendant abstraction for user
and user interface and it will actually
have a little diagram of how a dialogue
works but you can think of essentially
an application is appearing of a
hierarchy of these UI element
abstractions because a UI element can
actually have attributes in terms of you
know the title of the window or the
value of a particular control or slider
for example a UIL McCann also have
children attribute so for example we
have a window and it contains within it
a child with Mike which might be a
button and obviously certain controls or
intercept actions user interface events
and so you can actually have UI element
can have an action as well such as a
prep action or an increment action it's
a whole suite of those so let's actually
sort of take a look because I think it's
better to sort of describe how this
really works in reality
and say show how the UI elements and the
accessibility API work to overcome the
problem of one application a piece of
assistive technology need to know what's
going on in the user interface of
another application so here we have a
theoretical piece of assistive
technology and we have another
application that's just you know my app
that's running obviously they're
separated by protected memory model
learn different processes so we need to
somehow to do to start a dialogue to
determine the user interface state so
the first thing I do is get the process
ID and I was a couple different api's to
do this but once I have the process ID
that's going to I'm gonna have to
basically asked for you I LM at the
initial UI element for this process ID
so i will call a x UI element create
application path in the process ID and
i'm going to get back a UI element
that's going to contain some information
the top level information on that
application and it's generally going to
have a role in this case it's an
application so the object i vas is going
to say hey I'm an application it's going
to have a title hey on my app I'm not
frontmost in this case because that's
the interesting thing previously a lot
of trickery had to be done by always
targeting events at the frontmost
application to control you I because
that's going to be the application that
receives you I but the interesting thing
here is you know having a dialogue with
UI elements is not dependent on the
application being always frontmost and
then also we're going to get information
back that this application has a bunch
of windows and obviously if we're trying
to parse what's going to application
we're going to need look at the windows
so the next thing we're going to do is
take the UI elements and essentially ask
the application say x UI element copy
attribute value and say I'm interested
in the windows of that application and
it's going to return obviously the
window and applications and in theory
this application has a window called my
document it tells us with the position
it also can tell us that that window has
some children and those are probably
controlled so I'm going to keep on going
down this path and continuing to ask
these questions through the
accessibility API to find out more and
more about this application to sort of
explore and parse DUI so i'll say ax UI
element copy attribute value again but
asked for the
children value by asking KX child
children attribute and that's going to
give me back the children the children
that are in that the previous window
that we're in and so I'm going to get
back today this window contains a button
and it's labeled print and it's enabled
because that we get state information
back from using the accessibility API
and here's its position on the screen
well obviously I might want to be trying
to make this application print so I know
that hey it's a print button it's a
button I can have to send some sort of
event to effect that operation so I'm
going to basically say hey and say aax
UI element copy actions is going to tell
me the actions that are available to
that particular element I'm interested
in and in this case I see that I have
the X button this is print it sports
true that it's a that's enabled and it
accepts event called a xpress so now i
know i can actually send a button press
event to that control and then the a the
actual application will receive that
event just as if the user actually did
it themselves so that's an example to
dialogue that you can have with
accessibility API s and there's also a
second part which is a notification
obviously it would not be a really good
idea for us to make assistive technology
developers continuously parts of running
you I of every application continuously
having to basically you know always
sample an application to determine that
the UI state has changed so once you've
actually gotten the you know the lay of
the land inside an application you might
only be interested in certain controls
and you can basically say hey there's a
control over here you can notify me if
something happens to that control so
obviously the key ones are like if
windows get created move or destroyed if
the application itself is brought to the
front or sent to the back even things
like if UI element that I've been
interested in the past gets destroyed it
goes away I obviously need to know that
if I'm trying to control that that that
you I or even parse or adapt that you I
and also value change which is really
useful for controls like sliders that
have
values that can change on the basis of
user interaction it's like what I did
like to do is invite Mike umber to the
stage and he's going to take us and that
is his actual job title right there and
and he's can actually show us the UI
element inspector UIL inspector is
actually sample code that you guys can
pull down and take a look at and it's
basically a way that you can use the
accessibility API s and Roman action
okay hello my name is mike amber and i'm
going to start off with a demo of UI
elements vector that's going to sort of
give a live illustration of what travis
has been talking about and then I have a
few slides only six so don't worry and
I'm going to cover some of the things
we've added for panther in the
accessibility API s and then finally I'm
going to end with another demo of a fun
little app so you ilm inspectors
available at sample code from DTS it has
been for a while so some of you may
already be familiar with it and I'm
going to start by launching it and what
this program does is every time the
mouse moves it checks what UI element
the cursor is over so as I move over
different things you can see that the
contents of this window change I'm going
to increase the font size so people can
read it a little better
and and now I'm going to launch TextEdit
we're going to do some demonstrations of
what it can do with text edit so here
I'm over an untitled window and text
edit and we can see that we get the the
list of attributes that the window
supports things like the role the role
description the title at the top of the
window is showing the position of the
window in the hierarchy so you can see
at the top of the hierarchy is the
application itself and then this window
is one of its children now one of the
sort of challenging things with the UI
of this program is since whenever I move
the mouse it changes what I'm looking at
if I want to operate on that thing
that's a problem so use a hotkey and you
can lock in on the particular UI element
that you're looking at and everything
turns to red to indicate that and then
this little interaction window appears
and I can't increase the font size in it
so I have to apologize but I'll describe
in detail what I'm doing so we can see
where we are now locked into the window
and one of the things you can do is use
this pop-up menu and get a list of all
the attributes in the window which we've
already seen above and we can see some
of them have w's next to them and that
means they're writable so i'm going to
pick the position attribute we can see
its position is given an XY coordinates
and I can change the y coordinate and
hit set value and the window move
another fun one is the minimized
attribute and it's a boolean it's now at
zero and I could change it to one set it
and the window disappears and then I'll
bring it back by setting it back to zero
and this illustrates the point that
sometimes people new to the api's forget
is that actions aren't the only way to
interact with UI element a large amount
of your interaction is going to be
accomplished through setting the values
of writable attributes
ok so now i'm going to bring up text
edit preferences pane
and I'm going to illustrate a couple
more features of UI element inspector
i'm going to focus in on this text that
labels this box and there's this little
checkbox in the interaction window that
lets you highlight the thing that you're
locked on to and so this is useful for
going around the app and just seeing
what you can hit so I'll demonstrate
here we've hit a another button here we
can hit a pop up and I'll go back to
that label and then also in the
interaction window is a pop-up named go
to and you can actually walk the
hierarchy so if you use your mouse to
get close to where you want to be and
then we can go up to the parent we see
the highlight changes to reflect that
that title is actually a child of that
box and then we can go up another level
in the box as a child of the window and
I'm going to show one more thing so you
might be asking the question can you ILM
inspector inspect itself and the answer
is yes it can and I'll in the demo by
having it close itself
so we can hear I'll perform an action
which is a xpress on the close button of
the window and we're done okay
[Applause]
alright so as we just showed we have
this model of attributes and values and
you can ask a UI element you can pass it
int name as an attribute you can get
back to value and this really suffice is
for most purposes and this is and this
was really all we had in in Jaguar but
we found that sometimes just passing the
name as an attribute isn't quite enough
so in Panther we added this concept of
parameterised attributes and for
instance you might want to know what
character the mouse is over and in that
case you need to pass in the position of
the mouse or you might want to know the
bounding rectangle for some text and in
that case you need to pass in the range
of characters of the texture interested
in so we added this concept of
parameterised attributes it's
conceptually similar to attributes but
now when you ask for the value you pass
in a name and you also pass in a
parameter and the type of the parameter
varies and I'll go over some examples of
that in the upcoming slide
so to accomplish this we added two new
AP is in Panther the first one is ax UI
element copy parameterised attribute
names you can see we kept brevity in
mind when naming these api and the
second one is ax UI element copy
parameterised attribute value so the
first one is very similar to an existing
API that returns you a list of all the
attributes this one returns you a
separate list of the parameterised
attributes and it's a separate list for
compatibility reasons we didn't want
existing code bumping in to parameterize
the attributes and trying to get their
value and not passing a parameter so
since they're separate list that can't
happen and these and the second call is
what you're going to use once you've got
the name of the parameter ID attribute
you're going to use you get to pass in
the UI element the attribute the
appropriate parameter and then you give
back your value and for now we only
support reading not writing of the
parameterised attribute values but if
the need arises we might consider
extending that in the future so no
presentation would be complete without a
giant table of very small text so I
apologize if you're having trouble
reading it but i'll i'll go over some of
the items in detail this is a list of
the parameterised attribute that we have
currently implemented in panther and
they all have to do with text so there
are a lot of requests from developers to
find out more detailed information about
text out with stylist in how it's
formatted on the screen and this is
where we found we really need we really
needed a parameter along with an
attribute name in order to answer these
requests so the first one is KX line for
index parameterised attribute so this
one is addressing the issue of the text
layout if you pass the index into a text
string then we're going to tell you what
line is on and then the second the the
second attribute up there does the
opposite if you give me a line number
I'm going to tell you the range of text
it's on that line then there's a string
for range which will given a range of
text allow you to just get that much of
it before you could ask for a text
object text in its entirety this lets
you get it a chunk at a time then
there's one that'll tell you if you
specify character position what range of
texture over and you have to keep in
mind that you know the mouse might
appear to be over a single character but
in fact because of surrogate pairs or
multi-byte glyph you might be over
several characters so we return the
whole range then there's range for index
and this again you have to do with multi
by characters if you specify an index in
the in the text will give you the full
range of characters that comprise that
glyph so there's call similar to this
like range of compose characters in the
kit and this is sort of the
accessibility equivalent to that then we
have some that have to do with layout
given a there's the KX range for KX
bounds for range parameterised attribute
so you give a range of text and we're
going to give you back a bounding
rectangle and then there's the KX RTF
for range parameterised attribute so
again you can give us a range of text
and we're going to hand you back the RTF
representation of the text and from
there you can extract the font the size
the color just about everything you
could want to know and then finally
there's this KX style range for index
parameterised attribute I'm sure you
guys are going to enjoy typing these
names in just as much as I am speaking
them right now and this last one lets
you go through your text and style run
so if you're interested in when things
change you can you can use this and find
out you know the extent of the current
style run and then move on to the next
one
there's a couple of other attributes we
added for text accessibility in Panther
and these aren't parameterised
attributes the first one is just to tell
you how many characters are in a in a
text object and the second one is to
tell you what range those characters are
currently visible and the next two are a
little more esoteric these have to do
with text that flows if you have for
instance a two-column document that
might be two separate texts UI elements
and the text flows between them so the
first one the shared text UI element
attribute tells you what's your portion
of the text that you're displaying
because in general most of the
attributes if you use ask for the text
they do to treat this treat you as if
you have all the text because that's
most most the time what you want and
it's most convenient but if you really
need to get to find out what exactly
only this UI element has you can use
this attribute to find it out and then
the last attribute ax shared character
range attribute actually I got these
backwards the second one is the one that
will tell you your share the first one
is going to return you an array of other
UI elements where your text is shared
with so you know it would just return
you the other columnist your text to
split between two columns
now here's a few other untaxed related
attributes who's added for Panther first
we had a lot of requests for people
wanting to be able to be able to
identify the default button so we have a
we have a parameter on windows that will
tell you the default button and we also
have one that will identify the cancel
button we also added a couple of
notifications so when drawers appear or
when sheets slide down you can get a
notification when they're destroyed you
get the normal UI element destroyed
notification so we didn't have to add
anything there we also added some new
sub rolls for windows again this is a we
have a lot of requests from developers
wanting to be able to distinguish a
floating window possibly ignore it from
a dialog from a regular window and so we
added a variety of sub rolls that will
identify dialogues floating windows and
and then there's an all-purpose unknown
if someone puts up a custom window we
can't tell what it is and so I'm going
to finish up with a demo of a little
athlete call monkey see monkey do
so I'm going to start by launching text
edit and this is a typical document you
might work with on any day with a
variety of tastefully formatted text
different styles and different colors
and next I'm going to launch I'm going
to move this out of the way and then I'm
going to launch monkey see monkey do and
this app its name monkey see monkey do
because basically that's what it does it
uses only the accessibility API and it
tries to replicate the user interface of
another app so I'm going to from the
list it presents me i'm going to pick
text edit and it's computing its own you
know it's creating its own window that's
supposed to look as close as it can get
to the original app and again all
excusing is the accessibility API to get
the information it needed and it's
actually completed a while ago I'll just
have to move it off so again this
illustrates that all the rich text
information is available to you through
the accessibility API so I'm going to
demonstrate some some other features of
monkey see monkey do first I'll go to
TextEdit I'm going to close this window
and you'll notice monkey-see monkey-do
removed its copy that was a example of
getting a notification of the UI element
destroyed and now I'm going to bring up
preferences now this is actually a
little tricky because when text etics
deactivated its preferences panel
disappear so that adds a little bit of a
wrinkle to this but monkey see monkey do
is already done it's best to replicate
the Preferences panel and you might
notice you might have noticed the change
if you were paying attention this one
doesn't quite look right and that's
because there's no way through the
accessibility API to discover whether
you're using regular size controls tiny
controls or you know small control so it
take you could take a guess based on the
size and here it's not guessing quite
right so I'm going to pull aside monkey
see monkey do version of the preferences
and again text that it's not in the
foreground so that's
why you don't see the other window but
as soon as I bring it back you'll see it
and using notifications there's some
more connections we can show for
instance we can pick something from the
pop-up menu of course I picked one it
was concealed so you can barely see it
at the below the upper window i'll pick
i'll go back to automatic and you can
see a change next I'll click a radio
button and you can see it changes
between the two and i can do illustrate
some typing we can change the the window
with to a hundred and when i hit enter
it will propagate down so so again this
this application is just sort of a
illustration of what's possible to do
with the accessibility API ok back to
trap
let's go back to slide so now when I
talk about is if you're an application
developer and touch briefly on how do I
engage the accessibility API all the
sort of infrastructure we've been
talking about the first half the
presentation and there's go through it
really quickly because we did quite a
bit of this last year in terms of going
in great detail so there should be safe
sessions on ADC TV and the dvd-rom on
carbon accessibility there's also one in
cocoa text I mean that cocoa controls
and also the accessibility over D
session have a lot of this content in
greater detail but the key thing is
obviously there's multiple types of
applications that work on different
frameworks on Mac OS 10 and the
accessibility API can work with carbon
applications but there are some caveats
essentially we've been telling
developers and carbon applications to
use standard controls for a long time
for other reasons besides accessibility
if you were one of those developers who
took her advice you're in pretty good
shape because we were able to make a lot
of standard controls accessible in
Jaguar and Panther however there's a
long sordid past of carbon developers
doing their own controls if you do this
in your application you will have to
actually do some work to support the
accessibility API in your application
the other thing is important as your app
should support carbon events as well and
that also is one of the ways
accessibility API work the other thing
that is a sort of an area of difficulty
and it's something that we don't have
total control over as much as we'd like
to is that in carbon there's also a
history of people doing their own text
engines or you know you name it
license.txt engines they roll their own
there's several different ones available
in the osm LTE there's text at it and
it's too big of a problem for us to
solve on our own so carbon developers
want to use the parameterised attributes
are going to have to add some support
into their text engine for supporting
these api's something we'd love to fix
if we could but it's too big a problem
for just apple to tackle we're gonna
have more information published on this
topic on techno 2092
and that is going to be forthcoming and
we'll have lots of detail on what carbon
developers need to do to properly
support the accessibility API good news
on the other frameworks Coco it's a
great story there the cocoa team
actually modified app kit to support the
accessibility API last year and they
continued obviously increase that
support and because the way Coco works
essentially you're always inheriting
from parent classes applications that
may not have been a war aware of
accessibility and say for example Mac OS
10 dot one when those applications are
running 10 dot 2 or in Panther those
applications will be accessible there's
still the possibility for Coco apps tap
custom controls there are ways to make
those custom controls accessible Java
applications an interesting story here
is you don't need a Java bridge
accessibility bridge like you do another
platform well the java accessibility
bridges bridges out the accessibility
from the platform native side where the
assistive technology has to run over
through to the sandbox we don't need
that on Mac os10 it was you know Java
and Panther you just basically we've
done the work for you and awt and swing
applications that use standard controls
and those environments are going to be
accessible as well let's talk about
brief it about bringing your system
technology to Mac OS 10 this is a more
for the assistive technology developers
who are looking to engage our platform
there's a lot of ways you can go wrong
because it's sort of like a restaurant
with a huge menu and we want you to
choose certain poisons and putting a
certain items we have like the chef's
special to do the right technologies and
so I'm going to give you some tips right
here one of the things that we have been
dealing with assistive technology
community for three years for apple and
it's a continual ported confusion as to
where you should start how should you
begin coding your sister technology
application and it sort of become clear
to us and looking at all the options
repeatedly with a lot of developers that
if you're in a situation we're trying to
bring a new assistive technology
application up on that goes 10 you want
to do it in cocoa it's just movies
faster more efficient for you get a lot
of platform benefits by using cocoa and
you also get a lot of stability because
the way our
texture works where cocoa can actually
innovate and change and you'll inherit
those advantages and the rapid
application development aspect of cocoa
has a lot of assistive technology
developers a small shop where and this
is going to be tremendous productivity
gain for you guys so we hardly write and
you really recommend you take a look at
Coco that you're making assistive
technology device such as an alternate
input device ahead mouth you know I
tracker a switch I think that one of the
problems that we see a lot in this
community's you guys use a lot of legacy
i/o cereal you know ps2 mice the reality
is you know Apple sort of you know
really helped catalyze USB when we
introduced the imac a long time ago yes
these ready so you know if you have a
ps2 assistive technology device you're
going to want to really think how do I
move it to modern serial i/o and most of
your case it's going to be USB but
there's also interesting possibility
with things like firewire where an eye
tracker instead of being this large
multi-thousand dollar box and
specialized custom image processor done
in Hardware could be a relatively
inexpensive firewire camera with you
know using the you know the real-time
capabilities and the high-speed
throughput a firewire able to actually
pump it to data into software that's
running on Mac OS 10 which is using
maybe our platform tune DSP and math
libraries to do the actual calculations
to track the eye so then you can take by
using the proper i/o and clever software
you can essentially factor your problem
from a very expensive hardware
implementation down to potentially very
inexpensive and affordable hard hardware
the other interesting thing is when
doing any of the new serial i/o please
follow standards it doesn't do any good
to do the i/o and then not follow the
standards and this is agent Lee happens
where people deliver an excellent piece
of assistive technology but they'll make
it work like a keyboard so when the user
hits the switch input device it just
says letter a letter a letter a and then
it becomes a complicated software
problem to determine okay where'd that
letter A come from it comes from the
assistive technology device versus did
it come from the keyboard and software
layers of software have to be put in the
foot and putting the place to the code
that where if you follow for example USB
HID spec which is all sources of input
devices you could say this is a button
and that's much clearer than having a
switch emulate a piece of a keyboard
there's also interesting for a wireless
opportunity such Bluetooth and also
force feedback we support for to be back
in Mac OS 10 and i think that's to be an
interesting to mention to provide
feedback back to the user when using an
alternate input device and we have some
speech technology that's available in
the operating system a really good
speech technology we've been working on
it for a number a number of years and
that's available for assistive
technology developers to do text to
speech it's also available for to do
speech recognition and I'd like to
invite Kim Silverman up this stage and
we're going to quickly do a
configuration adjustment and this was
sort of an impromptu ad today and do a
quick demonstration of some of the
excellent speech capabilities built in
tobacco sent by Trevor wall shows is
setting up there are a whole lot of
speech features built into the operating
system we don't have time to show you
even a small proportion of them we're
just going to step through particularly
what's relevant to the accessibility API
thank you so there are speech synthesis
and speech recognition capabilities that
use that let's talk about the speech
synthesis first excuse my back I'll be
looking at the screen here we're going
to go to the system preferences and the
speech preferences we're in the spoken
user interface tab I can turn these
things on the first thing I'll turn on
is selected text when the key is pressed
it's going to ask me for hotkey i'll use
f8 and that means in any application if
i have some text selected I can speak it
just by pressing f8 so we've got some
news which I just brought up before the
session today I'll press f8 and we ought
to hear it security talks between and
clearly in Palestinian leaders are
making good progress so there's a
selective text when the key is pressed
and this is one of the disability
features that like many become popular
amongst people who don't have
disabilities as well it means if your
type
a document for example and you want to
hear how it sounds you want to prove it
you just select it and hit your hotkey
and it's read back that is useful for
detecting spelling mistakes because
spell checkers can only detect Tytos
that don't make words if you ask any
cognitive psychologist you'll point you
to a lot of data showing that most of
the typos that we make make other real
words and therefore spell checkers can't
catch them then we have this item here
text under the mouth this again uses the
accessibility API and when I turn it on
you'll hear it may speak already the
name of that checkbox thanks don't even
know and as I move this around and speak
other controls speech recognition I'm
not doing anything to launch this it's
happening spontaneously it's all boys so
can lose your interface ok we'll turn
that guy off otherwise your drivers
crazy now let's go through speech
recognition if ever you try out the
speech recognition the first thing you
should do is give it a chance to
normalize to the way your voice sounds
in the acoustic environment in which
you're using it we have built-in
automatic tracking and adapting to the
acoustic environments but you can speed
it up I'll show you how and I'm going to
do it now because I'm getting echoes
from the back of the wall which are
quite a long delay and it's going to
learn about that under we built the
speech recognition to work with the
microphone that's built into most of our
machines I mac ibook powerbook Emacs
tower machines don't have a built-in
microphone you can get a number of
third-party USD head mounted microphones
I'm using this one which was produced by
our buddies over at vxi those guys
really know what they're doing there are
other brands you can use as well so you
go to a speech recognition preferences
to the listening tab and click on volume
now as I'm speaking I can adjust the
volume so that my speech Peaks just go
into the red and then my task is to read
down these commands and it's listening
to me now and as it hears me say one of
these and recognizes them it will flash
what time is it
quit this application open a document
show me what to say make this page
speakable move page down hide this
application switch to find them so
ostensibly we just let you set the
volume on the larger phone but what you
don't know is we just set nearly a
thousand parameters they're sending our
spectral properties of this microphone
and me and this acoustic environment now
we're ready to use the accessibility
features I go to spoken user interface
what's no I don't speech recognition
command and here there are a number of
sets of commands that can be spoken from
the speakable items application oh I
should turn it on first we go to unearth
ternal on here this brings up speakable
item and there's our old familiar
command window and zooming under that
for a moment well there it is I like
that and I don't normally do this but I
will turn on speak confirmation which
means it will speak back to me anything
that our hears me say what time is it
when a giant head its 1136 ok this is
speaker independent I've not trained
this to my voice other than the
normalization that you just saw me do
and that all of you will do when you try
this in your hotel room tonight won't
you yes all right down here turned off
by default is front window command and
menu bar commend both of these users
accessibility API that will turn this
guy on and now I can speak through any
front window commands in any
applications using real control I'm
showing you this because you're going to
keep using real Apple controls and you
default voice in silence boys spoken
user interface spoken who's the
interface we have not programmed
anything specimens of these preferences
so we can go outside of our preferences
that will still work show all cheer all
let's go and do another one I'll just
pick one at random let's do a doc and
magnification magnifications so they
would even see that the speech was
developed as a client for the
accessibility API Travis mentioned that
it gives you and our common ground we
use speech as a client so that we could
make sure the accessibility API worked
before we gave it to you it also means
that if you use standard controlled you
get a lot of accessibility in your
application just for free and three
accessibility API you can add even more
I think that's enough for that thank you
thank you go back a slide please so
that's pretty cool stuff we got great
speech technology Mac OS 10 and it
builds on top of the accessibility API
to do some very interesting things so
let me talk about the last point I got
to kind of hustle through it Makka soon
has a unique visual pipeline we have a
compositing windowing system and also
windowing system it's very aware of the
idea of time to layer things on top of
one another and this is sort of unique
in the industry yeah I don't want to go
to a technical description of how the
actual pixels flow to the you know to
the frame buffer but the key thing is
that there's one process it always knows
what's going on the frame buffer and
that process as lowest level primitives
as a concept of a window and this means
assistive technology can do some very
interesting things and also solve some
problems for assistive technology one
thing is that the windowing system does
support the ordering so it's very
possible for assistive technology
applications to that need to be always
on top to have to do that it's just
basically setting in the proper layer
and the z-order stack and then
applications can't put their content on
top of and piece of assistive technology
content that needs to be always on the
screen and then also the compositing
gives all sorts of opportunities for the
visual presentation of assistive
technology because in many ways there's
always a fight between Skrillex green
real estate between the assistive
technology and the real estate that the
application that's being adapted assumes
is that it's using so I actually want to
show you a really quick demo of a piece
of sample code that actually sort of
shows an interesting way to maybe solve
this problem so let me go to
yes that's right there my machine
it's going to bring an application that
engineer wrote that's showing into
something called a overlay window and
this is a just a simple application that
I had one of our DTS engineers do works
with Coco come up with it's going to
essentially demonstrate a prototype if
you will of what's called a scanning
application assistive technology which
is soft it usually works in conjunction
with a switch that the user will hit the
switch and basically a pallet will pop
up and then a selection will start
iterating over that pallet they hit the
switch again and then they can select an
item of maybe actions that are encoded
in that pallet and the interesting thing
is you know this is showing the idea of
assistive technology being able to move
out of the way of applications because
it's able to use the macula strands
ability to composite and do a
transparency and blending to where this
could easily be made as the user needs
that it could fade forward and begin its
scan and then it could fade away when
the user needs to maybe view a web page
or something underneath it and also one
last point is this little sample is
about 200 lines of coke 0 code i believe
i remember my numbers correctly and the
interesting thing is that you know one
of the things that typically people need
to do in scanning applications and
support a large number of tiles or
different little iconic representations
of the actions and the interesting about
the power of cocoa and mac OS 10 is the
fact that i'm able to put a variety of
different file formats into this my
little scanning application here and the
system of the stall displays them
without me having to know anything as a
developer how to decode them this will
even work for PDF files so i could use a
i was a standard software manufacturer
that did an assistive technology scanner
i could then use an ilitch tool like
illustrator a free hand to create my
high-quality artwork which would then be
dynamically scaled down and have the
best possible vector fit for that piece
of assistive technology this is a piece
of sample code available off on the dts
website so let's go back to slide
when last point I want to make is that
there's an issue that is sort of
happening right now and between
assistive technology and the propensity
for applications that begin to grab the
full screen to take over the display
it's a functionality we offer in Mac OS
10 a lot of games use it and it creates
a problem that let's say for example
someone wanted to use an alternate input
device which required a piece of
assistive technology software running in
the computer to be able to display its
content on top of maybe the game that's
running that would be difficult to do
the game says the screen is mine so
here's sort of a recipe or
recommendation if you're a game
developer or you have an application
that does grab the full screen now you
can use the z-order capability of mac
lessons compositing window model to
essentially put your window at a
particular level allows us the
technology to work it's pretty basic and
so taking over the screen you just
simply hide the menu bar you then hide
the dock then what you do is you bring
up a full sky full size window without
any sort of borders or controls around
it and you put it in particular level
kind of screen size of a level plus 1
and this there's another layer that's
above this essentially allows assistive
technology to appear above your your
application this can be like 2d 3d it
can be multimedia so in theory you could
have a piece of assistive technology
with a semi-transparent menu sitting on
top of maybe a game a 3d game if it was
designed that way and the window layers
are defined in CG window level H which
is in the courts 2d headers so in
summary we basically covered a lot of
ground here we've talked a little about
accessibility from the aspect of why you
should do it we've talked about what
technology Apple's built into the
operating system to accommodate
accessibility both from a user and
developer perspective we've shown you
some new api's that we put in Panther to
really benefit assistive technology
developers who need to get at text
attributes and then additionally I've
made some recommendations for both
application developers an assistive
technology developers how they can
better leverage the technology portfolio
on Mac OS 10