WWDC2003 Session 428

Transcript

Kind: captions
Language: en
good afternoon and welcome to session
for 28 understanding the new ink adi of
mac OS 10 we are thrilled today to
introduce a new set of a TI to mac OS
that they will allow you to provide
additional support for handwriting
recognition in your application as you
know with jaguar apple introduced
inkwell the end customer solution for
handwriting recognition in quill
provides system-wide support for cocoa
and carbon application without requiring
any modification to the object and what
I'd like to do first today is give you a
brief overview of the functionality the
ink will support in the application so
what I have here is a Wacom tablet I
welcome tablet and the first and i have
downloaded the Welcome driver from the
web and the first thing I'm going to do
ready this is it's got to the system
preference and no that is a ink panel
dare say I'm going to launch that panel
and I can enter user control this is
going to allow me to turn black nation
on and off the condition is already on
but if you wear off I turn it on and
that's all basically all I need to do to
have system-wide support for handwriting
recognition or Jewish there's small
stuff that you can do in the ink
reference panel basically tweak the
settings for recognition look at the
gestures that we support our words to
the user dictionary but not now going to
go into those right now so let me quit
thanks reference panel and now I'm ready
to start inputting text in my cocoa
application I have texted it up here and
i'm going to start writing anyway I like
on the screen and you'll see that the
strokes and the writing guidelines will
show up the strokes get cheap as a unit
to the recognizer recognized
expect same Chicago as a carbon event
Coco will extract the information and
enter into the document you can continue
to write on this tablet strokes we Bo
collected recognize and boom they go to
my application as easy as that tax ID
have not been modified to support
recognition however equal allows you to
do much more than text that entered in
your test is all walla is always also
allows you to do editing so for instance
if I hold down the stylus the 10 will go
into Mouse mode and now I have a
selection I can ask upon the selection
and a can of Ages and I can erase it by
just doing a simple advice gesture on
the screen and boom that word is God but
I really didn't want to erase that word
I want it back in my document and I'm
going to do an undue gestures a sheikh
abdul gesture but there it is i got my
word bathrooms on the dog so now let me
switch to a carbon application and i'm
going to show you that exactly the same
paradigm works there i have i cube and
say that i bought a new CD and i want to
add it to my collection now i want to
give it a name they're going to pick up
my stylist and I'm going to write the
name for my new CD there you go I'm done
so without any modification I can enter
test edit and manipulate the UI however
this is that's pretty powerful but there
is much more than really applications
could do if they had access to the
information that the recognition system
provide and to give you just a glimpse
of that i'm going to show you what they
in fact that the inside is the real
input window that we ship on java that
is accessible from the in windows
and that we like to call it the first
inca web application so as you see you
can go to the inside and I right there
on the window again the text the ink is
recognized and will show up on the
document however if I can go now to that
same text and because this is an Inca
well application I can double click and
hold and I can look at all of the
alternatives that the recognizance
assistant has given us on top of that i
can choose one of them if I like or and
I can look at this trove that they use
it that I've written so that I get
feedback on what that word was meant so
i can select that and now the case for
the words changed okay let me use the
undo works right it okay so i can
continue to write on the inside this
exactly the same way that I'd wrote on
text edit well the phone rang and I
stopped in the middle of the word where
I can pick it up right where I left it
let the other part of the condition and
then do a gesture to merge those two
pieces of those two words as I wrote so
with a simple gesture that looks very
much like an E I can merge those two
string and now notice that because this
is an Inca web application I can erase a
word just by scribbling on top of that
without having to go through the process
of selecting and then erasing so there
is much more power that you can add with
your application by using the stylus
directly on the object and developers
have recognized that and since we
introduced in work we have had the
experience of requests to be able to for
developers to tap into the ink services
that we provide
the requests of range from questions
like how can I add support for devices
older than the Wacom tablet to request
that we're developers are asking about
separating the process of gathering the
digital ink from the process of
recognizing they basically doing the
federal Commission if you are familiar
with the term two questions about
developers who want to have more control
on where and when the ink is drawn so we
have taken for thunder we have taken
those frequent and combined it with our
vision on how handwriting support should
work on Mac OS 10 to provide you with a
set of very simple yet powerful a tree
eyes will allow you to provide
additional support for the handwriting
they will allow you to develop to ship
to your customers their solutions that
they are looking for so today what I
want to do for the rest of the
presentation is introduce you to some of
the concepts that we are going to use
throughout the presentation some very
calm basic concepts that we use in a in
da TI and then we're going to talk
briefly about the technologies that you
use behind the scenes to provide the
recognition support and finally we're
going to go into the need of the
presentation and give you a brief
overview of the API with support and how
you can use them to develop to develop
solutions for your customers one
question that we often get is what kind
of device do I need in order to get it
in well or in services to work and for
job adequately answer was you need a
Wacom tablet and welcome driver for
Paris we're going to continue of course
to support the Welcome devices but we
are going to add support for anything
for devices that allow you to provide x
y and pressure information either in a
real-time fashion like tablets do or an
offline way like
really available USB and Bluetooth
stance that you have to these days on
the market now let's go through the
concept they're basically size comes a
very simple concept we are going to use
throughout the presentation and in their
documentation the first one is an
interest and you are already familiar
with that think of a new trades as a
group of strokes that stays on the
screen before then ships you the
recognized they have a little bit of no
answers to that concert you can't leave
them the documentation but then
basically is what you need to think
about is food for this presentation now
as I was writing on the tablet and I
lifted the pan the increase was
terminated didn't recognize it and
then to the application there are other
ways that I can terminate a phrase and
basically I can if I hold a pen for a
while and nothing and I don't do
anything else a timeout is going to
happen and the edge place is going to
terminate the other way that in phase is
terminate is because I if I write my
stroke two apart on the X direction and
on the eyes y direction that's also way
the day in phrase is terminated and the
reason why I'm telling you this what is
it important concept to you is because
later on when we go to the developers
nails and the api's we are going to talk
about ways for you to take control of
its face termination in case if your
application needs to do so the next
concept is in tax rap and if if the
strokes you'd wrote are recognized by
dating services as text essentially what
you are going to get getting your
application is an in-text read an
in-text rest is an opaque object that
contains information about this digital
ink that you wrote the location of that
ink and the information from the
recognize that set of strings so that
recognizes provided
the alternate for the ink and the API is
provide a way for you to access most of
the information that is available on the
integration and of course the alternance
menu if you want to present you to use
it the results of the recognition so
that they can browse the information
we're going to provide a simple API for
you to construct that menu and show it
to the user and then a and the menu will
contain again the alternate splicing and
the final concept that I want to get a
go of it is gestures you saw me use all
of these gestures in the demo and so we
basically have pretty tired to gestures
the first time the needs position myself
on the right hand on the right hand of
the screen if they undo gesture and we
call those kind of gesture non-targeted
gestures and what their those kind of
gesture what it really means is no
matter where you draw draw them on the
screen they're going to perform the same
action the second type of gestures which
I have here a hundred top left of the
screen and what we call optionally
targeted gesture and an example of that
is the arrays gesture and you saw me use
it in two different context in text
studies which is at this point not an
Inca whereas I had to do a selection in
order to erase in the eighth pad in
which have been the ninka where
application you can just erase on top of
the text so that is that there is a
difference for those Gentry and we have
additional gestures that are optionally
targeted
and finally the third class of gestures
is the joint gesture those are gesture
that I've only supported in inca 'what
up because we need to know where on the
screen digestion occurs so there is no
way for the in services to differentiate
between a v4 the joint gesture unless
the application give us the information
or word on the context of the
application the juice will happen so
again there are three classes objection
and the reason why we're telling you
this is because you again can choose to
support those gestures in your
application to allow the user to do
direct manipulation to interact with the
stylus directly with the object without
requiring them to go through the
metaphor that is where that is needed
for the keyboard but you have to select
and then act upon the selection you can
just go and use the stylus directly in
the other and so the non-target objector
basically usual need to add the support
the system can handle it for the
optionally targeted gesture you may want
to add the support if you want that is
direction to happen because you need to
add the information of the location and
for the always targeted gestures the
support has to come from the application
and then concludes kind of the basic
concept that you need to do know today
now let me move on to the ink component
and we have basically force components
for the inch services on the right hand
side of the screen that blue rectangle
shows the application context and in the
application context two of the
components live there the ink input
method is the part of the ink services
that provides data collection and
drawing of ink so as you right pink
input method is going to gather the data
android the other components on their
context of the application is a
framework that's what the implementation
of the API
on the left-hand corner of the screen
the box shows you the user preferences
which I show you in the demo and that
contains it with the ending contains the
press thing that allows users to set
their recognition mode look at the
gesture turn gestures on and off add
words in the dictionary and finally on
the top corner of the screen we have the
inserts that's where the call of the
recognition lives whether it's a
segmentation the recognition process the
language model is instantiated and in
addition the ink we know but the
important for you today is the fact that
that's where the core of the recognition
sleep and with that it is my pleasure to
introduce Larry Jaeger Larry Larry Larry
to distinguish engineer at apple the
world for us for several years in
handwriting recognition and I'm very
pleased to have human stage today Thank
You Julie okay I'm going to give you a
quick run through some of the
technologies underlying what julie has
just demonstrated for you and even dive
down a little bit as recognition
technology though that doesn't directly
affect your apps it's sort of what
provides all these services the
recognizer can be boiled down to
something very simple basically these
three boxes pretty much define the last
ten years of my life the it starts off
with XY points and pen lists it just
exactly what you would imagine are
coming from the graphics tablet and we
take those and in strokes what you'd
normally think of a strokes with pen and
paper and we look at combinations of
those strokes to see what might possibly
be characters and that's what we call
character segmentation so out of this
tentative segmentation block comes these
character segmentation hypotheses and
some of them are going to be writing
some of them are going to be wrong and
we can't possibly predict upfront which
the the next block with it is weak
there's a neural network classifier it
takes these combinations of strokes that
might be characters
looks at them in a certain way and tries
to decide whether they are or not and
which characters they are so what idiots
are these character class hypotheses
it's trying to say if it's ABCDEF or
whatever that sequence of probability
vectors are fed into a search engine and
the search engine sits there with a
language model that's the context and
tries to just find the best combination
of all those possible character
combinations and vectors of
probabilities of characters and look for
the maximum likelihood pass through this
graph and end up with finally word
probabilities what the person wrote so
started to see what that looks like
suppose a person wrote this on the on
the computer you you know hopefully
maybe fifty percent of the people in the
audience thought it was clogged and
titty percent thought it was dog
although dogs such common word may be a
little skewed that way well what we do
is take those strokes and take the what
the first stroke that kind of looks like
a sea and feed it into the neural
network classifier it emits this vector
of probabilities and let's say it's
pretty good you know point ninety
percent certain that it's a see okay
that's great next we'll probably try the
first two strokes together and let's say
that well you know there's a written
kind of far apart that's awfully big gap
in there let's say the neural net kind
of picks up on that and says well
seventy percent probability that's a
that's a D and you know a few other
things scattered around when the next
when that second stroke is then tried by
itself the only legal transition is from
the sea to the L you can't have the C
and the D because they use the same
strokes so let's say that the neural
Nets pretty good and it's got about an
eighty percent probability that that's
an L well I deliberately jury-rigged the
numbers so that that looks like point
nine times point eight is point seven
two so seventy-two percent chance that
the theory so far is is CL 74 chances
that's right well and I assigned seventy
percent to
the D so so far our little engine thinks
that it's more likely to be CL than it
is d okay well when the Oh stroke comes
along let's say it's pretty good there's
an interesting thing going on here that
it doesn't almost doesn't matter what
that probability is at this point the
CLO is going to wind over the do because
the CL 1 over the D and that's going to
be true when we bring up the G you
multiply those probabilities together
and indeed ever so slightly the way I've
jury-rigged the numbers it would
probably pick clog but in fact as I
mentioned earlier it's guided by context
we have a language model that's fairly
complex is guiding all this and you can
write outside of the dictionaries you
don't have to write in the dictionaries
it's a loosely applied language model
but you but it does guide the search and
so if you were searching with the pet
philosophy newsgroup where the only
words were cat dog and dogma that would
guide this search and so the only viable
path through that graph of the language
model would be dog and so really dog
would win and there's there's tons more
information actually published available
on the web that will give you a URL for
later okay so given that the recognition
engine is sitting under there ready to
provide these answers let's take a look
at how the data flows through the system
before it gets to your application and
again you don't have to worry about
every little detail in here but
hopefully this will give you an idea
enough about how things are working that
it'll make sense when you have to when
you do have to make calls to make things
happen just the way you want you'll
understand why those are the right
things to do so initially data comes in
through the graphics tablet that tablet
data is fed into really this is kind of
two layers in this next block but I just
lump them together to fit it on the
slide gracefully there's um there's a
tablet driver living at the i/o kit io
hid layer and what it's really emitting
is Mouse events with tablet data
them so they really are true honest to
god first first-rate citizen mouse
events but they also have this block of
data that has to do with pressure and
orientation of the pen and all the
special things that pins can do those
Mouse events left alligator are fed
through the core graphics layer of
course and the core graphics layer does
something very special for us you may
have noticed while Julia was demoing ink
text and some of the other things she
did not have to write in any of ink
texts windows she could write off
apparently on the desktop we call that
our write anywhere model and it's
something the user can turn on or off
but we turn it on by default if you turn
ink on at all because it's beamed we
think it's an easy way to interact with
the system when you're using a pen
basically you right where you want to
the recognition results flow to the
insertion point so core graphics is
sitting there and if this mat this mouse
event coming through the pipeline has
tablet data in it and recognition is
enabled and we told it that in the
background then it will route those
events to the frontmost app that's going
to be important later because you might
very well in the process of handling ink
related events get points that lie
outside your windows and you kind of
need to know that okay so there's one
other thing that core graphics does for
us if we're going to allow you to right
outside of windows and write all over
the screen well there are some places
where you're going to put the pent the
user is going to put the pen down that
they really want it to just behave like
a mouse when you put it down in the drag
region of a window you don't want to
have it start writing while you're
trying to drag the window if you put it
down on the scroll bar you don't want to
start writing while you're trying to
scroll the window and the doc is the
same way so we've invented this idea of
instant mousing regions and so there are
certain special places that if the pen
lands in one of these regions it will
instantly behave as a mouse instead of
inking and so core graphics has its own
idea of a few very special places like
the menu bar and the dock that are known
and the draggable regions of windows
that are server-side drag that that it
knows to go ahead and treat instantly
emmaus then we take so it's finally
still emits just a mouse event with
tablet alien and this is now we're
getting into your application context
area but first our input method our tech
services manager TSM input method gets a
chance at the data first we have our own
idea about instant mousing and take care
of some things for you like most
controls and user interface probably
want to be instant mousers what we know
about that so all your standard carbon
and cocoa controls are automatically
instant mousers except for the ones that
where we think you ought to be able to
write there's going to be a way for you
to manage your own custom controls in a
minute as Julia said we gathered that
ink data up into strokes we had those
strokes to the idea of this current
phrase and eventually get around to
terminating the phrase as Julia said we
send that off to the recognizer we get
the results back and in Jaguar we post
first we post a Unicode for key event so
that apps that are already TSM savvy and
looking for that can get the whole block
of text at once but if you don't handle
that then by gosh we turn into rocky
downs so finally your application will
get the data the text data corresponding
to what the user wrote for Panther we
still have Unicode for key and key down
at the end of the chain but we've just
introduced for events that lets you
manage what happens with ink in your
application there's an instant Mouser
event that lets you decide if some
special region on your document layout
needs to be an instant now sir hopefully
there won't be any of those but this
lets you manage it just in case we send
an ink point event for each of the
points that comes in that we think
should be inking now you know I've
talked about instant mounting and write
anywhere and all that there's always
this decision going on when a person is
using a pen whether it is supposed to be
the mousing or inking and there's a fair
bit of logic we cue events up and wait
until they've moved far enough soon
enough and to make a decision that yeah
they're really think they're inking and
all these things not the right place on
the screen and so on so we do have all
this logic in place and take care of
that for you if you install any point
event handler you will only get these
events when we've already done all that
and decided that yeah really the person
thinks they're writing at this point or
at least that's our system view of when
they ought to be writing so if you want
to for example draw your own ink you
have a calligraphy application where you
want to pay attention to pen orientation
and all these things and produce really
beautiful ink basically you can tell us
is stopped drawing handle the ink point
again and do what you want with it and
yet you'll get the same user interface
for your application is every other ink
aware or non Inc aware application in
the system ok the ink gesture event
allows you to intercept those gestures
that Julie was showing you so that when
you want to provide targeted gesture
support you just put in an ink gesture
event handler and now we'll send you
things when the user has written one of
these special gestures finally there's
the ink text event and that's what
provides that ain't text dress that
Julie described that lets you provide
the correction model with the alternate
word list and so on ok now I'm going to
do a quick run-through the api's this is
not the end F portion you're going to
get a really good walkthrough by Brad
regal in a minute who's going to show
you how to actually you know some sample
code show you how to actually use them
in some very specific examples this is
an attempt to give you a broad overview
so you know what to draw from and what's
possible in addition to the specific
that you'll see ok I mentioned those
four events there they are don't need to
go through them again because we've just
talked through them there's the instant
mouser there's the ink point there's the
ain't gesture and the text
this is the there's only one object in
in ink world and that is the ink text
ref and there are a very limited set but
a hopefully fully flexible set of ink
text API there's that you can find out
how many of those recognition alternates
there are the in there will never be
more than five you can find out you can
you can get the CF string for the top
choice or the second choice or third or
whatever you can get you can hand us a
menu ref and we will populate it for you
with all those alternates so it's very
easy to creat to create this you can
just in fact you'll that'll be an
example so you'll see that you can find
out what modifiers keyboard modifiers
were held down while the user was
writing there may be occasions where
that's useful you can grab a copy of the
text ref you can fly that its bounds you
can draw it into a wreck that you
specify you can flatten it that just
means you collapse the data structures
into a contiguous block of memory into a
CF data ref that you can write out to
disk should you want to save any of
these things and the counterpart to that
is that you can do it create from CF
data graph there are a few ink state AP
is that lets you query the state of the
system and set the state of the system
in certain cases for example you can
find out that I can't even see it but
there we go ink user writing mode that
lets you find out if the user has
basically turned on that right anywhere
option or not and the alternative to
write anywhere is right only an inca
where applications which is what you
will be writing the ink set application
writing mode as opposed to the user
writing mode allows you to basically
tell the system that you are one of
those ink aware applications that
should work in when the user has turned
off right anywhere ink set application
recognition mode lets you control text
recognition versus suggester recognition
and appropriate combinations of those
set phrase termination mode allows you
to take control over when and how
phrases are terminated including if you
want to eliminate our handling of it all
together and have it only Terminator
phrase when user presses a i'm done
writing button and you'll get an example
of that and ink is phrase in progress
well that's important for an AP i'm
about to show you this if if you think
you're going to handle control
terminating the phrase well it's
probably best to make sure there's
actually one in progress before you go
terminating and the only public
structure from ink is this ink point it
contains just three things the DHI point
the location on the screen where it
happened it gives you the tablet point
rec that's defined in carbon events h
its standard data structure that's
passed up all the way from the tablet
via is a tablet driver and on and the
keyboard modifiers that were held down
basically it's everything that we know
about the state of under which this ink
was written there's add stroke to
current phrase now especially if you're
a device specific solution where you
have a bunch of data that you want to
pass all at once to have recognition
performed on it doesn't involve the user
writing real time you'll probably be
adding a stroke at a time to the current
phrase until your idea of a phrase is
done at which point you'll call ink
terminates current phrase and add stroke
to current phrase just passes an array
of n points ok with that i think i'll
head it over to Brad Brad will take you
through some specific examples of
applying these things
[Applause]
Thank You Lori good afternoon everyone
we're going to take a look at some
developer scenarios we're going to walk
through the AP is that large is briefly
introduced and see how we can use some
of them in our applications to go beyond
the basic services that you already get
in Jaguar allow you to provide more
functionality for your users take
advantage of their tablets we're going
to take a look at three basic
applications or three basic scenarios
first one is going to be text engines
ways of text engines can take advantage
of the functionality we provide and make
better use of ink services second one is
going to be input devices these are
going to be you've got your own
whiteboard or pen or whatever and you
want two people to provide a custom
solutions for your users that takes
advantage of handwriting recognition
through ink services and the third one
is controlling phrase terminations it's
not really an application obviously its
applications that wanted to control when
phrases are terminated they don't want
to use the standard termination that we
have in place they may want to do
something custom because of the way that
users would input.text into their into
their application things along those
lines I'm not going to go through
in-depth the api's we've actually got
full documentation believe it or not we
have two documents one's a reference
document they actually goes through all
the api's and describes all of them in
depth and we have an overview document
that talks about a lot of what we're
talking about here today so first we're
going to take a look at text engines in
this case we're going to look at first
how you can use your how your text
engine can interact directly with
gestures so Julia showed earlier in
Jaguar right now you already get support
with gestures but only untargeted
basically what this means is if the user
words were to write a cut gesture clear
space things like that it only applies
to the current selection it's wherever
the selection is or the the insertion
point so in this case you'll notice up
at the top of the screen we have an
example of this actually boots done an
ink pad its ankle where
application you have the cut gesture
this case you'll notice that the
insertion point there's no selection
insertion point is sitting down next to
the word menu but we're actually using
the cut gesture to target cutting the
word contextual doesn't matter where the
selection is where the insertion point
is you can actually use your users can
use those gestures to directly interact
with text objects whatever makes sense
for your applications the bottom of the
screen you'll see another example which
is the clear gesture in this case Julie
showed you earlier you can write the
clear gesture today and it simply clears
the lease whatever the current selection
is in this case you'll see that the
insertion point is at the end of the end
of the phrase there but we're actually
going to clear delete the word
recognition right out of the middle of
the phrase using gestures so how do you
do this there we go uses you do this
using the ink gesture event that Larry
referred to in Chester event is pretty
straightforward so I got three
parameters the first parameter is just
what type of gesture is it what kind it
could be clear could be undo cut copy
paste if you look in the ink prep pain
it actually has one of the panels in
their lists out all the gestures that we
support the other two parameters are the
bounds of the gesture this is just the
global coordinates on the screen where
the user wrote that gesture and the last
one is the hot spot and we'll take a
look at those two and see how you take
advantage of those so hot spots and
bounds you need these you need to pay
attention to this in order to support
targeted gestures on the upper corner up
here you've got the cut gesture and
you'll notice that there's a red circle
around the very beginning point
basically you write the cut just you're
starting from the bottom point going up
the hot spot is treated as the starting
point for that gesture so when you're
interacting with gestures you're going
to look at that point to determine what
piece of text is easier trying to
interact with what object is easier
trying to interact with the other corner
we've got the space gesture in this case
the hotspot is actually the uppermost
point of that
upside-down V however you want to
describe it and wherever that point
lands is where you're going to insert a
space so once again as we saw in the
screenshot a minute ago doesn't matter
where the insertion point is what the
selection is you can take a look at the
gesture and the hot spot of it and do
the right thing the bottom of the screen
we've got the join gestures this is one
as Larry or start as Julie mentioned
earlier the join gesture is actually one
where it depends upon where it's written
on the screen how it's handled and in
this case you want to take a look at the
balance of the gesture there's not a
single hots but you take a look at the
bounds and so this one the bounds are
the upper to most corners of our the
hotspots really that you're looking at
are the upper tube points for the bound
and you're going to take a look at those
two points and say okay did those land
on text if so I want to treat as a join
join those two pieces of text together
if they didn't land on text you're
actually going to end up returning event
not handled for that gesture will then
turn around and recognize that gesture
as the letter V so real quickly it's
just a standard event handler your go
through and grab the kind grab the
hotspot and grab the bounds and save
those off and depending upon which
gesture you've got your going to do
things differently the undo gesture if
you're getting that one it's not
targeted toward anything it's just if
you get an undue gesture that's just
your standard whatever undo the left
user action currently in Jaguar the way
we make this work is and then pants or
if you don't handle these events what we
actually end up doing is turning around
if you don't handle the events we turn
around and for undo we end up issuing a
command you or whatever it makes sense
the standards command key shortcuts for
those actions so you actually don't have
to handle the undo if you don't we'll
just end up fish and command you
assuming your application supports that
so you're walking through handling the
events first one then do gesture just
call your your basic undo routine second
one is the cut gesture as we just saw
cut depends upon a hot spot so you're
going to have some routine that handles
that gesture and you're just in panda
hand at the hot spot and if the hot spot
lands on text then you want to cut
whatever object or text that gesture
lands on top of the hot spot lands on
top of if you determine that the hot
spot landed somewhere outside of texts
are not on object or whatever you'll
probably want to handle that as just
cutting whatever the current selection
is whatever makes sense for free
application the join gesture as we just
looked at this one you're going to look
at the bounds you're going to routine
that looks at the upper two points of
the bounds determines that they land in
text and if they do do the right thing
for your application and it's all that
comes back through they did hint you did
handle it and everything you just
returned nowhere and the vents done with
if you determine that that join gesture
didn't land on top of text then you're
going to go ahead and return event not
handled there like I said before and we
go ahead reprocess that and treat it as
letter V so second thing we're going to
take a look at is how to implement an
alternate menu for a text engine so as
you saw earlier this already supported
an ink pad and it's pretty
straightforward easy to implement
actually the ink pad and Panther uses
our public API as well and doesn't do
anything special with it so in this case
you've got your text editor what or
actually just gets has to be text editor
and you want to support bring up a list
of alternates for the word that's been
selected usually would be just a
contextual menu click or whatever makes
sense on that word so how do you do that
first thing you need to do is handle the
ink text event and as we saw earlier the
ink text event is sent once we finished
recognition users written everything
they're going to on the screen for a
phrase and we issue one in text event
per word that we recognize so if the
user wrote hello world or something like
that they're going to end up receiving
two ink text events and each text event
contains an inked xref
which the as we saw earlier one of the
pieces of information contains is the
list of alternates basically recognition
choices for that word and you'll end up
handling that in text event go through
the first thing for the Inca next event
is and actually it's missing out that
last slide there's actually one other
parameter which is is it a keyboard
shortcut because we handle just single
letters things like that and you could
be holding down a modifier key at the
same time the user could have actually
can write a command key shortcut purely
using the tablet without actually using
the keyboard and most cases free
application you're probably not going to
want to treat that as text you're going
to want a lod that filter through as a
standard command key shortcut so first
parameter is this a boolean says you
know do we believe this is the command
key shortcut command or control along
with a single character assuming it's
not then you're going to go through and
extract intex rep from the event you're
going to call any checks create CF
string with that text ref the 0
parameter there just means it's he took
the topmost choice this is you know the
piece of text that we think the user
just wrote your is then going to take
that CF string if your application
doesn't want CF strain you can you know
get the data out the CF string ref but
you're then going to insert that in your
document and one of the things you need
to do is when you insert that into your
document you also need to take a look at
the inked xref and keep that around
because you and you need to associate
that inked xref with the word that you
just inserted the reason why is is we'll
see here in a moment we end up using
that in text reps to create the
alternates menu so you want to implement
an all trans menu one single API does it
for you basically you're going to create
a menu ref this could be one that you've
already got laying around that you use
it's completely up to you how you want
to handle it you pass in concerted
ultimate sand menu and tap succinct
ex-rep is why we need to keep it that
last chip that corresponds to that word
that the user is clicking on and your
menu ref and we then inserts just like
you
on that screen shot like you saw an ink
pad demo earlier winds are at the five
alternate and we actually insert an item
that draws the original ink so the user
then you know Sid they look at it and
that's not what I wanted but then they
look at what they wrote that was what I
wrote they can see it all on the fly
there so we insert in the menu it's up
to you then you just call pop-up menu
select contextual menu select whatever
makes sense free application and when
you're done you'll just get the
selection that was made from that menu
and that's the piece of text that you
should replace in your documents so say
they wrote hello it got recognized with
the capital H they really wanted lower a
lowercase H at the beginning dull trans
men you would contain that item if user
selects that you go grab that menu item
text replace the capitalized version
with the opposite you know whatever
whatever the user chooses and that gives
you an alternate min and all trans men
you a single api along with a couple of
the menu many manager routines will do
it for you third scenario we're going to
look at input devices and how you can
create a custom solution for you your
users and the demo I have little demo
here it's actually pretty it's really
basic what our demo does just for the
sake of simplicity it could vary a lot
for whether or not you've got a pan or
your should be gathering data just in a
single application it varies a lot but
essentially what you're going to do is
collect the ink data in your application
and then you're going to tell us to
perform recognition on it and let me
show you a demo real quick here of what
I'm talking about so we've got a simple
application here and all it does is it's
got a area in the window here words in
the collect data and in this case I'm
using a tablet but it's just because I'm
used to in putting it that way this
application though turns off ink
recognition we don't get in the way at
all the application itself ends up
collecting the mouse events
and draws them on the screen so you'll
see you know this is not the standard
black all that kind of stuff the
applications actually drawing all this
on their own they're collecting the data
and then they end up sending us that
data we perform recognition on it and
just like the standard behavior is wind
up sending a series of ink text events
and then you get the text out of that so
let me show you how we do that so first
thing you're going to do set the
recognition state with ink set
application writing mode and you're
going to tell us don't do any
recognition don't do any don't allow
writing with ink services in your
application I'm going to handle it all
on my own for our demo app here like I
said for sake of simplicity we're just
looking Mouse events to paint on your
device and such you could gather data in
whatever manner you wanted to and as we
looked at earlier we gather data it's
just like we do internally you're going
to want to gather data into strokes and
strokes as simply as we said earlier a
basically mouse down mouse drag the
mouse up sequence defines one stroke so
you're going to gather data and once
you've gathered anantara stroke you're
just going to call ink add syrup to
current phrase and send us a stroke
along with how many points are in there
and we start building up a phrase for
you you continue doing this iterating
through for each short that you've
gathered data for and when you're
finished you call interment current
phrase and we go ahead and perform
recognition at that point so the code to
do it initial setup in this case we just
have a mouse event handler we turn off
on recognition for our applications
don't want equal getting in the way and
then we just have a simple mouse event
handler and as we go through we're going
to look for first mouse down event when
we see that first one we'll go ahead
start a new array of ink points start
building that up so you get mouse down a
serious Mouse drag events for each Mouse
dragged event you just may continue
building on your raving points and when
you get a mouse up event in our case or
whatever defines a stroke for your for
your case you
call ink adcirca current phrase send us
at array and the number of points in
that array and continue on in that loop
and when you finish gathering the data
you just call in terminate current
phrase no parameters or anything to it
we terminate the phrase that you were
sending us a data for we perform the
recognition send a series of ink text
events for each word that we recognized
and business as usual so last scenario
we're going to look at our controlling
phrase termination this is applications
where we your user may be interacting
with the application and I won't don't
want to a non-standard way but you want
to control wind phrases determinate you
want to control when you're getting the
data you wanted user to be able to
continue writing until the easier said
I'm done with this we're going to do
this by looking at the ink point events
that we talked about earlier and by
doing your own custom phrase termination
so let me show you quick demo of that as
well that will show you what we're
talking about so in this case it's just
a window and we don't do anything
special here except for the fact that we
have an API that says turn off automatic
for a determination so in this case the
application that is in complete control
and as Julie was saying earlier say the
phone rings oh I want to go answer the
phone or whatever whatever makes sense
free application notice I've taken the
pen away i'm waiting a really long time
no termination is happening you're a
complete control of it so the thing you
need to keep in mind here is you're in
complete control that the phrase will
not get terminated until your
application says to terminate it so this
ink will stay here until you finish so
we have a button down here at the bottom
and it's actually clicked on it there we
go and we've terminated the phrase
recognition happens we send ink text
events we get the data out of it go
along from their business as usual how
we do that
so once it back as we said I'm a
determination can be turned off you
thinking set phrase termination Julie
went through earlier when termination
happens normally you I actually also
have finer grain of control you can
control so that termination only doesn't
happen when the pen leaves proximity it
might only have or you can turn off the
only doesn't terminate if a timeout
happens or if it takes too long and if
you turn off termination all together
then you can control termination through
in terminate current phrase so the way
we do this one things you need to keep
in mind is we continue handling all
those Mouse events that are coming
through so normally you know it's we had
a button in there that controlled phrase
termination normally we're going to
continue writing once you started a
phrase we don't you know send anything
through two controls we don't do
anything we can keep sucking all that
data in is further data for writing one
of the things we talked about earlier
was for each mouse event that we get
through that we've decided okay the
users now inking for each mouse event
that comes through we send an ink point
event and the ink point event contains a
single parameter which is actually the
original mouse event and things you can
use this for is controlling web not to
even allow inking a certain region say
phrases started but you don't want to
allow inking outside of your outside of
a certain region in your window or in
our case we're doing with this
application is you want to if a mouse
down event happens in a certain area
such as on a button or things like that
you want to do something special with
that so if you return Avett not handle
therefore the mink point you can just
look at it see if you want to do
anything with it if your return event
not handled we continue on with
recognition continue sucking all the
data and if you return nowhere then we'd
basically drop that and regenerate or
repost the original mouse event so if
you say you've looked an ink point event
you've determined okay it landed an area
where I don't want them to treat ink i
want to terminate and start or just
repost as a mouse event so it gets to my
control you just return nowhere for that
ink point event
so the steps to do it except for a
determination of turn off automatic
termination you've got an ink point
event handler checks the location and
mouse down events in our case if the
mouse down lands on basically the region
where that terminate phrase button is
and if this currently phrase in session
then you just call internet current
phrase and we finished we do recognition
and you get think text events otherwise
just return event night handled error
and we continue building up bank data
code to do it is simply your handle any
point event it's not the cave and ink
text event that shows up intersection
point event you saw a hand learn it and
you call ink set phrase termination and
termination nuttin don't do any
automatic termination the applications
and complete control any point event
handler you're going to grab the mouse
event out of there as a single parameter
you're going to check to see for our
applications purposes you're going to
check to see was that a mouse down
advance if so check to see you know hits
us where did that event land if it
landed on the area where your button is
or you know wherever you're trying to
terminate then check to see if there's
currently a phrase happening you know
are you the actually inking right now
and if so terminate the phrase and
return to error everything finishes we
perform recognition otherwise like I
said just return the event I handled
there and we continue building up the
data and that's all you need to do for
that so summery things to keep in mind
if you're keeping account we have
actually less than twenty api's total
they're really straightforward we put a
lot of effort and make them easy for you
to use without a lot of complexity and
the carbon events that we have basically
take advantage of the current event
model they're really easy to just flow
into your current event handlers and
basically we just want to encourage you
to add more support to inc allow your
users to take advantage of the service
that are provided to you and ink
difference so we have as I mentioned
earlier we have two documents that are
available on the ADC website and they're
already up there one is the services
started the ink services reference which
goes through and actually documents all
these api's and everything and goes into
pretty in-depth we've got some sample
code in there as well that shows you how
to use them and also we have an overview
document that shows you how to integrate
ink services in your application goes
through some of the samples we've got
here review all that and see how you can
take advantage of it the application
that's just using here is called ink
sample we've also got the code for that
posted so you can go take a look at how
we do it and if you're interested in
more information on how the recognition
technology works Larry's got a website
it's p.o.box calm / tilde Larry why and
she's got a lot of good information up
there as far as how the recognition
system works and if you have further
questions you can contact Savior and
send them questions feedback