WWDC2003 Session 428

Transcript

Kind: captions Language: en good afternoon and welcome to session for 28 understanding the new ink adi of mac OS 10 we are thrilled today to introduce a new set of a TI to mac OS that they will allow you to provide additional support for handwriting recognition in your application as you know with jaguar apple introduced inkwell the end customer solution for handwriting recognition in quill provides system-wide support for cocoa and carbon application without requiring any modification to the object and what I'd like to do first today is give you a brief overview of the functionality the ink will support in the application so what I have here is a Wacom tablet I welcome tablet and the first and i have downloaded the Welcome driver from the web and the first thing I'm going to do ready this is it's got to the system preference and no that is a ink panel dare say I'm going to launch that panel and I can enter user control this is going to allow me to turn black nation on and off the condition is already on but if you wear off I turn it on and that's all basically all I need to do to have system-wide support for handwriting recognition or Jewish there's small stuff that you can do in the ink reference panel basically tweak the settings for recognition look at the gestures that we support our words to the user dictionary but not now going to go into those right now so let me quit thanks reference panel and now I'm ready to start inputting text in my cocoa application I have texted it up here and i'm going to start writing anyway I like on the screen and you'll see that the strokes and the writing guidelines will show up the strokes get cheap as a unit to the recognizer recognized expect same Chicago as a carbon event Coco will extract the information and enter into the document you can continue to write on this tablet strokes we Bo collected recognize and boom they go to my application as easy as that tax ID have not been modified to support recognition however equal allows you to do much more than text that entered in your test is all walla is always also allows you to do editing so for instance if I hold down the stylus the 10 will go into Mouse mode and now I have a selection I can ask upon the selection and a can of Ages and I can erase it by just doing a simple advice gesture on the screen and boom that word is God but I really didn't want to erase that word I want it back in my document and I'm going to do an undue gestures a sheikh abdul gesture but there it is i got my word bathrooms on the dog so now let me switch to a carbon application and i'm going to show you that exactly the same paradigm works there i have i cube and say that i bought a new CD and i want to add it to my collection now i want to give it a name they're going to pick up my stylist and I'm going to write the name for my new CD there you go I'm done so without any modification I can enter test edit and manipulate the UI however this is that's pretty powerful but there is much more than really applications could do if they had access to the information that the recognition system provide and to give you just a glimpse of that i'm going to show you what they in fact that the inside is the real input window that we ship on java that is accessible from the in windows and that we like to call it the first inca web application so as you see you can go to the inside and I right there on the window again the text the ink is recognized and will show up on the document however if I can go now to that same text and because this is an Inca well application I can double click and hold and I can look at all of the alternatives that the recognizance assistant has given us on top of that i can choose one of them if I like or and I can look at this trove that they use it that I've written so that I get feedback on what that word was meant so i can select that and now the case for the words changed okay let me use the undo works right it okay so i can continue to write on the inside this exactly the same way that I'd wrote on text edit well the phone rang and I stopped in the middle of the word where I can pick it up right where I left it let the other part of the condition and then do a gesture to merge those two pieces of those two words as I wrote so with a simple gesture that looks very much like an E I can merge those two string and now notice that because this is an Inca web application I can erase a word just by scribbling on top of that without having to go through the process of selecting and then erasing so there is much more power that you can add with your application by using the stylus directly on the object and developers have recognized that and since we introduced in work we have had the experience of requests to be able to for developers to tap into the ink services that we provide the requests of range from questions like how can I add support for devices older than the Wacom tablet to request that we're developers are asking about separating the process of gathering the digital ink from the process of recognizing they basically doing the federal Commission if you are familiar with the term two questions about developers who want to have more control on where and when the ink is drawn so we have taken for thunder we have taken those frequent and combined it with our vision on how handwriting support should work on Mac OS 10 to provide you with a set of very simple yet powerful a tree eyes will allow you to provide additional support for the handwriting they will allow you to develop to ship to your customers their solutions that they are looking for so today what I want to do for the rest of the presentation is introduce you to some of the concepts that we are going to use throughout the presentation some very calm basic concepts that we use in a in da TI and then we're going to talk briefly about the technologies that you use behind the scenes to provide the recognition support and finally we're going to go into the need of the presentation and give you a brief overview of the API with support and how you can use them to develop to develop solutions for your customers one question that we often get is what kind of device do I need in order to get it in well or in services to work and for job adequately answer was you need a Wacom tablet and welcome driver for Paris we're going to continue of course to support the Welcome devices but we are going to add support for anything for devices that allow you to provide x y and pressure information either in a real-time fashion like tablets do or an offline way like really available USB and Bluetooth stance that you have to these days on the market now let's go through the concept they're basically size comes a very simple concept we are going to use throughout the presentation and in their documentation the first one is an interest and you are already familiar with that think of a new trades as a group of strokes that stays on the screen before then ships you the recognized they have a little bit of no answers to that concert you can't leave them the documentation but then basically is what you need to think about is food for this presentation now as I was writing on the tablet and I lifted the pan the increase was terminated didn't recognize it and then to the application there are other ways that I can terminate a phrase and basically I can if I hold a pen for a while and nothing and I don't do anything else a timeout is going to happen and the edge place is going to terminate the other way that in phase is terminate is because I if I write my stroke two apart on the X direction and on the eyes y direction that's also way the day in phrase is terminated and the reason why I'm telling you this what is it important concept to you is because later on when we go to the developers nails and the api's we are going to talk about ways for you to take control of its face termination in case if your application needs to do so the next concept is in tax rap and if if the strokes you'd wrote are recognized by dating services as text essentially what you are going to get getting your application is an in-text read an in-text rest is an opaque object that contains information about this digital ink that you wrote the location of that ink and the information from the recognize that set of strings so that recognizes provided the alternate for the ink and the API is provide a way for you to access most of the information that is available on the integration and of course the alternance menu if you want to present you to use it the results of the recognition so that they can browse the information we're going to provide a simple API for you to construct that menu and show it to the user and then a and the menu will contain again the alternate splicing and the final concept that I want to get a go of it is gestures you saw me use all of these gestures in the demo and so we basically have pretty tired to gestures the first time the needs position myself on the right hand on the right hand of the screen if they undo gesture and we call those kind of gesture non-targeted gestures and what their those kind of gesture what it really means is no matter where you draw draw them on the screen they're going to perform the same action the second type of gestures which I have here a hundred top left of the screen and what we call optionally targeted gesture and an example of that is the arrays gesture and you saw me use it in two different context in text studies which is at this point not an Inca whereas I had to do a selection in order to erase in the eighth pad in which have been the ninka where application you can just erase on top of the text so that is that there is a difference for those Gentry and we have additional gestures that are optionally targeted and finally the third class of gestures is the joint gesture those are gesture that I've only supported in inca 'what up because we need to know where on the screen digestion occurs so there is no way for the in services to differentiate between a v4 the joint gesture unless the application give us the information or word on the context of the application the juice will happen so again there are three classes objection and the reason why we're telling you this is because you again can choose to support those gestures in your application to allow the user to do direct manipulation to interact with the stylus directly with the object without requiring them to go through the metaphor that is where that is needed for the keyboard but you have to select and then act upon the selection you can just go and use the stylus directly in the other and so the non-target objector basically usual need to add the support the system can handle it for the optionally targeted gesture you may want to add the support if you want that is direction to happen because you need to add the information of the location and for the always targeted gestures the support has to come from the application and then concludes kind of the basic concept that you need to do know today now let me move on to the ink component and we have basically force components for the inch services on the right hand side of the screen that blue rectangle shows the application context and in the application context two of the components live there the ink input method is the part of the ink services that provides data collection and drawing of ink so as you right pink input method is going to gather the data android the other components on their context of the application is a framework that's what the implementation of the API on the left-hand corner of the screen the box shows you the user preferences which I show you in the demo and that contains it with the ending contains the press thing that allows users to set their recognition mode look at the gesture turn gestures on and off add words in the dictionary and finally on the top corner of the screen we have the inserts that's where the call of the recognition lives whether it's a segmentation the recognition process the language model is instantiated and in addition the ink we know but the important for you today is the fact that that's where the core of the recognition sleep and with that it is my pleasure to introduce Larry Jaeger Larry Larry Larry to distinguish engineer at apple the world for us for several years in handwriting recognition and I'm very pleased to have human stage today Thank You Julie okay I'm going to give you a quick run through some of the technologies underlying what julie has just demonstrated for you and even dive down a little bit as recognition technology though that doesn't directly affect your apps it's sort of what provides all these services the recognizer can be boiled down to something very simple basically these three boxes pretty much define the last ten years of my life the it starts off with XY points and pen lists it just exactly what you would imagine are coming from the graphics tablet and we take those and in strokes what you'd normally think of a strokes with pen and paper and we look at combinations of those strokes to see what might possibly be characters and that's what we call character segmentation so out of this tentative segmentation block comes these character segmentation hypotheses and some of them are going to be writing some of them are going to be wrong and we can't possibly predict upfront which the the next block with it is weak there's a neural network classifier it takes these combinations of strokes that might be characters looks at them in a certain way and tries to decide whether they are or not and which characters they are so what idiots are these character class hypotheses it's trying to say if it's ABCDEF or whatever that sequence of probability vectors are fed into a search engine and the search engine sits there with a language model that's the context and tries to just find the best combination of all those possible character combinations and vectors of probabilities of characters and look for the maximum likelihood pass through this graph and end up with finally word probabilities what the person wrote so started to see what that looks like suppose a person wrote this on the on the computer you you know hopefully maybe fifty percent of the people in the audience thought it was clogged and titty percent thought it was dog although dogs such common word may be a little skewed that way well what we do is take those strokes and take the what the first stroke that kind of looks like a sea and feed it into the neural network classifier it emits this vector of probabilities and let's say it's pretty good you know point ninety percent certain that it's a see okay that's great next we'll probably try the first two strokes together and let's say that well you know there's a written kind of far apart that's awfully big gap in there let's say the neural net kind of picks up on that and says well seventy percent probability that's a that's a D and you know a few other things scattered around when the next when that second stroke is then tried by itself the only legal transition is from the sea to the L you can't have the C and the D because they use the same strokes so let's say that the neural Nets pretty good and it's got about an eighty percent probability that that's an L well I deliberately jury-rigged the numbers so that that looks like point nine times point eight is point seven two so seventy-two percent chance that the theory so far is is CL 74 chances that's right well and I assigned seventy percent to the D so so far our little engine thinks that it's more likely to be CL than it is d okay well when the Oh stroke comes along let's say it's pretty good there's an interesting thing going on here that it doesn't almost doesn't matter what that probability is at this point the CLO is going to wind over the do because the CL 1 over the D and that's going to be true when we bring up the G you multiply those probabilities together and indeed ever so slightly the way I've jury-rigged the numbers it would probably pick clog but in fact as I mentioned earlier it's guided by context we have a language model that's fairly complex is guiding all this and you can write outside of the dictionaries you don't have to write in the dictionaries it's a loosely applied language model but you but it does guide the search and so if you were searching with the pet philosophy newsgroup where the only words were cat dog and dogma that would guide this search and so the only viable path through that graph of the language model would be dog and so really dog would win and there's there's tons more information actually published available on the web that will give you a URL for later okay so given that the recognition engine is sitting under there ready to provide these answers let's take a look at how the data flows through the system before it gets to your application and again you don't have to worry about every little detail in here but hopefully this will give you an idea enough about how things are working that it'll make sense when you have to when you do have to make calls to make things happen just the way you want you'll understand why those are the right things to do so initially data comes in through the graphics tablet that tablet data is fed into really this is kind of two layers in this next block but I just lump them together to fit it on the slide gracefully there's um there's a tablet driver living at the i/o kit io hid layer and what it's really emitting is Mouse events with tablet data them so they really are true honest to god first first-rate citizen mouse events but they also have this block of data that has to do with pressure and orientation of the pen and all the special things that pins can do those Mouse events left alligator are fed through the core graphics layer of course and the core graphics layer does something very special for us you may have noticed while Julia was demoing ink text and some of the other things she did not have to write in any of ink texts windows she could write off apparently on the desktop we call that our write anywhere model and it's something the user can turn on or off but we turn it on by default if you turn ink on at all because it's beamed we think it's an easy way to interact with the system when you're using a pen basically you right where you want to the recognition results flow to the insertion point so core graphics is sitting there and if this mat this mouse event coming through the pipeline has tablet data in it and recognition is enabled and we told it that in the background then it will route those events to the frontmost app that's going to be important later because you might very well in the process of handling ink related events get points that lie outside your windows and you kind of need to know that okay so there's one other thing that core graphics does for us if we're going to allow you to right outside of windows and write all over the screen well there are some places where you're going to put the pent the user is going to put the pen down that they really want it to just behave like a mouse when you put it down in the drag region of a window you don't want to have it start writing while you're trying to drag the window if you put it down on the scroll bar you don't want to start writing while you're trying to scroll the window and the doc is the same way so we've invented this idea of instant mousing regions and so there are certain special places that if the pen lands in one of these regions it will instantly behave as a mouse instead of inking and so core graphics has its own idea of a few very special places like the menu bar and the dock that are known and the draggable regions of windows that are server-side drag that that it knows to go ahead and treat instantly emmaus then we take so it's finally still emits just a mouse event with tablet alien and this is now we're getting into your application context area but first our input method our tech services manager TSM input method gets a chance at the data first we have our own idea about instant mousing and take care of some things for you like most controls and user interface probably want to be instant mousers what we know about that so all your standard carbon and cocoa controls are automatically instant mousers except for the ones that where we think you ought to be able to write there's going to be a way for you to manage your own custom controls in a minute as Julia said we gathered that ink data up into strokes we had those strokes to the idea of this current phrase and eventually get around to terminating the phrase as Julia said we send that off to the recognizer we get the results back and in Jaguar we post first we post a Unicode for key event so that apps that are already TSM savvy and looking for that can get the whole block of text at once but if you don't handle that then by gosh we turn into rocky downs so finally your application will get the data the text data corresponding to what the user wrote for Panther we still have Unicode for key and key down at the end of the chain but we've just introduced for events that lets you manage what happens with ink in your application there's an instant Mouser event that lets you decide if some special region on your document layout needs to be an instant now sir hopefully there won't be any of those but this lets you manage it just in case we send an ink point event for each of the points that comes in that we think should be inking now you know I've talked about instant mounting and write anywhere and all that there's always this decision going on when a person is using a pen whether it is supposed to be the mousing or inking and there's a fair bit of logic we cue events up and wait until they've moved far enough soon enough and to make a decision that yeah they're really think they're inking and all these things not the right place on the screen and so on so we do have all this logic in place and take care of that for you if you install any point event handler you will only get these events when we've already done all that and decided that yeah really the person thinks they're writing at this point or at least that's our system view of when they ought to be writing so if you want to for example draw your own ink you have a calligraphy application where you want to pay attention to pen orientation and all these things and produce really beautiful ink basically you can tell us is stopped drawing handle the ink point again and do what you want with it and yet you'll get the same user interface for your application is every other ink aware or non Inc aware application in the system ok the ink gesture event allows you to intercept those gestures that Julie was showing you so that when you want to provide targeted gesture support you just put in an ink gesture event handler and now we'll send you things when the user has written one of these special gestures finally there's the ink text event and that's what provides that ain't text dress that Julie described that lets you provide the correction model with the alternate word list and so on ok now I'm going to do a quick run-through the api's this is not the end F portion you're going to get a really good walkthrough by Brad regal in a minute who's going to show you how to actually you know some sample code show you how to actually use them in some very specific examples this is an attempt to give you a broad overview so you know what to draw from and what's possible in addition to the specific that you'll see ok I mentioned those four events there they are don't need to go through them again because we've just talked through them there's the instant mouser there's the ink point there's the ain't gesture and the text this is the there's only one object in in ink world and that is the ink text ref and there are a very limited set but a hopefully fully flexible set of ink text API there's that you can find out how many of those recognition alternates there are the in there will never be more than five you can find out you can you can get the CF string for the top choice or the second choice or third or whatever you can get you can hand us a menu ref and we will populate it for you with all those alternates so it's very easy to creat to create this you can just in fact you'll that'll be an example so you'll see that you can find out what modifiers keyboard modifiers were held down while the user was writing there may be occasions where that's useful you can grab a copy of the text ref you can fly that its bounds you can draw it into a wreck that you specify you can flatten it that just means you collapse the data structures into a contiguous block of memory into a CF data ref that you can write out to disk should you want to save any of these things and the counterpart to that is that you can do it create from CF data graph there are a few ink state AP is that lets you query the state of the system and set the state of the system in certain cases for example you can find out that I can't even see it but there we go ink user writing mode that lets you find out if the user has basically turned on that right anywhere option or not and the alternative to write anywhere is right only an inca where applications which is what you will be writing the ink set application writing mode as opposed to the user writing mode allows you to basically tell the system that you are one of those ink aware applications that should work in when the user has turned off right anywhere ink set application recognition mode lets you control text recognition versus suggester recognition and appropriate combinations of those set phrase termination mode allows you to take control over when and how phrases are terminated including if you want to eliminate our handling of it all together and have it only Terminator phrase when user presses a i'm done writing button and you'll get an example of that and ink is phrase in progress well that's important for an AP i'm about to show you this if if you think you're going to handle control terminating the phrase well it's probably best to make sure there's actually one in progress before you go terminating and the only public structure from ink is this ink point it contains just three things the DHI point the location on the screen where it happened it gives you the tablet point rec that's defined in carbon events h its standard data structure that's passed up all the way from the tablet via is a tablet driver and on and the keyboard modifiers that were held down basically it's everything that we know about the state of under which this ink was written there's add stroke to current phrase now especially if you're a device specific solution where you have a bunch of data that you want to pass all at once to have recognition performed on it doesn't involve the user writing real time you'll probably be adding a stroke at a time to the current phrase until your idea of a phrase is done at which point you'll call ink terminates current phrase and add stroke to current phrase just passes an array of n points ok with that i think i'll head it over to Brad Brad will take you through some specific examples of applying these things [Applause] Thank You Lori good afternoon everyone we're going to take a look at some developer scenarios we're going to walk through the AP is that large is briefly introduced and see how we can use some of them in our applications to go beyond the basic services that you already get in Jaguar allow you to provide more functionality for your users take advantage of their tablets we're going to take a look at three basic applications or three basic scenarios first one is going to be text engines ways of text engines can take advantage of the functionality we provide and make better use of ink services second one is going to be input devices these are going to be you've got your own whiteboard or pen or whatever and you want two people to provide a custom solutions for your users that takes advantage of handwriting recognition through ink services and the third one is controlling phrase terminations it's not really an application obviously its applications that wanted to control when phrases are terminated they don't want to use the standard termination that we have in place they may want to do something custom because of the way that users would input.text into their into their application things along those lines I'm not going to go through in-depth the api's we've actually got full documentation believe it or not we have two documents one's a reference document they actually goes through all the api's and describes all of them in depth and we have an overview document that talks about a lot of what we're talking about here today so first we're going to take a look at text engines in this case we're going to look at first how you can use your how your text engine can interact directly with gestures so Julia showed earlier in Jaguar right now you already get support with gestures but only untargeted basically what this means is if the user words were to write a cut gesture clear space things like that it only applies to the current selection it's wherever the selection is or the the insertion point so in this case you'll notice up at the top of the screen we have an example of this actually boots done an ink pad its ankle where application you have the cut gesture this case you'll notice that the insertion point there's no selection insertion point is sitting down next to the word menu but we're actually using the cut gesture to target cutting the word contextual doesn't matter where the selection is where the insertion point is you can actually use your users can use those gestures to directly interact with text objects whatever makes sense for your applications the bottom of the screen you'll see another example which is the clear gesture in this case Julie showed you earlier you can write the clear gesture today and it simply clears the lease whatever the current selection is in this case you'll see that the insertion point is at the end of the end of the phrase there but we're actually going to clear delete the word recognition right out of the middle of the phrase using gestures so how do you do this there we go uses you do this using the ink gesture event that Larry referred to in Chester event is pretty straightforward so I got three parameters the first parameter is just what type of gesture is it what kind it could be clear could be undo cut copy paste if you look in the ink prep pain it actually has one of the panels in their lists out all the gestures that we support the other two parameters are the bounds of the gesture this is just the global coordinates on the screen where the user wrote that gesture and the last one is the hot spot and we'll take a look at those two and see how you take advantage of those so hot spots and bounds you need these you need to pay attention to this in order to support targeted gestures on the upper corner up here you've got the cut gesture and you'll notice that there's a red circle around the very beginning point basically you write the cut just you're starting from the bottom point going up the hot spot is treated as the starting point for that gesture so when you're interacting with gestures you're going to look at that point to determine what piece of text is easier trying to interact with what object is easier trying to interact with the other corner we've got the space gesture in this case the hotspot is actually the uppermost point of that upside-down V however you want to describe it and wherever that point lands is where you're going to insert a space so once again as we saw in the screenshot a minute ago doesn't matter where the insertion point is what the selection is you can take a look at the gesture and the hot spot of it and do the right thing the bottom of the screen we've got the join gestures this is one as Larry or start as Julie mentioned earlier the join gesture is actually one where it depends upon where it's written on the screen how it's handled and in this case you want to take a look at the balance of the gesture there's not a single hots but you take a look at the bounds and so this one the bounds are the upper to most corners of our the hotspots really that you're looking at are the upper tube points for the bound and you're going to take a look at those two points and say okay did those land on text if so I want to treat as a join join those two pieces of text together if they didn't land on text you're actually going to end up returning event not handled for that gesture will then turn around and recognize that gesture as the letter V so real quickly it's just a standard event handler your go through and grab the kind grab the hotspot and grab the bounds and save those off and depending upon which gesture you've got your going to do things differently the undo gesture if you're getting that one it's not targeted toward anything it's just if you get an undue gesture that's just your standard whatever undo the left user action currently in Jaguar the way we make this work is and then pants or if you don't handle these events what we actually end up doing is turning around if you don't handle the events we turn around and for undo we end up issuing a command you or whatever it makes sense the standards command key shortcuts for those actions so you actually don't have to handle the undo if you don't we'll just end up fish and command you assuming your application supports that so you're walking through handling the events first one then do gesture just call your your basic undo routine second one is the cut gesture as we just saw cut depends upon a hot spot so you're going to have some routine that handles that gesture and you're just in panda hand at the hot spot and if the hot spot lands on text then you want to cut whatever object or text that gesture lands on top of the hot spot lands on top of if you determine that the hot spot landed somewhere outside of texts are not on object or whatever you'll probably want to handle that as just cutting whatever the current selection is whatever makes sense for free application the join gesture as we just looked at this one you're going to look at the bounds you're going to routine that looks at the upper two points of the bounds determines that they land in text and if they do do the right thing for your application and it's all that comes back through they did hint you did handle it and everything you just returned nowhere and the vents done with if you determine that that join gesture didn't land on top of text then you're going to go ahead and return event not handled there like I said before and we go ahead reprocess that and treat it as letter V so second thing we're going to take a look at is how to implement an alternate menu for a text engine so as you saw earlier this already supported an ink pad and it's pretty straightforward easy to implement actually the ink pad and Panther uses our public API as well and doesn't do anything special with it so in this case you've got your text editor what or actually just gets has to be text editor and you want to support bring up a list of alternates for the word that's been selected usually would be just a contextual menu click or whatever makes sense on that word so how do you do that first thing you need to do is handle the ink text event and as we saw earlier the ink text event is sent once we finished recognition users written everything they're going to on the screen for a phrase and we issue one in text event per word that we recognize so if the user wrote hello world or something like that they're going to end up receiving two ink text events and each text event contains an inked xref which the as we saw earlier one of the pieces of information contains is the list of alternates basically recognition choices for that word and you'll end up handling that in text event go through the first thing for the Inca next event is and actually it's missing out that last slide there's actually one other parameter which is is it a keyboard shortcut because we handle just single letters things like that and you could be holding down a modifier key at the same time the user could have actually can write a command key shortcut purely using the tablet without actually using the keyboard and most cases free application you're probably not going to want to treat that as text you're going to want a lod that filter through as a standard command key shortcut so first parameter is this a boolean says you know do we believe this is the command key shortcut command or control along with a single character assuming it's not then you're going to go through and extract intex rep from the event you're going to call any checks create CF string with that text ref the 0 parameter there just means it's he took the topmost choice this is you know the piece of text that we think the user just wrote your is then going to take that CF string if your application doesn't want CF strain you can you know get the data out the CF string ref but you're then going to insert that in your document and one of the things you need to do is when you insert that into your document you also need to take a look at the inked xref and keep that around because you and you need to associate that inked xref with the word that you just inserted the reason why is is we'll see here in a moment we end up using that in text reps to create the alternates menu so you want to implement an all trans menu one single API does it for you basically you're going to create a menu ref this could be one that you've already got laying around that you use it's completely up to you how you want to handle it you pass in concerted ultimate sand menu and tap succinct ex-rep is why we need to keep it that last chip that corresponds to that word that the user is clicking on and your menu ref and we then inserts just like you on that screen shot like you saw an ink pad demo earlier winds are at the five alternate and we actually insert an item that draws the original ink so the user then you know Sid they look at it and that's not what I wanted but then they look at what they wrote that was what I wrote they can see it all on the fly there so we insert in the menu it's up to you then you just call pop-up menu select contextual menu select whatever makes sense free application and when you're done you'll just get the selection that was made from that menu and that's the piece of text that you should replace in your documents so say they wrote hello it got recognized with the capital H they really wanted lower a lowercase H at the beginning dull trans men you would contain that item if user selects that you go grab that menu item text replace the capitalized version with the opposite you know whatever whatever the user chooses and that gives you an alternate min and all trans men you a single api along with a couple of the menu many manager routines will do it for you third scenario we're going to look at input devices and how you can create a custom solution for you your users and the demo I have little demo here it's actually pretty it's really basic what our demo does just for the sake of simplicity it could vary a lot for whether or not you've got a pan or your should be gathering data just in a single application it varies a lot but essentially what you're going to do is collect the ink data in your application and then you're going to tell us to perform recognition on it and let me show you a demo real quick here of what I'm talking about so we've got a simple application here and all it does is it's got a area in the window here words in the collect data and in this case I'm using a tablet but it's just because I'm used to in putting it that way this application though turns off ink recognition we don't get in the way at all the application itself ends up collecting the mouse events and draws them on the screen so you'll see you know this is not the standard black all that kind of stuff the applications actually drawing all this on their own they're collecting the data and then they end up sending us that data we perform recognition on it and just like the standard behavior is wind up sending a series of ink text events and then you get the text out of that so let me show you how we do that so first thing you're going to do set the recognition state with ink set application writing mode and you're going to tell us don't do any recognition don't do any don't allow writing with ink services in your application I'm going to handle it all on my own for our demo app here like I said for sake of simplicity we're just looking Mouse events to paint on your device and such you could gather data in whatever manner you wanted to and as we looked at earlier we gather data it's just like we do internally you're going to want to gather data into strokes and strokes as simply as we said earlier a basically mouse down mouse drag the mouse up sequence defines one stroke so you're going to gather data and once you've gathered anantara stroke you're just going to call ink add syrup to current phrase and send us a stroke along with how many points are in there and we start building up a phrase for you you continue doing this iterating through for each short that you've gathered data for and when you're finished you call interment current phrase and we go ahead and perform recognition at that point so the code to do it initial setup in this case we just have a mouse event handler we turn off on recognition for our applications don't want equal getting in the way and then we just have a simple mouse event handler and as we go through we're going to look for first mouse down event when we see that first one we'll go ahead start a new array of ink points start building that up so you get mouse down a serious Mouse drag events for each Mouse dragged event you just may continue building on your raving points and when you get a mouse up event in our case or whatever defines a stroke for your for your case you call ink adcirca current phrase send us at array and the number of points in that array and continue on in that loop and when you finish gathering the data you just call in terminate current phrase no parameters or anything to it we terminate the phrase that you were sending us a data for we perform the recognition send a series of ink text events for each word that we recognized and business as usual so last scenario we're going to look at our controlling phrase termination this is applications where we your user may be interacting with the application and I won't don't want to a non-standard way but you want to control wind phrases determinate you want to control when you're getting the data you wanted user to be able to continue writing until the easier said I'm done with this we're going to do this by looking at the ink point events that we talked about earlier and by doing your own custom phrase termination so let me show you quick demo of that as well that will show you what we're talking about so in this case it's just a window and we don't do anything special here except for the fact that we have an API that says turn off automatic for a determination so in this case the application that is in complete control and as Julie was saying earlier say the phone rings oh I want to go answer the phone or whatever whatever makes sense free application notice I've taken the pen away i'm waiting a really long time no termination is happening you're a complete control of it so the thing you need to keep in mind here is you're in complete control that the phrase will not get terminated until your application says to terminate it so this ink will stay here until you finish so we have a button down here at the bottom and it's actually clicked on it there we go and we've terminated the phrase recognition happens we send ink text events we get the data out of it go along from their business as usual how we do that so once it back as we said I'm a determination can be turned off you thinking set phrase termination Julie went through earlier when termination happens normally you I actually also have finer grain of control you can control so that termination only doesn't happen when the pen leaves proximity it might only have or you can turn off the only doesn't terminate if a timeout happens or if it takes too long and if you turn off termination all together then you can control termination through in terminate current phrase so the way we do this one things you need to keep in mind is we continue handling all those Mouse events that are coming through so normally you know it's we had a button in there that controlled phrase termination normally we're going to continue writing once you started a phrase we don't you know send anything through two controls we don't do anything we can keep sucking all that data in is further data for writing one of the things we talked about earlier was for each mouse event that we get through that we've decided okay the users now inking for each mouse event that comes through we send an ink point event and the ink point event contains a single parameter which is actually the original mouse event and things you can use this for is controlling web not to even allow inking a certain region say phrases started but you don't want to allow inking outside of your outside of a certain region in your window or in our case we're doing with this application is you want to if a mouse down event happens in a certain area such as on a button or things like that you want to do something special with that so if you return Avett not handle therefore the mink point you can just look at it see if you want to do anything with it if your return event not handled we continue on with recognition continue sucking all the data and if you return nowhere then we'd basically drop that and regenerate or repost the original mouse event so if you say you've looked an ink point event you've determined okay it landed an area where I don't want them to treat ink i want to terminate and start or just repost as a mouse event so it gets to my control you just return nowhere for that ink point event so the steps to do it except for a determination of turn off automatic termination you've got an ink point event handler checks the location and mouse down events in our case if the mouse down lands on basically the region where that terminate phrase button is and if this currently phrase in session then you just call internet current phrase and we finished we do recognition and you get think text events otherwise just return event night handled error and we continue building up bank data code to do it is simply your handle any point event it's not the cave and ink text event that shows up intersection point event you saw a hand learn it and you call ink set phrase termination and termination nuttin don't do any automatic termination the applications and complete control any point event handler you're going to grab the mouse event out of there as a single parameter you're going to check to see for our applications purposes you're going to check to see was that a mouse down advance if so check to see you know hits us where did that event land if it landed on the area where your button is or you know wherever you're trying to terminate then check to see if there's currently a phrase happening you know are you the actually inking right now and if so terminate the phrase and return to error everything finishes we perform recognition otherwise like I said just return the event I handled there and we continue building up the data and that's all you need to do for that so summery things to keep in mind if you're keeping account we have actually less than twenty api's total they're really straightforward we put a lot of effort and make them easy for you to use without a lot of complexity and the carbon events that we have basically take advantage of the current event model they're really easy to just flow into your current event handlers and basically we just want to encourage you to add more support to inc allow your users to take advantage of the service that are provided to you and ink difference so we have as I mentioned earlier we have two documents that are available on the ADC website and they're already up there one is the services started the ink services reference which goes through and actually documents all these api's and everything and goes into pretty in-depth we've got some sample code in there as well that shows you how to use them and also we have an overview document that shows you how to integrate ink services in your application goes through some of the samples we've got here review all that and see how you can take advantage of it the application that's just using here is called ink sample we've also got the code for that posted so you can go take a look at how we do it and if you're interested in more information on how the recognition technology works Larry's got a website it's p.o.box calm / tilde Larry why and she's got a lot of good information up there as far as how the recognition system works and if you have further questions you can contact Savior and send them questions feedback