WWDC2001 Session 127

Transcript

Kind: captions Language: en thank you thank you so the good news is that uh can you walk hear me yep the spoken language technologies in case you haven't noticed are in Mac OS 10 we got on there whoo thank you what we're going to do today is describe briefly the speech recognition and the speech synthesis that are there we're going to give you guidelines about where to use them in your applications and why you would want to use them in your applications and we're going to actually lead you through the process of getting your applications talking and listening so let's start off by doing a demo of what we've got over here can we switch to far oh I'm supposed to do an ad buy I'm supposed to do it whoo there we go the user interface to speech recognition is this round window that you may have seen this replaces the face that some of you might be familiar with in OS 9 it consists of three parts the middle if you you might be able to see says escape this shows you the listening mode you there are two different listening modes with speech recognition what we call push-to-talk mode and continuous listening in push-to-talk mode it's only listening when you hold down a key it says ESC which means that by default is the escape key users can configure that in continuous listening mode is listening all the time and optionally you can have it wait for a key word like computer before you speak your commands I am using push a talk mode here and I recommend that you do as well when you're demoing so that so that when you're explaining things to people it's not try to recognize commands that you're actually intending to other people not to the computer what time is it it's 10:30 what day is it it's Thursday May 24th show me what to say okay the other part of us of the feedback is this speaker speech commands window which has two halves the top half which is scrollable shows what it has recognized and if it speaks back to you what it says to you the bottom half which also is scrollable and all of which is resizable by the user at last Thank You cocoa shows what you can say there are now disclosure triangles so that it no longer Scrolls off the bottom of the screen the middle item there I don't know whether you can see it says speakable items that shows the commands that can be spoken all off the time no matter what application is running I mentioned at the start I spoke what time is it what day is it their items down in there these are actually kept in the speakable items folder let's take a look at it open the speakable items folder there it is mm-hmm so any item that is in this speakable items folder can be launched by speaking it and it's just the same as double-clicking on it applications aliases documents servers URLs anything that you can launch by double-clicking you can now launch by speech on OS 10 the real power of this is that users can customize it to the way they work by dragging their own items into the speakable items folder in addition just like in OS 9 know the speakable items folder itself contains a folder called applications speakable items this contains folders which are named by applications the items in those folders are only speakable when that application is in the foreground that's shown in the speech commands window in the top item the top disclosure triangle so you see at the moment let me close the others that it says finder and the finder ease of in the foreground let me demonstrate this in action I'll switch to my browser and as I do you watch the items there change open my browser there so it now stays Internet Explorer and there are some different items there this is an opportunity for you developers you can make application specific folders for your applications and populate those folders with scripts or other commands that control just your application they won't be speaker ball when your application is not itself is not in the foreground oh I'd like to encourage you to explore this by yourselves I just want to show you one other thing hide this application and that is that you may have noticed we shipped one game and that is chess Oh in telling you this I should point out we keep a track of all the applications that you've launched since we started we actually restart the machine and you can switch to any application running or not just by saying its name switch to chess so the chest application you might have seen mentioned and demoed in the keynote on Monday one of the keynotes as being an example of a good user interface aqua quality and one of the things about it is that you can control it by speech porn d2 to d4 Knight b1 to c3 what's it going to do to that let's see it's thinking it's thinking how than I am but it's a better player than I am too one of the things I like about this is this one thing I can do with this that I can't do when I'm playing with real people and that's the following I can say take back move my twelve-year-old son does not let me do that when I'm playing with him okay let's move on so what speech is there in Mac os10 bear with me there are a lot of people at this conference this year who are you new to the Mac OS platform some of you who are already familiar with the platform will know already what we have so it's just a brief mention for those that are get from it familiar with it we have speech synthesis and speech recognition the speech synthesis will take any text and convert it into audible speech there are 22 different voices they range from adult male and adult female through two voices that sing and sound like aliens and novelty voices and we have speech recognition there are a number of characteristics about the speech recognition they're important first is it's speaker independent that means you don't have to Train it to your voice you just take it out of the box and it just works the Mac was and remains the only computer platform I know of that you can just take out of the box put on the desk and command by voice it's continuous speech you don't have to pause between words it uses a far-field microphone this is a technical point but a very important point speech recognition is very sensitive to background noise so sensitive that all other recognizes that I know of require that the users mount purchase and use a noise canceling close talking head mounted microphone we tune up our recognizer to work with the inbuilt microphones that are built into the IMAX and the other CPUs that Apple delivers that means that we are getting the background noise that other recognizers are getting so we have several layers of software to adaptively model and subtract and deal with that background noise but there are limits on how much background noise we can deal with so we've tuned it up to work well in the situations where most of our users are using their computers in the office at home if you have users or customers who are using speech recognition in noisy environments such as in classrooms then it might be pushing the limits of the head mount of microphones a little bit too far one such environment for example is giving a presentation in an auditorium with a what 300 watt sound system that's providing a slapback echo from from the back of walk hall and so for that this is one of those environments where I've been using this head mounted microphone this one is produced by VXI they have worked quite a lot with us over the last couple of years to optimize their microphones to work with our speech recognition and so you can direct your users if you want to those as an alternative solution our speech technologies currently are US English only on OS 10 so what's new in speech synthesis we've done a few things we merged the Mac and talked three and the map and talked Pro codebases into a single codebase this has got a couple of advantages one is means that we have at last a totally new codebase we have divested ourselves of deep intricate and interpretable legacy code and positioned ourselves to at last be able to fold in the research improvements that we've been making in speech synthesis in the last few years so we've got ourselves into a new platform ready to go forward an immediate benefit right now is that you get consistent behavior across voices a lot of developers have said to me that they they change from one voice to another and words are pronounced differently or the intonation is different or that will no longer happen all the pro voices are now just the high-quality versions so what was Victoria high quality in OS 9 is now called Victoria and there is no lower quality version in OS 10 that means the speech is crisper it's easier to understand it's more robust in the presence of background noise people can understand it without it being so loud and we've improved the pronunciation in a of ways we've enlarged the dictionary from about 20,000 words to about what a 120,000 Jerome's work down here the morphological decomposition is now recursive if you want to know what that means ask us in the question time it's fascinating really really it's very cool it's a it's a multiplier on the effectiveness of the dictionary and the letter to sound rules are now automatically trained based on the new large dictionary rather than handwritten based on one linguists intuitions for speech recognition we also have a benefit here we have factored out that pronunciation subsystem from the speech synthesis and made it a separate subsystem that's now shared between speech synthesis and speech recognition this means first of all it reduces the overall RAM which improves performance of everything running on the platform it means the recognition is more accurate because the pronunciation subsystem is expecting the correct pronunciations for more words and it gives consistent behavior across speech recognition and speech synthesis developers have said to me I was prototyping some spoken commands for my application and one of the commands was not recognized very well and I thought that perhaps the recognizer was not listening for the correct pronunciation so to find out what the recognizer was listening for I tucked my command into your text-to-speech system and sure enough it was spoken with an inquiry incorrectly pronounced word well up until today that's been irrelevant now it is relevant now if you want to know whether the recognizer is expecting the correct pronunciation they type the word though you're the word or the commander that speech synthesis and it will tell you the way the recognizer thinks people will pronounce that word the user interface has been completely revised as I showed you and there's an improvement of speakable items we've added xml-based command files that you associate a spoken command with a keystroke sequence Kevin Aiken later in this presentation will show you that in some more detail I want to talk about why you should use speech in your applications there are two classes of applications I think from our perspective there are applications that are centered around speech where speech technology is central to the users experience and central to the value that application delivers and then there's a huge number of applications I think most of you write that for which speech is not centrally relevant at all I'd like you to think about places that you can use speech in those applications as well chess is an example speech is not really relevant to chess but we entered it and people say hey that's cool if you add speech to your application then you'll increase the number of potential users and that increases your market for example younger users will find your application more approachable people with disabilities will be able to use your applications and people who are less familiar with computation will be less scared of trying out your application speech is a very natural form of communication we've all been talking and listening since we were what two years old wait one and a half years old speech and layout enables you in your application to move beyond the limits of point-and-click there's nothing wrong with point-and-click in fact it's very good at letting you control things that you can see on the user interface and reach with a single gesture but there are lots of things that you want people to control that they can't see to point to to click on speech gives you a way to get past that if you think about it clicking is rather like grunting turns we turning our back on what about 200 thousand years of human evolution because when I click on things I'm just going that's true I'd like like to think that we've come somewhat further than that similarly speech output I think can be a lot better than just beep so many of us are still using alert sounds well beep was the mentality of 1960s when all that computers had was this tiny little speaker we've come forward since then speech gives you a way to bring yourselves into the 21st century and conversation is an appropriate modality for delegating tasks to a computer we'll illustrate that a bit more shortly so what are the weights some of the ways that you can use speech synthesis in your application one is notifications we recommend that you judiciously get the users attention back if the users attention has wandered away from your applications some of you may have known about or experienced talking alerts that we did in OS 9 now there's a slider that lets you set how long between the delay between an alert coming up and it being spoken you may want that longer or shorter than the default but the point is you normally should not hear any speech except when your attention wanders and the computer wants your attention and then if you don't respond it gets your attention back you can use notifications for asynchronous events for example in the AOL Instant Messenger if you're buddies enter your chatroom then it will announce that with our speech synthesis it will say buddy Smith just into the room you can give you speech synthesis to give a fictional additional feedback for younger users and we have found and our users have found that applications for the 6 to 12 space become accessible for the K through to 6 space if they don't change anything but just read out the text messages that they're putting up on the screens and you can use speech synthesis for proofreading for example you have an application where people are entering data to say a spreadsheet and people are into entering budget figures in a column allow them to select that column and have it read back so they can just check what they were entering to make sure that there aren't any errors speech will give you your application more accessibility for those with disabilities I think that's pretty obvious is really cool in games you saw that we did it with with chess there are a lot of games already that you're using speech for things like cheat codes and changing weapons use speech for non time-dependent control I don't think it would be appropriate to use a speech recognition in your game to say fire now quick quick quick right lift but you can do it for that as well if you want to try it in noisy games these are headsets are probably the right thing to do so the speech recognition doesn't confuse your commands with oh there are a lot of people who are successfully using speech for in education applications and I think there are a lot more of you with education applications that could take advantage of that one example would be the dinette product which is using speech recognition for pronunciation correction for adults learning English is way cool check it out you can use speech to enhance the web browsing experience for navigating within a browser if those of you who have explored will see that we actually ship a couple of spoken commands for Internet Explorer that let you do some simple navigation if you have a browser you can do a much better job than we do by working within it for example people could speak speak the links jump to pages by topic read out web pages and there's a big opportunity with voicexml the enterprise industry that's moving more and more of its information onto web access is now doing two or three different versions of all of their websites they're doing the HTML version they're doing a web version for personal digital portable digital assistants to access their stuff via wireless and they're making voicexml versions so that people can ring up the webpage and have the information read out over the phone the way this is done is by an extra set of tags that are in the web pages that are just the thing that you need to interpret for speech speech access and you can do that on a desktop from with a Mac using our API is it would be pretty straightforward because the infrastructure and the hard work has all been done for you by the web developers we recommend that you think about using speech for form-filling as an alternative to people filling out things with pop-up menus you can now have people speak the contents of each field and you can use it that lets you use a constrained language model for each field to increase the recognition accuracy for example the person could say create a new customer record and the computer could then respond what type and then narrow down its search to just the alternative customer records you person could say okay corporate account and then the next field would be payment schedule and the person could then say 30 days and and that's saved through the recognition model was changed again to just listen to the possible let's just listen for possible possible payment schedules okay there are lot of tasks where people's eyes are busy their hands are busy for example you're in a graphics program you're drawing you've got the mouse down you're putting a line across some an object and you want to move it around send it to the back or change the pressure sides when the eyes are busy in the hands are busy speech gives you a another way for your users to control your app so at this stage I'd like to invite salsa going up salsa goyan is the Apple script product manager and he's going to show you some way cool ways that he's been using speech so is this huh great hi this is amazing that I got up this early for this whole thing those that know me know that it's an impressive feat what we're gonna be showing today is can you switch me to this or me switch me to this is how to use Apple script with speech that's one of the best integrations on Mac OS is the ability to use these two technologies together and on Mac OS 9 we introduced the ability to have a script listen for a response and based upon the users response perform a different set of actions and incorporated a technology called the speech listener so I'm going to show a couple scripts today that use this technology on Mac OS 10 and both scripts will involve a conversation to get a task done the first one is a rather straightforward example where I asked the script for some music and it prompts me with a series of questions and we have some music played so let's see if my voice is back where it should be and we'll try this out some music please which artist or category Christine Caine which song from Christine Kane Tucson [Music] stop playing songs so in this example the script starts up it got the information about which artists were available held it into memory and said which artist or category when I say Christine Kane it matched that then queried and found out which songs were available by Christine Kane held that into memory and then said which song I said Tucson that matched and then it had the song play with iTunes so this is a simple example you got a program in a certain amount of grief with these things just to keep you honest so there's an example of being able to carry on a conversation with the script it's a limited scon verse a ssin but it is a way of gathering information and moving forward and the hide this application in the next example I'm going to use a program from by up systems it's called goat reap and one of the things that this application does is it accesses information over the Internet and I'm going to use a script acting as a person a personality called Victoria and Victoria will act independently of the speech recognition speakable items in that she will have her own set of scripts that she's going to use in conversing with me so here we go let's try this out and see if she's awake - Victoria yes L show my newspaper here you go something else clear all stories all stories have been removed from your newspaper Edie multiple anything else add multiple stories ready Motley Fool adding monty fool ready apple stock quote adding Apple stock load ready Appl top story adding apple top story ready that's all anything else not at this time anything else not right now not right now goodbye so in this instance when the script was called Victoria which exists in speakable items when the script loads up he goes to a subfolder in the speakable items folder called tasks and within that folder our individual scripts the names of which she holds into memory when I say show my newspaper she loads that script and then executes that so you have a script running a script and all of the commands that Victoria were doing are not included in the standard speakable items commands they are a subset so you can create these individual personalities so that's just two examples of how you can use Apple script in speech and if you're interested in how to do this the Apple script website has a complete overview it has an apple script guide book of how to use speech and Apple script together thank you [Applause] some issues if you're going to include speech in your application then there's a few things that you need to keep in mind educate you users about how to speak a good example would be go to speech preference panel and turn on speakable items and you'll see a sheet come down and you'll see how we explain to users that they shouldn't pause and and so on let them know about background noise being a problem you might want to refer them to head mounted microphones we train the speech recognizer on North American English and so officially that's what we say we support that happens that it is somewhat forgiving and so I'm Australian and our group we have Jerome from France we have Devon where are you Devon native Gujarati speaker we have mutt ears from speech swiss-german as a native language we even have Tom from the Bronx and understands all of us but there again there are limits and localization is an issue currently as I said we are US only so it's time to code for speech programming 101 I'd like to invite Matias near Hoffa up on the stage Matias okay now that Kim has told you what to do with speech Kevin and I are going to talk about how to do this using our speech technologies in Mac OS 10 is pretty simple we are installed in every install of Mac OS 10 to use us just link with carbon framework or if you have a carbon-based application with the carbon lip our api's are identical for cocoa for carbon the same API is that you used on Mac OS 9 you can use them from Objective C from C or from pretty much any language that we ship on Mac OS 10 let's start with speech synthesis let's say you want your application to say something how difficult can this be turns out it's not difficult at all it's a single line and you will get hello world okay that was simple enough if you want to have a little bit more control you open a speech channel giving a voice this can either be something that you get from a menu you give to the user or if you pass in now you get the default voice you probably shouldn't hard-code a voice unless you know exactly why you would want to do this then you can adjust parameters as you like them and once you have them to your liking you can speak the actual text by calling speak text all of these calls are asynchronous so it will actually return control to you before the text is entirely spoken we offer a lot of control you can control the speech rate to peak speak slower for younger users for instance or quicker in a game situation for instance we can control the speech pay modulation so it sounds more lively at the volume to customize the way speech sounds in your application we also give you callback routines so you can when you have a screen reader you can highlight words as on-screen as they are spoken or if you have an animated character on-screen you can animate the lips of the character as the phonemes are spoken you can see many of these options in action in our Koko speech synthesis example which ships on the developer CD most of these controls you actually don't have to write any code because you can simply embed them in text so for instance in the sentence if you want to emphasize the word next you just embed an emphasis and emphasis command in front of the word don't meet LLL make Tuesday this is very important because to really have speech synthesis work for you the best it can you should customize what is spoken basically your application just knows a lot more about how things should be spoken then our engines can know by default so there are a number of things you can and should do with your application first of all you should filter the text that is passed on to the text-to-speech engine for instance if you have a stock ticker application and you come across the acronym AAPL what you should do is tell the speed texture speech engine to say Apple computer instead second you should customize the pronunciation of words that don't come out right and last of all you should customize the intonation of what is spoken now we try to have a huge dictionary as cam already said but even the biggest dictionary cannot possibly handle all the words and not all proper names especially for instance my first name is tricky it's certainly our system cannot pronounce it by default in the past some developers have just used funny spellings to get it to work approximately the right way like this my name is Monty here sounds almost right but we don't recommend this because if you use words that are not part of the English language or strange combinations we might change in the future how this is pronounced second this is not a very precise way of specifying what you want said so instead what you should do is use embedded commands to temporarily switch to phoneme input using the phoneme notation we describe this notation in inside McIntosh speech it's explained with examples on a page doesn't take very long at all to learn how to use it and the result is something like this my name in whom RTL sounds somewhat better second you should customize the intonation of the text you pass on to text-to-speech because words the written words alone are not always enough to convey the meaning for instance if you see the sentence you can read it as John only introduced Mary to Bill he didn't introduce her to anybody else you could read it as John only introduced married to Bill he didn't introduce Caroline to him or you can read it as John only introduced Mary to Bill he didn't ask her to marry him so these distinctions can be very important so you should annotate the text you pass on our system tries to do it the best it can to to find out how a sentence should be spoken but it this can be very difficult if not impossible to do in the general case your application has domain knowledge of my much of the text that is spoken and has the potential to do bus much better for instance take a flight reservation system at the end it gives it a confirmation text and I'm going to play your two different versions of saying this confirmation text first of all is not annotated at all and the second is annotated I'm not going to say anything between the two versions your first flight is with Alaskan airlines flight 2762 departing from San Jose on Monday May 24th at 6:10 landing in San Francisco at 7:10 p.m. thank you for using tea two years travel your first flight it's with Alaskan airlines flight 27 62 departing from San Jose on Monday May 24th at 6:10 p.m. landing in San Francisco at 7:10 p.m. thank you for choosing TT years travel so so did you hear a difference raise your hand if you heard the difference between the two versions excellent I see that the hands are up so we did this with quite a bit of annotation and this can basically be distilled to five principles of how to improve the intonation of the spoken elements the first principle is let the user catch up by adding pauses at strategically important points at punctuation wherever appropriate and appropriate does not mean appropriate in the sense of English grammar nobody is going to see what is spoken so feel free to add a comma if you think a pause is necessary at a point breakups larger sentences into smaller ones and insert some explicit pauses with the silence command at major pause points so in our example we added punctuation we added pauses all of this lets the user catch up second command second principle is to let family familiar things go into the background by de-emphasizing repeated words for instance if the minutes are identical you should de-emphasize the second instance also de-emphasize in items inferable from your overall application scenario you know that you're booking a flight so you don't have to emphasize this word third principle is to liven it up simply by adding an exclamation point at the end fourth principle is to focus the users attention by emphasizing the important words this can be done with an emphasis command or simply by inserting a colon before the most important item and faced and maybe most important use paragraph intonation group your sentences together into international paragraphs and the first sentence in this paragraph you should raise the pitch pitch range and then reset it for the rest this makes quite a bit of difference for longer texts that are read you write you raise the pitch base and increase the pitch modulation and then decrease it after the first sentence and between paragraphs add an extra pause so to summarize you should customize the pronunciation of words that you say if a word you notice that a word you have hard-coded in your application gets mispronounced used for nemo to get it pronounced correctly you should customize the intonation of the texture that is said which helps understanding that you helps the user understand the text a lot better and gives the user a much better overall speech experience now let's move on to speech recognition with my colleague Kevin Aitken who is not lazy at all I might add can y'all hear me yes I can hear myself well my manager Kim is actually yeah assured me that this is not necessarily a commentary on my work ethic so I feel better but if you're like me once in a while I do feel a little bit lazy and at those times I just love having a simple solution to be really productive and so on the next few thing next 15 minutes or so I'm going to show you two easy methods that you can use to add spoken commands to your application and so hopefully an afternoon is worth of work you can walk into your manager's office or your co-workers office and say oh by the way our Mac os10 application understands spoken commands so let's get started so as I mentioned I'm going to provide two methods so the first method is to use the speakable items application that's built into Mac OS 10 as Kim demoed in the beginning it's designed for end-users so they can easily add spoken commands to any application and so you as a developer can use this also so it's great because you don't have to write any speech code speakable AIIMS takes care of this for you by taking a list of items building the language model and then waiting for the recognition result and itself showed in his demonstration it understands how to execute Apple script so that you can easily send Apple events to your application or even other applications for that matter now the second method I'm going to describe is to use the speech recognition API you may be familiar with this a mac os9 it gives you a little bit more flexibility you can have multiple command lists so you can have one set of commands for when the user is selected in an object and one set of commands when they haven't and I'm going to show you in a little bit of an example that gives you a really easy three-step approach to adding spoken commands to your application using the speech recognition API well one of the things that both of these methods have in common are commands and so let's talk just for a second about what makes a good command well commands are like menu items but we suggest that they're normally from three to six words long the longer the better generally because the recognition system can understand them easier and they're more unique amongst your other commands but you don't want to too long to where the user has a hard time speaking and fluently also you should avoid single words and especially words like hot cut and quit because those are oftentimes Mis recognized or they sound a light to the recognition system the other important item is that you should test your commands especially test them together to make sure that they're confused with each other and test them with the global commands that are shipped with Mac os10 and a prototype your commands you can use speakable IMS or the SR language modeler applications you'll find on a developer CD okay let's talk about method one in a little more depth and this is using the speakable items application so the first thing is I mentioned you want to create a number of items you can easily do this by bringing your application to the foreground and speaking the command make this application speakable that creates a folder in the speakable items directory as Kim showed you earlier inside that application speakable items folder and then you can begin adding your items as he showed you so once you have all your items together the next thing is you want to bundle those inside of your application so I'm gonna show you a little bit an example you can use project builder to easily copy these files into your application bundle at the time you build it and then finally you need to install those items now we really suggest that you install them at runtime this gives you a couple of added benefits it allows your application to be drag-and-drop installed so therefore in order to support speakable items you don't have to have a separate installer to install the items and it's great for Mac OS 10 support for multiple users because after a user after your application has been installed let's say the administrator creates new users well that new user is just going to get those speakable items the next times they run your application because you'll automatically install them well let's talk about items real quick Kim briefly mentioned those it's basically a file that can be open but there's really two types that are best for you developers the first are Apple script files as I talked about allow you to send Apple events your application the other one that Kim mentioned are the new XML based command files and then allow you to sand keyboard events to your applications so that you can activate menus or controls via keyboard shortcuts well one of the things that I wanted to do in preparing for this WWDC this year is create an example that really showed how easy it is to add spoken commands your application I really wanted to make as simple as copy paste go so as you saw in the sales demonstration you use iTunes and that's a really pretty good real-world application it's shipping and you can see how those are integrated into the application well since I can't ship or give you the source code for iTunes I thought well let me create a clone of it so I've named mine foe tunes courtesy of our French person in the the group and it's up on the web right now that you can go grab it at this URL and hopefully you can either grab it this week or right now or when you get back and start taking a look at it it's really I believe shows a really easy way of getting going so let me go show it to you oh I need to switch like everyone else I forgotten to switch okay let me hide this real quick so let me show you the application my clone of iTunes real quick it has the identical menu items and the window is you know pretty close I mean if you haven't done anything in cocoa or interface builder I basically took 15 minutes through all the menus in there laid out the window and pretty much got an automatic resizing window it's really awesome okay let me show you that it really is listening for commands show commands window show speech commands window okay that's of course display visual get song info get song info there we go as you see it doesn't do anything it just shows the command down below okay so it really is folk tunes okay let's switch into project builder and I'll show you how this is set up let me try the command switch to project builder yeah okay go I'll move those out of the way so we can see the window here I'll put this down since I'll be giving it more commands okay so let me show you real quick what this basic object looks like that manages the window it's really simple has a couple of instance variables and then it pretty much has a method for each one of the menu items and a couple extras to handle some of the controls in the windows so it's really simple all these methods do is basically display at the bottom of that window what's happening okay so as I mentioned the first step was creating items so we've created those items the next step is that we need to include them in our application bundle so what we'll do is we'll go to at the active target and the way we do this is we use a file copies build phase and so we include those down here let me show you where you do that and here if you haven't already seen it you got a new build phase and you get a new copy build phase it's not highlighted right at the moment because I haven't selected a particular item but as you can see I've included the items here I have two sets of of items I have the command files it's a majority of them and so I'm saying place these any folder name command files inside the inside the resources directory of the application bundle and then I have a single Apple script file that I've included in here as well okay so now project builder has made that easy now they're being copied when I build it being copied inside the application bundle the next task is to install them at runtime so we've tried to simplify this a lot by providing a single routine that you can call so here it is install speakable items for this application you'll pass in the name of those folders that you placed your items into your resources directory and then you'll call it and it's smart enough to go out creates the folder if the folder is already there it doesn't create it again in the case of this demo I actually call this routine every time it startup but you could choose to call it lazily later or you could call it in response to a user specifying it and the Preferences dialog or something like that the rest of this file has a tutorial or documentation in more depth than I've talked about here about creating the items adding those items to your application bundle and then how to call this routine and special notes so it's all there ok let's go back to the slides for a second ok great so let's touch on the second method for a minute here and that's using the speech recognition API well you know just like the previous example I want to make it a copy-paste and go solution so we're trying to provide some really simple routines that you can use so what I've done is I've broken down the process of recognition setting up for recognition and handling the recognition into 3 easy steps and you'll see in a minute where I provide you with a single routine to execute each one of these steps so let's talk a bit about basically what the recognition process looks like this is a very simplified version of it and a graphical version of it that just kind of cements it in your minds as to what this example is doing and generally what recognition process is what happens during that ok so step one we provide a routine that basically sets up all the recognition objects instantiates a recognition system a recognizer object a language model object hooks them all together and that's all set up and ready to go something that most virtually every developer has to do when they adopt the the Apple speech recognition API the second step is that you need to tell it what commands will isn't for so in the routine that we give you pass in the recognizer object and then you pass in an array of commands it basically gives a recognizer object those commands to display and the speech commands window then you also pass in the language model object and it gives the commands to that so the recognition engine knows what to listen for and then for the third step you need to implement one Apple event handler the speech dun Apple event handler and so now your application is just sitting there you've set up it's running ready to handle the user spoken command so when the user says something the recognition engine passes it off to the recognizer object it then sends an apple event to your application in this case play this song so we provide you a single routine you pass in the apple event that you get and it returns an ID and then you can take this ID and map it through a switch statement or a table lookup or however you like to a particular action or routine okay so let's go back to project builder and I'll show you how this is done by the way this single project has two targets one for building the speakable items based version of photons and one that uses the API so it's all in here you can build either one and to some extent contrast and compare okay we provided in this speech routines dot C and header file those three three routines so let's see him here it is you basically call setup speech recognition that accomplishes step one then you call add commands with a list of your commands you can call add commands over and over as you need to to change the set of commands if you want to be able to provide the user different commands based on whether they've selected a particular object and and your interface or they're in a particular portion of your application and then inside the Apple event handle that you've created you'll call this one routine pass in the Apple event and you'll get the ID that was connected with that original command that you set up okay so let's look at basically the application delegate object where I do this what I've done is I use an application delegate that gets the application did finish launching method after KOCO has brought my app up and is basically running and so I pretty much do it all at the front just for this demonstration so that code is all in one place the way I approach it is I create a simple table basically that has my command name and then the method to be called so that's the the way I'm doing it here actually do it programmatically at the bottom of this file you could do an XML file you could do it differently it's up to you I register the speech done apple event handler by calling the AE install event handler then I call the routine to set up speech recognition that we provide in that utility file I create an array of the commands because I need that to pass to the add commands routine that we provide and then finally I call our SR start listening routine that we that's part of the API and now the application is up and running page down here this is the Apple event handler just a couple of lines of code I call the routine that we provide an utility file to convert the Apple event into the ID and then so the Wonder is an objective-c I just use as index into a table and then basically go off to the appropriate method selector so that's pretty much it I really urge you to go out and grab this and see how this can be applied to your application so let me summarize real quick are you summarize real quick so we saw in the method of the first method using the speakable AIIMS application that's really easy because you don't have to write any additional speech code all you need to do is include those items inside your application bundle and then install them at runtime with the routine that we provide the second method is using the speech recognition API as I explained that's an easy three-step process that we give you a single routine to execute each one of those steps so I've discussed the lazy way to do it there's more things you can do with the speech recognition API and Matthias is gonna come up here and talk about what to do if you're feeling a little bit more ambitious so thank you thank you Kevin my manager assures me that overachievers does not necessarily apply to my performance either so kevin has shown you how to get 95% of the benefits but 5% of the work however there are some situations where you might need the extra 5% one example of this is chess you've seen it demoed it ships with Mac OS 10 as of this week you can get the source code from this URL here you will find the speech related code in chess listener and chess illustrates important lessons in language model design now you might think the language model of chess is not very complex right d2 to d4 all simple sentences problem is if you just do this as a list of possible moves it gets out of hand pretty quickly if you do the math you find out that if you just have a model with all the possible moves you end up with more than 24,000 moves and clearly this is unacceptable it doesn't help accuracy plus you're not doing the user any favor if you're listening to stuff like rook a1 to h8 its won't do him any good at all in fact it turns out that each in each chess position there are only 20 to 30 moves that are actually legal so there is no reason whatsoever to include the extra moves performance is going to go away up and user satisfaction is going to go up if you only include legal moves however you shouldn't quite over constrain your model there are some illegal moves which are still plausible for instance people frequently put their King into check accidentally even experienced chess players so what you would want to do is to leave a move like this in so you can say I heard you but I won't do it another technique that we use in chess is to use prefabricated parts there are not so many words that are actually used in this language model so we we fabricate that them at start up by calling s our new word to get these word objects and then when we come to a position and see that for instance pawn d2 to d4 would be appropriate we simply grab these prefabricated objects and paste them together to form this this command so to summarize for complex language models you will want to constrain your language model to only those commands which are plausible in each situation and consequently you adapt the language model when the situation changes furthermore in very complex situations you might consider using prefabricated language objects to quickly get your list of commands to build these language models we've included a tool called SR language modeler which helps you to quickly experiment with different language models how well they work for your users SR language modeler allows both live microphone tests for rapid turnaround if you want to try something or grab somebody into your office to have him try something if you want to do systematic scientific research tests you can record your users saying these commands record them into AIFF files and feed those files into SR language model to get a systematic evaluation of how well this performs this tool and all of our sample code which we shaped with Mac OS 10 you will find on the developer CD in the examples slash speech folder and we encourage you to start with that if you want to do anything with speech so let me now turn our session back over to our fearless leader came Silverman for some values of fearless so to summarize speech synthesis and speech recognition are there we've given you a conceptual overview of the api's and I try to give you some ideas about why you would want to use them and Mathias and Kevin have followed up with how to use them well so I want to encourage you all to speech enable your apps at this stage I'd like to single out just a couple of developers who've been doing this you might remember thinking home which got the Apple Design Award last year they've poured of their application to OS 10 and have been at they added speech to it and they found that that adds a lot of value to their users I was talking to one of their developers on the phone yesterday he said they're getting a lot of feedback saying that users just think it's great when they can walk into the room and say things like dim the lights in the living room or turn the upstairs thermostat to cool the folk who were working on Omni web you may have seen their cool browser on OS 10 they've been experimenting also with using speech to its integrate speech e to the browsing experience for those that don't want to or can't deal with the keyboard in the mouse we saw it a prototype of that this morning it's looking really good they've got some great ideas about how to do it so now you're going to give you the guidelines of good better and best about how to put speech into your apps good is the easy way use speech recognition to allow people to speak the visible controls on on the screen things that they can would normally do manipulate them and say them and use speech synthesis to speak simple alerts and alert panels when they come up you can do these by the way with either the speakable items framework or by calling the API directly if you want to go better then use delegation I've mentioned this a few times you probably inferred what I mean normally when we interact with a computer we specify explicitly each step we want the computer to take in order to reach a goal that we have in mind with delegation we delegate the goal to the computer and have it figure out the steps about how to get there and then execute them for us so group what would be otherwise multiple interactive actions into one coke in command for speech synthesis start to customize your texts using the guidelines that Mattias went through so if you want any help on those read back information to your users and if you want to be best then move to interactive spoken dialogues like you saw Sal demonstrating where you delegate a goal to the computer or your agent anything comes back and asks you questions to refine that goal and think about using speech for form filling so that's it thanks a lot for coming I am I