WWDC2001 Session 502

Transcript

Kind: captions
Language: en
today's session is going to be about
describing at a high level what we have
to offer and three new Apple Java
frameworks Java spelling framework Java
speech synthesis framework and Java
speech recognition to framework these
API are free and they will be available
this week
surely by Friday they're in the pipeline
and we just need to put them on the web
from an apple developers website and you
can download them and use them with Java
and your favorite IDE jbuilder that you
got in the bag metro works project
builder what have you and have some fun
with them we're not going to go into a
lot of detail about how speech synthesis
and speech recognition work or about all
their capabilities but by that I mean
the underlying technologies built into
OS 10 that have been around for a while
in different versions of OS 9 OS 8 etc
but you can find more information about
those things on our Apple developers
website and we're assuming that you all
are familiar with Java
now one thing we're going to talk about
is Java beans one of the things we've
designed into these api's is to allow
them to be used in visual builders the
premiere example is jbuilder and before
we go on I just wanted to review exactly
what a Java Bean is so it's basically at
a high level method that that was
designed by Sun and its partners to be
able to share discrete components among
third party developers and tool vendors
so that each tool could interoperate and
use visually these discrete components
so technically a Java Bean is basically
a class that has certain method
signatures or another class called a
bean info that is its pair that
describes properties it can also
generate events so for example the
classic bean would be a button and you
could get an action event on that button
for example and the job of being
specification Alta also defines methods
to provide editors and customer
that is something richer then you'd get
by default in a visual builder such as
just editing a string or cranking up a
number up and down switching true to
false etc and Java beans of course are
part of the of the Mack exchange j2se
implementation part of all j2
implementations so the first framework
will discuss is the java speech
framework and the job of speech
framework is basically layered upon the
synthesis and recognition technologies
building to us - and as I said before
these are from carbon they exist on mac
OS 9.1 and carbon as well the native
versions and of course we're running on
OS 10 here today so what are some of the
differences between what we're talking
about today and the underlying
technologies well the java speech API
object based whereas the carbon API our
C API native API the Java frameworks
provide programmatic as well as visual
programming for the frameworks the
carbon API are only programmatic java
speech API are a higher level the the
carbon C API provides a very very low
level every single possible thing you'd
ever want to do with speech synthesis or
speech recognition the java speech API
are only available on Mac OS 10 the
native API are available on carbon 7.5
and beyond
so speech synthesis basically it's about
converting text to speech and we provide
in our Java frameworks about 99%
coverage of the functionality you'll
find in the carbon API its architectural
II neutral now what we mean by that is
speech synthesis provides the ability to
get callbacks instead of just saying
speak this text you may want to know
when it's done speaking a particular
word a sentence when it's done speaking
a part of a word you can even stick
other cues into the string that it's
speaking and we'll go into that more
later and get callbacks from the speech
technology now those callbacks won't
certainly happen on the UI thread of
whatever particular UI toolkit you're
using swing and I Bhatia IFC cocoa etc
so what we've done is we've put in a
thread task handler and a default
implementation for swing to provide that
synchronization for you by default and
an abstraction in the task handler
interface that allows you to add your
own synchronization between the speech
thread callback thread and the UI thread
and we provide javabeans and customized
property editors for tight integration
with the job IDs so the main classes
there are others but the main two
classes of speech synthesis or the sense
of synthesizer itself and it basically
allows you to speak the text and attach
event listeners to the different events
it generates the voice class which
embodies all the characteristics of a
voice now Mac os10 as Mac OS 9 and in
previous Mac OS s have has built-in
voices about a half dozen different
voices of different characteristics one
sounds like a robot one sounds like a
young girl one sounds like a older man
etc etc and these all these voices have
names and the voice class allows you to
pick one of these voices by name adjust
its properties properties such as the
rate with which the voice speaks its
pitch its volume etc and we have three
basic events we generate from the speech
synthesizer the first event is the word
event and you can listen for that event
whenever the synthesizer concludes
speaking a particular word you can also
listen for the sync event now the sync
event is something that you can embed a
cue for in the text you're asking the
synthesizer to speak and I'll
demonstrate that later in a photo event
which is used to animate for example the
face let's say you wanted a
computer-generated face with a mouth to
really move with the sound of the word
you would use these callbacks and again
you can find out detailed information
about that call
those events in the speech synthesis
documentation for carbon on the Apple
Developer website so you also have other
events since you basically say here
synthesizer speak this text and then
your thread continues on you're not
going to be able to listen for an
exception right there that may be
generated down the road so the
synthesizer itself is a problem say you
embedded incorrectly some sync event or
some other embedded command and we'll go
into embedded commands later you can
listen for error events you can also
listen for exactly when the synthesizer
begins speaking when it finishes
speaking whether that is someone stopped
it some other API just called stop stop
talking or it concluded itself it read
all the text and that would be the done
event so let's do a speech synthesis
demo and this demo is packaged in the
speech synthesis SDK and let's launch it
here this is just a normal Java app
wrapped with mrj app builder and it's
built this is all all Java with a swing
aqua look and feel and it's built to not
only show not only be a little fun but
be educational because I'm assuming that
Java programmers that want to do speech
synthesis don't necessarily know all the
capabilities of the synthesis system
underneath on Mac os10 so it's fun to
play with we'll go through and we'll
just ask it to speak something we
basically have a voice here agnus and we
can see her age 35 about female etc the
rate that she speaks that the pitch base
and pitch modulation the volume that she
speaks at and we'll just ask her to
speak isn't a nice to have a computer
will talk to you and maybe we want her
to speak a little faster it'd be nice to
have a computer that'll talk to you so
you can play around with these and see
the different voices we'll go through
what's it what's some more interesting
ones
do that act on all these strings by the
way are into the system they for the
particular voices you choose there
particularly they particularly exemplify
the type of voice that you have selected
and we'll look at one more let's look at
some really aliens that looks like a
peaceful planet so different voices it's
pretty interesting and this again is is
is just the Java framework now another
thing that you can do with speech
synthesis is embed commands so let's say
for example we have many commands that
you can embed you and embedded command
basically is just a bracketed keyword in
the text that's how you send speech
synthesis and Betty commands that's not
part of the Java speech framework that
is actually part of the underlying
speech synthesis technology on OS 10 and
you can do many things you can change
the emphasis of a particular word if you
really want to hit it heavily
as the computer speaks it you can change
a number mode for example and this is
interesting so let's let's go play with
this for a minute speech synthesis is it
is pretty intelligent it tries to guess
how it should pronounce numbers but
sometimes it it you need it to pronounce
it single digit by digit other times you
need it to read it as if it was a whole
number say 502 instead of 502 but let's
type something in let's say please call
me at 5 5 5 - 1 2 3 4 at extension 1 2 3
4 5 6 7 8 5 6
I mean okay so first we'll listen to the
computer speak to snooze call me at 5 5
5 1 2 3 4 at extension 5678 so there was
a little bit of interesting hiccup in
the highlighting there but it pronounced
the first number correctly it recognized
as a phone number so it read it
individually but an extension number you
wouldn't normally tell someone to call
you at 5,000
seventy-eight you'd say extension five
six seven eight so will embed here the
number command to ask it to speak this
literally and then once we're done at
the other end of that at the number so
returns to a normal number mode will
insert the normal command again will ask
it please call me at five five five one
two three four at extension five six
seven eight so you can do things like
that so it's a little more intelligent
about what it's pronouncing another
thing that you can do is among all these
embedded commands is ask for a sync
event as I mentioned a few slides
earlier let's say you're using speech
synthesis to do a presentation or
something and after a certain sentence
or a certain word certain paragraph you
want to present a graphic on the screen
and then you want it to continue on with
what it's doing or you want to sync in
some other way a UI with with the speech
so let's type in a sentence such as you
can buy product a for five dollars or
product B for ten ten dollars and then
let's enter a sync command and the way
in our sync command this is sort of
legacy OS 7.5 or so OS nine ish of how
they had many four-letter keywords for
different technologies in the OS so in
this case since the roots of this are
still in carbon and in earlier OS is
even though on OS 10 now we basically
need to add a four-letter keyword to our
sync events so that when we get a call
back we'll know exactly which callback
that was for for the sync event so first
all embed this here and then and by the
way this is this is just a convenience
this whole mechanism I'm using to embed
these commands this is just text so I
can actually just
cut and paste here yes and I'll change
this to BCC or something and pay
attention to the embedded command
feedback area here at the bottom of the
window we return by blood-like to four
or five dollars for Block B for ten
dollars so that's a method that you
could use to sync something with the
speech synthesis and before we go on let
me just replay that and pay attention to
the phone um feedback down here phone on
key codes error codes are actually in
the speech synthesis documentation for
carbon on the Apple developers website
but you'll see it roll through all the
different sounds as we speak you attend
by a blood like two four or five dollars
or blood for ten dollars and so again
you could use that to animate properly
the shape of an the mouth of an animated
head for example so that is speech
synthesis that's about all the
capabilities that are in speech
synthesis of course there are variations
not shown in this demo for example you
can pause the speech you can choose to
stop the speech at a word break at a
sentence break immediately I won't have
you insane for pausing but let's
continue on now to java speech
recognition and this is the nightmare
demo because we've already tried it in
here and i feared this before i even got
in this room and the acoustics even with
none of you in here really plays havoc
with this microphone but we'll give it a
shot anyways but first let's just talk
about java speech recognition so a java
speech recognition
it recognizes spoken language contained
in a language model and we'll talk a
little bit more about a language model
later but that could be something for
example like call a meeting or a phone
home that's a simple language model
itself we provide about 70% coverage for
speech recognition when compared to the
carbon speech API
again when architecture neutral as I
said before with speech synthesis so
that you can coordinate the recognition
callbacks with your UI callbacks as
convenience and again we provide
javabeans and customized property editor
for tight integration with visual IDs
such as Jay builder or Mitchell works
the two main classes force java speech
recognition are the recognizer and the
recognizer does about what you'd expect
it allows you to start recognition stop
recognition set a particular language
model
i--listen add your listeners for events
etc the second important class is the
language model there are two different
ways to use the language model class one
is just hey here's an array of different
sentences I want you to listen for tell
me when you get one of them and tell me
which one you got with which one you
recognized another one is more complex
and it really is something that if we
have time I'm going to demonstrate a bit
but it's still in really an alpha stage
for for a couple different reasons
that'll probably be clearer when when we
give that demo the events that the
recognizer can generate are unrecognized
hey you know somebody said something but
I don't really know what they said I it
didn't match up with anything in my
language model that you gave me and you
would use that to do something like ask
the user to speak that phrase again
another event that you can listen force
the detected event that just means
someone has started to speak but the but
the recognizer has not yet completed
recognizing what they said they are not
done saying what they're saying and they
the other event is the done event this
means that the recognizer actually
recognized what they said and it can
tell you exactly what that was so
language model and editor this is an
example of a language model the top
model top l limb is the top language
model and it has two possibilities call
person or schedule meeting now schedule
meeting is literal listen for the word
schedule meeting call person actually is
a concatenation of two other models the
call model third line down and the
person model
Cole model can be Coughlin or dial the
person model could be literally Arlo
Brent Matt or my wife so in this
language model and then also you have
the top language model with person or
view today's schedule variations of the
other two so in this language model you
could possibly receive dial Brent which
would be the first part of the first
language model or schedule meeting or
view today's scheduled or scheduled
meeting with Brent for example so let's
go to the speech recognition demo keep
our fingers crossed and this is the
speech recognition mic here I'm going to
start up speech recognizer and this is a
very simple demo is actually something I
used to control my web browser but right
now I just have it as a simple list
that's going to pick up what I'm saying
so let's wait for it to start okay and
if we can cut this Mike for a second
so in the acoustics of this room some of
the softer sounds I found will hard to
pick up between this microphone and the
fans and all the computers and in the
different space in the room but there we
see the speech recognition working and
by the way we're going to go into sample
code and and actually write some code
take a look at some other code after
we're done with these demos let me quit
this here and can we have the main
presentation board back thank you so the
last framework that I want to take a
look at is the job of spelling framework
the java spelling framework is a set of
java api that allows you to access the
underlying cocoa spelling service and
it's built on the cocoa spelling
services which in turn are built on top
of mac OS 10
it provides three levels of usage three
styles or levels one is just a simple
set of API you can call to ask it is
this particular word spelt correctly
give me a list of suggested Corrections
modify the system dictionary that is I
have a new word it shouldn't be in the
dictionary it spelled correctly it just
isn't add it to the dictionary and that
would be system-wide so then another
cocoa apps such as TextEdit would then
pick up that word in its spell-checking
session another way to integrate is
complete integration with the swing text
components now I don't know if you're
aware but in swing all text components
derived from J Tex component in J text
component has a classic
model-view-controller paradigm design
and that makes it very easy for us to
tie in to all text widgets than in swing
whether it's a one line on stylized text
multi-line on stylized text or a fully
stylized multi-line text and it also has
an abstraction for integrating with non
swing text components now its main
classes are the spelling checker itself
and this just basically has the the
simple static API that you would call if
you wanted to just roll all this
yourself another important class is the
misspelled word class you'll receive
that in callbacks from the spelling
panel which we'll go into later
that will say what word was misspelled
where it was in a group of text how long
it is etc so that you can use that to
mark up your text etc if you're not
using the full-featured second mode of
operation we described earlier which is
full integration with Java text
components on a word class which is a
parent class of misspelled it just
indicates a correctly spelled word so
this is a snapshot of the classic
spelling session these are all swinging
aqua widgets and as we can see we have a
classic spelling panel on top of our
demo and I'll show that in a second and
through this panel you can receive a
bunch of different events but let's see
how we bring that panel up so in the
case of the second usage just go to that
session on a standard Java text
component it takes these two lines of
code and your text component has to be
nothing special
so you just instantiate the j' text
component driver and you ask it to check
the spelling of the text widget and you
can still if we go back you can still
edit the underlying text component if
you wish and you can just close the
spelling panel and manage all of your
dictionaries etc this spelling panel
that's on top now will appear as soon as
we call driver dot spelling check
spelling so another manner of checking
the spelling is with the inline pop-up
in real time spell checking in real time
spell checking you will get cues for
misspelled words which are red dashed
underlines just as you would in any
Cocoa application and by right by
control clicking on the misspelled words
you can get a list of suggested
suggested Corrections and you can add
and delete things from the system
dictionary the Java spelling framework
UI API which is to do with this window
on top allows you to integrate the spell
checking panel with some other text
component there is since swing
didn't come along for a while in javis
life many different text widgets out
there and so you can still use for the
most part the Java spelling framework as
is with with the spelling panel with
your own custom text widget as long as
you do a few things now one is
implementing the J text component driver
a new implementation of driver J text
component drivers that is the swing
default swing implementation and you
would need it implements the interface
below driver and you would need to
implement your own implementation of the
driver interface and then once you did
so you would tie in with the spelling
panel the class above and you would be
able to have this spelling panel tied in
with your text component of course you
would have to handle the highlighting of
the misspelled word you would just get
cues and the cues you would get are
events generated by the spelling panel
one of the cues would be an event called
ignore the user wants to ignore the
currently suggested misspelled word find
next these just basically all apply to
the buttons in that spelling panel
correct the user wants to correct the
misspelling the correct event would have
the choice of the new correct spelling
and it would be your job if you were
implementing this driver yourself to
replace that text another event is found
and it would be your job to highlight a
misspelled word when you receive that
event so let's take a look at this spell
checking demo
and we'll go through three methods of
doing this so the first one is the
standard spell checking session and as
you can see we have a list of possible
suggested corrections to the misspelled
word and it's picked the top one the top
one is always the the most likely to be
correct it's not always though and we'll
see that so in this case I will choose
that word but for the third word I
really want a fourth word I really want
one not the first and I'll correct that
then and this is just your typical
spelling session but the thing to
remember is that this is all Java these
are all Java widgets this is all swing
accessing the underlying cocoa spelling
service will correct that will correct
that etc and since this this particular
integration is that first two line call
using full swing integration with a
spelling service when we have a word
down below that needs to be highlighted
the the text panel scrolled for us etc
we'll replace that word and we'll
replace the last word so that is the
first manner with which you can check
spelling and let's get back our eye
misspellings I start over
the second manner is just to simply
check them all so it checks them all and
again puts a red underscore misspelling
cue under all the misspelled words and
we can right click as we saw on the
presentation is selected the correct
word we'll just we'll just select a few
here so you can do that as well let's
get back a fresh copy and go to the real
time spell checker and while that's
coming up all of these calls are being
made to check the spelling when it
eventually has a word then it wants to
find out is this spelled correctly it's
taken from the world of Java in this
particular process and it's sent via do
I don't know if you know anything about
do do is part of is a distributed
objects mechanism in Cocoa so it's
packaged up in Cocoa sent over the wire
to another
running on Mac os10 the spell server
that spell server looks at the word
finds it in the dictionary or doesn't
and since back response whether it's
spelled correctly or not so all that
happens pretty quickly and to give an
example let's turn on the real-time
spell checker and let's just type
garbage at first let's just type blah
you know who knows what this is but as
you can see as I type it is checking
every single correction I make to the
document and all that it's just
happening extremely rapidly now let's go
back and let's actually do something
look at our classic sentence again so
since the real-time spell checker is
still on we can delete make our
correction manually we know that this is
s for example should be is but we can
also then do as we did for the check all
option which is to ask it to give us a
list of suggested corrections and we can
correct it in this manner so that is a
demo of Java spell checking so where to
get the code you'd go to Apple's Java
developer website it's sometime this
week I'd look by Friday and I'll
definitely be there but well on the
subject to code let's let's see how he
it easy it is just to roll some of these
things our own and let's see a bit about
what I was talking about when I said
Java beans so let's launch jbuilder and
let's close this nice little app again
okay oh go back okay let's start this
guy up again since the demigods don't
like me I think I'll be again to save
incrementally
okay okay let's save all let's go back
to the visual designer
and let's add our speech since this
being again okay and let's set the
layout of this this will be the world's
most horrible GUI do null meaning XY
let's go back to swing buttons okay
that's not a button to start speech
wrecking our speech synthesis another
one to stop it
and let's add a swing scroll panel and
inside of that let's add text widget and
let's put some default text in it okay
and save again and let's go and you know
we won't even bother naming this buttons
let's kill and asked the synthesizer to
begin speaking so let's say this I love
this feature of jbuilder it's just
totally awesome oh this is codeinsight
so especially during these demos is it
really great jacent the sizes should be
down there somewhere did it name it just
synthesizer I think it did okay so the
size R 1 dot speak text and we'll get
that text from this dot J text area 1
dot get
next I can't get any easier than that
well you can write it itself no I'd be
out of a job
okay let's don't add that feature Blake
please so let's go back and add stop
stop stop speech okay and okay let's run
it so obviously this is about the
simplest form of synthesizer but you
could also add callbacks to it but we'll
just take a look at those events on it
so we can speak it this one's neat let's
add watch the stuff and hope that stop
works huh because otherwise it's gonna
repeat this okay and ask it to speak
this was neat okay that's good this was
neat this was neat okay that's good it
worked not actually near an orphan thank
you so so that is speech synthesis so
while we're in this demo let's add one
more button to the beauty of jbuilder
and let's add our spell checking button
spell checking beam
one more time this J this J text yeah
okay compile it again I'll fast it
compiles and let's type something more
interesting in let's see try to think of
something new okay that's right and
let's bring up the spell checking panel
that one line of text that we added and
let's see if it has a guest for me here
and yes it does okay so we'll do that
and correct that spelling doesn't have
one for there so I'll just change it
myself there it does have one for that
for the heck of it let's just say ignore
that one and let's also say that for
whatever reason we want that word to be
declared as spelled correctly let's
learn it and continue on and and then
just kill our session here and then we
could just continue on or whatever now
let's open up let's quit that so there
are two examples of easily incorporating
the speech synthesis and spell checking
framework into your application if we
look again we need to look at the top
this application because J builder did
some things for us it instantiated the
jate X component driver when we added it
the to the application visually it also
do the same thing for the synthesizer
class as well and then when we look it
down at our code below it just took one
line of code to ask the synthesizer to
speak the text another line of code to
ask you to stop it
one line of code to ask it to check the
spelling and and protecting a spelling
they're also checking real-time and
check all API as well so any way you
slice it you have a couple lines of code
that only would work on Mac OS 10
but then again since there are only a
couple of lines of code and because of
the types of frameworks they are for
example spell checking you might want to
consider integrating them into your
application java applications that you
want to run on all platforms the way
there are several ways you could do this
but the benefit is that it would
basically run the same on all the
platforms it's just that when you got to
Mac OS 10 and users users weigh in it
there they'd also have for example spell
checking so you might use since it's
only a couple lines of code for spell
checking reflection to call this you
might also wrap it in another API build
it on Mac OS 10 or building on a
platform that has these same API stubbed
out so you could compile and then run it
on all of your systems and and simply
win it when it ran on Mac OS 10 it would
have the spell checking capabilities
built into it and same for basic speech
synthesis although you know that may be
more involved in your application when
you do speech synthesis so those are the
first two easy frameworks to demo now
the bleeding-edge guy is speech
recognition so let's actually go and
open up a new file open up a file that
was put on our system when we installed
the speech framework okay let's see here
dudududu where am i pilot
was that okay Wooper
and research and java speech by mark
samples speech okay let's look at
example one this is the sample code to
the speech recognition sample we saw and
for the most part what it's doing is
instantiating the recognizer and
creating an stanching and language model
setting the language mode on the
recognizer and in this case using the
simplest manner of setting the data in
the language model that is just a array
of strings so and then it asks the
recognizer just start now the code we
don't see here is that there's a utility
class built into the java speech
recognition framework called the
feedback panel and that was that set of
strings i'm not going to go through the
whole thing actually we don't need to go
that machine I can look I can launch it
up here that was the set of strings that
you saw like the the list of strings in
the small sample and we'll just bring it
up just to associate again so everything
in this panel is the feedback panel and
you can use that as well you might want
to cue your user into what they can
speak and we've just wrapped that in a
Java frame and then we have told the the
feedback frame what recognizer it should
attach to and it's gone and attached to
all the listeners that especially the
done event listener so that can
highlight the correctly spelled
correctly a spoken phrase now the other
thing that I wanted to show you that is
the real bleeding edge stuff let me
close this file
let's go back to G butter
example one let's go back to the
designer and let's go and just add a
recognition being to this a little
sample application and take a look at
the rough early version of the language
model customizer so the thing that
actually makes this rough is that
drag-and-drop
probably if you're familiar with Java at
all you know is still in a rough state
of affairs not only do we have some
books on our site but there are just
some bugs and the shared code that all
the VM implementers share with Sun and
I'd basically discovered those specific
bugs while doing this editor and we need
to get those changes fed back to Sun get
them into the pipeline so everything
everyone can benefit from them but I
will try to struggle my way through this
to give you a bit of an example of how
it works
so I'm going to try to recreate that
first language model we had on the
slides when we had a call person for
example scheduled meeting with person so
let's say for first of all that we just
wanted to say scheduled meeting with
person so we could say we could add a
new language model down here oops I need
to add anyone first called person and we
could say this is let's see we could say
that this is Dan Mickey
Vincent mind not in the room and then we
could say we could drag this person now
this is part of this feedback windows
one of the one of the bugs of that I was
referencing before so we could type a
peer call person
and then I should be able to say call
Dan
or call Nicky now the actual we had a
bit of a difficulty with setting up the
mic so the development development
environment for this sample is on this
machine I'm sitting on here and the mic
actually has attached to the other
machine so I can't actually show you
this working but trust me it works
dragging an absolutely problem but you
can go into a test mode and build your
language model you could speak it etc go
back to edit mode to edit other things
once you're done you can save it as a
file somewhere and it'll bring up the OS
10 a save dialog box just save it
somewhere and say ok you're done and
then if we go back and we look at the
source J builder because J bloated
here's to even the nitty-gritty of the
bean specification it's gone in and used
the java speech recognition recognizer
being language model being info and
being to allow us our framework to add
source code to the file that you're
editing that's that's pretty neat in
this case the source code we added was
and allow me to just just add this to
two lines so you can see it better the
source code that we added was new
language model set data file for the
particular file that now contains our
language model and then we could listen
for those same phrases called and Cole
Nikki out of that complex language model
more will be added to that so that you
can have easier ways to recognize a more
complex model but that's the state of
the situation with speech recognition at
the moment so speech recognition is
provided as one of those frameworks made
freely available on our website as of
this week the way that I would recommend
you use it for the time being is with
the simple language model that we saw
below which is just setting a simple set
of phrases listening for the recognition
event the done event and then getting
the sentence out of that event to see
what the user has spoken so that's a bit
of code and taking a look at how easy it
is to use the frameworks of course
there's a lot more in the frameworks
than we've shown today there are all
sorts of interesting things and
callbacks and synthesis and recognition
that you can get at we've actually
covered most of the spelling framework
as is so again you can get the code from
the developer website and send me email
if you have any questions
you