Transcript
[ Music ]
[ Applause ]
>> Good evening.
My name is Robby Walker,
tonight I will be joined
by Brandon Newendorp and Corey
Peterson to talk about SiriKit.
First I'm going to start
by talking a little bit
about what SiriKit is, and at
a high level, how it works.
Then Brandon's going to talk
about building an extension
or two to connect your
application to SiriKit.
And then Corey's going to
talk about how to build
and design a great
conversational experience,
which is meaningfully different
than designing a great
graphical user interface.
But first, let's
talk about Siri.
But first, let's
talk about Siri.
Who is Siri?
Siri is your virtual assistant,
always ready to help you,
whether it's by adding
a bit of humor,
or much more frequently, by
helping you get things done,
and alerting you to potential
conflicts proactively.
We're going to talk about an
example app that integrates
with Siri, and when
we built this app,
we built a fake application.
So we wanted to name it
after something fake,
but we kind of felt like it was
really valuable so we wanted
to name it after a very
valuable sounding fake thing.
So we called it Unicorn Chat.
And then because we wanted it
to seem even more valuable,
we gave it a really sweet icon.
[ Applause ]
So you can imagine someone
saying something like, "Hey,
Siri, send a Unicorn
Chat message."
Siri might respond "To whom?"
Then you might say,
"Celestra", which is one
of the Unicorn names of
one of my colleagues.
Then you would -- then Siri
would ask, "What do you want
to say to Celestra, and then
I might say something like,
"Let's add more sparkle
transitions,"
because as everyone knows,
Unicorns love sparkle
transitions.
But there's a lot of ways to
say the same thing to Siri.
Someone else might say
it in just two steps,
saying something
like, "Send a message
to Celestra in Unicorn Chat."
And then of course, Siri
would ask a question
and then the user would fill
in the rest of their intent.
Other users might do the exact
same thing in only one line,
"Unicorn Chat Celestra, let's
add more sparkle transitions."
Somebody else might say an
equivalent concept in Chinese,
which I will not attempt to say.
And other users will be
incredibly polite and verbose,
and might say something
like, "Hey, Siri,
can you please do me a favor
and launch the Unicorn Chat app
to send a text with the
message let's add more
sparkle transitions."
And despite using 27 words
to describe a simple concept,
they actually forgot to
say who they were going
to send the message to.
And then even then, they
would be incredibly verbose
And then even then, they
would be incredibly verbose
and say something like, "My
very best friend Celestra."
So the point is, there
are a lot of ways
to say the same thing to Siri.
And there are also a lot of
ways that Siri will respond.
So for example, if
you're holding your phone,
and you're pressing the
Home button, Siri's going
to provide more visual responses
and say less things out loud.
But if you're invoking
using Hey Siri,
like our friend Cookie Monster,
or if you're in the car,
Siri's going to say
more and show you less.
And the Siri experience is
also carefully optimized
to meet the accessibility
needs of our users and responds
to a lot of different
accessibility settings.
So in the end of the day,
there's basically a
combinatorial explosion of ways
that people use Siri,
both in terms
of how they phrase
something to Siri and in terms
of how Siri will respond.
And the question then becomes,
how do we make sure we're
building a high quality
experience for our users?
How do we make sure it's really
consistent across applications
within a single domain?
And most importantly, for
everyone in this room,
how do we make sure it's
nice and easy to adopt,
how do we make sure it's
nice and easy to adopt,
so that as many people as
possible can use as many
of their favorite applications
in a conversational way?
And the answer is that Siri
handles the conversation
and your application just
handles the functionality
So I'll go into a little more
detail about what that means.
The first thing that Siri does,
is understand what
the user said.
So we take audio and
we turn it into text.
Then we take that
text and we turn it
into a structured representation
of what the user is trying
to do, something
we call an intent.
Based on that intent, Siri then
takes action performing the task
the user wanted to accomplish.
And then Siri provides
responses as I said,
both visual and verbal.
And so there's a lot here.
There's a lot of work that
goes into these four steps.
There's a lot of deep
learning technologies.
There's convolutional
neural nets.
There's LSTMs, there's CRFs,
there's SVMs, there's
a lot here.
And we wanted to make sure
that as app developers,
you could really
focus on the things
that your app already does, and
there's three parts to that.
The first one is vocabulary.
This is how your application
tells Siri words Siri needs
This is how your application
tells Siri words Siri needs
to know about.
And Brandon will tell you
more about it in a bit.
The second one is app logic.
This is the functionality
of your application.
And then the last one is of
course, a great user interface.
So let's go through
this example again,
and see a little bit
about how it works.
So somebody says, "Hey Siri,
please send a Unicorn
Chat message to Celestra."
The first rule that your
application plays is
to help Siri understand
the speech.
So Siri doesn't have
the name Celestra
in its database,
but your app does.
So by sharing that information
with Siri, you make sure
that Siri understands
what the user said.
This also helps Siri
identify the fact
that Celestra is a contact
because when you provide
this kind vocabulary,
you will provide it by saying
these are names of contacts,
or in another example
it might be,
these are names of photo albums.
Now Siri understands that
in the messaging domain,
to send a message you
actually need more than,
of course, just a recipient.
You need a message.
And so Siri will quickly check
with your extension to ask,
"Do you actually require
content for this message?"
And if your application
says yes,
then Siri will ask a
follow up question.
then Siri will ask a
follow up question.
So really that's all your
application has to do is sort
of specify on an individual
parameter basis what are
your constraints.
The next thing that happens
is Siri structures all
that information
up into an object.
So rather than having to
deal with all of the vagaries
of language and phrasing, and
all those things that we talked
about earlier, your application
really just gets this nice
simple object.
And then it hands off to your
application to help with,
you know, processing the intent
and performing the action.
And then of course,
along the way,
Siri provides a user
interface for the user.
And you have the option to
provide a second extension
to customize the user interface
to make it look extra Unicorny.
So that's a very, very high
level of how SiriKit works.
The idea was to make it
incredibly simple to adopt,
but still connect
you to the power
of conversational interactions.
And now to tell you a little bit
more about how that all works,
is my colleague Brandon
Newendorp,
also known to me, as Celestra.
[ Applause ]
>> Thanks Robby, I
am incredibly excited
to tell you all a little bit
more about how SiriKit works,
and how you're going to be
speaking Siri's language
when you adopt SiriKit
in your own applications.
Now pay close attention
because this is some seriously
"intents" stuff.
To get started, I want to talk
about how SiriKit
communicates about the world.
At a high level, Siri
thinks about everything
in the idea of a domain.
Where a domain is a
broad category of things
that Siri understands and
knows how to talk about.
For example, things like
messaging, placing VOIP calls,
and sending payments, are
great examples of domains,
and are some of the
domains that we are opening
up with SiriKit this year.
Within a domain,
we have an intent.
Where an intent describes
a known action
that Siri understands, and can
communicate to your application.
Now I'd like to break down
an intent a little bit more
with another example, in
a little bit more detail
than what Robby was
just talking about.
than what Robby was
just talking about.
So in this case, we have
the Unicorn Pay app.
Hey Siri, send Buttercup
85 bucks
on Unicorn Pay for
rainbow polish.
Now Siri understands
that this is a request
in the payments domain.
And specifically,
we've recognized
that this is a send
payment intent.
Now every intent has
the idea of parameters.
Where parameters are specific
items that Siri can recognize,
understand, and communicate
to your application.
And there's a lot of parameters
on the send message intent.
We've recognized that
the app is Unicorn Pay,
and that the user is
trying to send the money
to someone named Buttercup.
We've also recognized an amount
of 85, and then Siri is going
to be very smart, and recognize
that when the user said "bucks,"
they probably meant US Dollars.
And Siri can use a lot
of powerful capabilities
to understand what we think the
user means in a case like this,
basing it on things like
the user's location,
their Siri locale, and
other information available
their Siri locale, and
other information available
to our system.
And finally, we recognize
that the user is asking
to send a payment
for rainbow polish.
So there's this common language
that we've created for Sirikit,
of intents that describe
actions and intent responses
that are used to
describe results coming
from your application
back to Siri.
So let's talk a little
bit more about an intent.
As I said, an intent
is an action
that we're asking your app
to perform, and something
that we can communicate
about between Apple
and your application.
Every intent will have zero to
many parameters, and it's going
to vary based on the
individual intent and domain,
and what Siri understands
within that domain,
and what we think is relevant to
communicate to your application.
And no matter what, an intent
will always be classified
into a domain.
So this is something
you want to think
about when you're getting
started with adopting SiriKit
in your application, is look
at the domains that are a part
in your application, is look
at the domains that are a part
of SiriKit, and think
about which ones are going
to make the most sense
for your application.
Now we also have
intent responses.
And these are how you tell Siri
about the results of the intent
that your app just performed.
You're always going
to use one of these
to communicate the result
because there's always an intent
response paired with its intent.
Intent responses also can have
parameters attached to them,
which are used to describe
what your app actually did
with that intent.
Sometimes those parameters are
going to be the same as the one
on the intent, and other
times they'll be different
because there's something
that's only communicated
from your app back to Siri.
We also accept a response code
attached to an intent response.
These are used to tell
Siri at a broad level
about whether your app was able
to complete the request or not,
and there's default ones,
such as success, or failure,
or failure that requires
an app launch.
Some intents and domains have
more specific error codes.
It's very important that you
tell Siri an accurate response
code back for your application
because we're going to use this
to drive dialogue and make
a better user experience
for all of your users.
Finally, we have an
NSUserActivity attached
to an intent response.
Now if you're already supporting
hand off in your application,
you're familiar with
NSUserActivity.
We use NSUserActivities
to launch your application
if the user needs to leave
the Siri experience and move
onto your application.
Now by default, Siri is going to
generate our own user activity
with the intent and
intent response to give
to your application, so
that you can pick up exactly
where you left off from Siri.
If you'd like to, you can
provide your own user activity,
and populate the
user info dictionary
with information that's
unique to your application.
Just to make sure you create
the best possible experience
when users leave Siri and
move on to your application.
So those are intents,
and intent responses.
They're the fundamental
language that Siri uses to talk
They're the fundamental
language that Siri uses to talk
to your application
and understand what your
application was able to perform
on behalf of the user.
The next thing I'd like to
talk about, is how your app
and Siri are going to
communicate about intents.
Now there's three steps
that we're going to ask you
to go through, while
communicating about intents.
The first of those is resolve.
We're going to ask your app
to resolve every
parameter on an intent.
And in fact, we will call
resolve multiple times
for every parameter and
sometimes multiple times
for the same parameter
until you and the user agree
on what they requested.
The second step is confirm,
which is a chance for your app
to make sure that it's going to
be able to handle the intent,
and tell us about
the expected outcome.
And finally, most
importantly, is the handle step,
where you perform the specific
action the user asked you to do.
Let's talk a little bit more
about the resolve stages first.
The purpose of the
resolve stage,
The purpose of the
resolve stage,
is for you to help Siri
understand what the user said,
and more importantly,
what the user meant
when they started asking a
request of your application.
This is also your
app's big opportunity
to influence Siri's behavior
and have some control
over the conversation
that we're having
with the user of
your application.
In order to do that,
your app is going
to provide a resolution
response.
And there's a number of
different resolution responses
for every type of data that
Siri understands, and is going
to ask your app to
help us understand.
So let's go back to our example
with sending Buttercup
some money on Unicorn Pay
for that rainbow polish.
And specifically, I'd
like to go pay attention
to the payee field.
This is used to describe
who we are sending money to.
And it's important that
Siri, the user, and your app,
all agree on what the user
meant when they said Buttercup.
Now hopefully, the most
common resolution response
from your app will be success.
Which means that you understood
exactly what the user said,
what they specified, and
then you're going to tell us
that you successfully understood
that and provide the object back
to us to tell us
what you resolved to,
what you successfully matched.
In this case, it means
that Unicorn Pay totally
understood who Buttercup was.
Now sometimes, your app is going
to make some smart choices,
but it wants to make sure
that the user really meant what
you think you'd made a smart
choice about.
So we have a confirmation
required option.
This is where you can ask Siri
to have the user double
check your resolution result,
and provide a yes
or a no response.
So in this case, for Unicorn
Pay, Siri's just making sure
that the user meant
Sir Buttercup
when they said Buttercup.
Now there's other times that
your app is going to need
to ask the user to pick from
a short list of options.
And Siri calls this
disambiguation.
If you'd like to ask the user
to pick from a list of options,
we recommend five
to ten at most.
we recommend five
to ten at most.
You can provide a
disambiguation list to Siri.
And then we'll present the
options you told us about.
In the case of this Unicorn Pay
example, we're asking the user
if they meant Sir Buttercup,
Buttercup the Bold,
or Sunrise Buttercup?
Now one of the powerful parts
of Siri, is that we're going
to take care of the interaction
with the user on behalf of you.
You don't have to
worry about this part.
So users can use a
natural way to select one
of these options with Siri.
They could tap on an item in the
list, they could speak the name
and say "Sir Buttercup", or
they could just refer to one
and simply say, "the
second one."
And no matter how the user
wants to interact with Siri,
we're going to take care of that
for your application for free.
Now there's other times that
you need more information
from the user.
Maybe they weren't
specific enough,
or maybe your application has
80 people named Buttercup,
and you don't want
to present 80 options
of Buttercup to the user.
In these cases, you can say
that you need more details,
and Siri will prompt the
user to be more specific.
Once the user provides
more input,
we'll call the resolve
method again.
If the user didn't specify a
value, and you need one in order
to proceed, you can tell
Siri that you need a value.
And in this case, we'll
tell the user, "Hey,
you've got to say something
for this because, you know,
Unicorn Pay needs to know
who you're sending money to."
You can also tell us that you
don't understand the input the
user provided, and that you
need a completely new value.
And for that we have
unsupported with reason.
And just tell us, you don't
support this particular value,
and Siri will ask the
user for a new one.
And finally, if there's a value
that if a parameter is given,
and your app doesn't require it,
or need it, just tell Siri that.
We won't prompt the user for
it, and if the user happens
to specify a value,
we'll simply ignore them.
So those are the resolve steps.
It's how we make sure that
Siri, your app, and the user,
all agree on what they're
asking your application
to do before we move on.
The next step is confirm.
So the confirm stage is
really about two things.
First, it's a chance
for you to tell Siri
about the expected result
of handling an intent.
We will want you to
provide this response
that we can tell the user
what's going to happen.
It's also an opportunity
for your app
to check any required state
that it might need in order
to complete the request
that the user is asking for.
Now no matter what, you're going
to provide an intent response
back from the confirm step,
and we want you to populate as
much information as you can,
in that intent response,
so that we can communicate
that to the user
very effectively.
And sometimes, Siri is going to
prompt the user for confirmation
for certain intents and certain
domains where we've decided
that that's important.
For example, things like sending
payments, or requesting rides,
are things that Siri's
going to confirm
with the user before we
move onto the next step.
So let's think about what
we're going to confirm
So let's think about what
we're going to confirm
in the Unicorn Pay example.
Now Unicorn Pay has to be
able to talk to our service,
make sure the user signed in,
and a number of other things
in order to complete a request.
So we're going to check all
of those in the confirm step.
We're going to say, "Can we
reach the Unicorn Pay servers?
Is the user signed
in to Unicorn Pay?"
And of course, this is
expensive rainbow polish,
so we want to make sure
the user has enough money
in their account before
we send this payment.
Once we've made sure
that our app is able
to handle this request, we
generate an intent response
with a success result code,
and then pass it back to Siri
so we can move onto
the final step.
And that's the confirm stage
of dealing with an intent.
The final step is handle.
And this is the most important
part of working with an intent,
because at this point,
you're going to do the thing
that the user asked you to do.
You're going to perform the
action on behalf of the user.
And just like in the
confirm step, it's important
that you provide
as much information
as you have available back
to Siri, because we want all
as you have available back
to Siri, because we want all
of that information so that we
can create the best possible
user experience for all
of your app's users.
Sometimes we're going to display
this information visually,
and other times we're
going to use it as part
of the conversation and
speak it back to the user,
so that they understand
what your app did.
Now we understand that sometimes
your app might need a little bit
of time to complete a
request on behalf of the user.
Networking calls might take a
while, or maybe there's a human
on the other end of your
service that's, you know,
processing transactions
or something.
While we're waiting for a
response back from your app,
were going to show a
waiting UI to the user,
and say that your app
is currently handling
their request.
We really would like you
to provide a response
within a few seconds, two
to three seconds at most.
And ideally, you can get
back to us with a success
in a very short amount of time.
However, if you know
that your app is going
to take a little while to get
back to the user with a result,
we'd like you to send an
in progress response code.
we'd like you to send an
in progress response code.
This allows Siri
to tell the user
that your app has received the
request with as much information
as you've told us about, and
that it's going to continue
to process the request
even if they leave Siri.
So in the case of our
Unicorn Pay example,
as soon as we get the handle
step, we're going to talk
to our servers, we're
going to send the payment,
and then give Siri an intent
response with a success code
so that Siri can tell the
user their money is safely
on the way.
And that's what involved
in how we talk
to your application
with SiriKit.
We have intents,
and intent responses
that are creating
this common language
between your app and Siri.
The next thing we're going to
talk about is what's involved
in adding SiriKit
to your application.
So with SiriKit in iOS 10,
we're creating two
new extension points.
An intents extension, and
an intents UI extension.
I told you this was
some intents stuff.
I told you this was
some intents stuff.
We also are adding some API
to help you help Siri recognize
what the user is saying
for your application.
And we're going that in form
of what we call app
vocabulary, and user vocabulary.
I'd like to get started
by talking
about the intents extension.
This is what's going to
implement the resolve, confirm,
and handle methods that we
were just talking about.
Now, at its core, the intents
extension is the most important
fundamental part of SiriKit.
This is what we build all
of our interactions off
of between Siri and
your application.
Your intents extension can
support one or more intents,
and we leave it up to you to
decide what makes the most sense
for your app and how you'd like
to implement your extensions.
Now we encourage you to minimize
the number of extensions
so that we aren't
launching as many processes
if the user makes multiple
requests to your application.
But do what makes sense for you.
It's also important to realize,
that your intent
extension is going to run
that your intent
extension is going to run
in the background while
Siri is in the foreground.
This allows Siri to maintain
a conversation with the user,
and still talk to your app
to get data back and forth
and communicate about
the user's request.
And as I mentioned, your intent
extension is what's implementing
the resolve, confirm, and
handle methods for every intent
that your app supports.
Now when we built SiriKit, we
put a lot of time into thinking
about security, because
security is fundamental
to everything we do at Apple.
Part of Siri's built in
behavior is that we're going
to restrict certain behaviors
from being taken while
the device is locked.
For example, we wanted to
make sure users authenticate
into their device and unlock
it before they send a payment
or request a ridesharing
vehicle.
And that's a built in behavior
we have for certain intents.
Now we also understand that
there are some behaviors
that your app would like to
increase that security for.
By default, Siri's
going to allow things
By default, Siri's
going to allow things
like sending messages
from the lock screen,
but if your app would like to
ensure that the user has logged
into their device first, you
can tell Siri that you would
like to increase our
default security,
and restrict your
intent while locked.
And this is simply done
through an info plist key.
We've also worked to make sure
that the local authentication
framework is available
for your intent extension,
so that you can ask the user
for additional authentication,
even while the device
is already unlocked.
If you're not familiar with the
local authentication framework,
there's a great talk from 2014
that goes into more detail.
And finally, we've made sure
that Apple Pay is available
for your intent extension.
So if you'd like to complete
a transaction using Apple Pay
with your users,
that's available
to your intent extension and
the Apple Pay sheet will appear
right above Siri.
If you'd like to
learn more about this,
I'm glad someone
uses Apply Pay here.
I'm excited about it.
You can learn more in the
What's New with Wallet
and Apple Pay talk that
just took place yesterday.
So that's your intent extension.
This is a fundamental part of
SiriKit and it's how we're going
to communicate with your app
to complete the users request.
The next piece is your
intents UI extension.
Let's go into a little
bit more detail
about what that's all about.
So the intents UI intent
extension is a chance
to bring your app's interface
into the Siri experience,
and make Siri really feel
like your users are
using your application.
You can bring your brand,
your visual identity,
as well as other information
into the Siri transcript.
And you do that by
adding these views
to what we call a Siri snippet.
Your intents UI extension
is always going
to return a UI view controller.
And you can add anything you'd
like inside that
view controller.
Now it's completely optional to
have an intents UI extension.
By default, Siri has an
interface for every intent
that we support in SiriKit.
And we think that those will
effectively communicate all
of the pieces of information
your user wants to know about,
of the pieces of information
your user wants to know about,
but if you'd like to make a more
unique and branded experience,
this is your opportunity.
We've also made intents
UI extensions available
for certain intents.
Where we think it makes sense to
show an interface to the user.
So there's a number
of things you can do
with an intents UI extension.
You've used it to show
additional information that goes
above and beyond
what's described
in the intent and
intent response.
Maybe some information that's
unique to your application.
You can also use it to update
the user with the status
of a long running intent.
So if you've provided an
in progress response code,
as we mentioned earlier,
you could continue
to update the user about
the status of that intent
in your intents UI extension.
Now no matter what,
we're always going
to show your intents UI
extension alongside other
Siri content.
So we're always going to
show what your app icon
and app name at the
top in a sash.
We may choose to show some
buttons and other information
at the bottom of the snippet,
as well as other
pieces of information.
as well as other
pieces of information.
Your view controller is always
part of our Siri snippet.
Now there's a couple of
cases where it makes sense
for your app to replace
the content
that we're showing
in Siri by default.
And this is things like
messages, or maps, where we felt
that you should have
the opportunity
to replace the default messaging
or maps experience with the one
that you feel suits your
app and your users best.
So you can tell us that your
displaying a map or a message
with your intents UI extension,
and then we will omit certain
Siri interface elements
in favor of yours.
Now it's also important
to keep in mind,
that just like the new
notifications extensions,
intent UI extensions
are non-interactive.
So you can draw anything you'd
like, you can put animations,
you can play any kind
of views you would like,
but we would discourage
you from putting controls
in your UI extension,
because users will not be able
to interact with them.
So that's the intents
UI extension.
Together these two extension
points are everything you need
Together these two extension
points are everything you need
to add your app's
capabilities to Siri,
and bring your experience and
your user interface inside Siri
to make a better experience
for all of your users.
The next two parts of SiriKit,
are all part of your
application.
So let's talk about what
those all involve next.
Now, all of the responsibilities
for your application lie
in the area of vocabulary.
All of your applications
have unique ways of talking
to your customers, and that
your customers use to refer
to features of your application.
And Siri needs a
little bit of your help
in understanding the things
that users are going to say
to your app, and the things
that they're accustomed
to talking about.
Some of these phrases are
part of your application.
They're things that every user
of your app knows
and understands.
Other phrases are going to be
unique to individual users,
and we have the API to
support both of these as part
of SiriKit, and both
are very important
to making a great experience.
Let's dig into app
specific vocabulary first.
These are words or phrases that
are part of your application,
and they're known to every
user of your application.
They're things like workout
names or vehicle types
that everyone uses to refer to
certain features of your app.
Now because they're part of your
application, you define these
in a plist that's part
of your main app bundle.
And also, because they're
part of your main app bundle,
we encourage you
to localize these,
so that they can be
supported for every locale
that your app supports,
and every locale
that Siri's going to be using.
Let's talk about what goes
into that vocabulary plist,
because there's a number of
parts to a vocabulary plist.
The first of them, are
examples of how users are going
to talk to your application.
These are kind of
what we call, like,
your hero use cases
for your app.
In the case of our new Unicorn
Fit app, it'd be an example
like start a workout
with Unicorn Fit.
The common things that
you think users will say
to talk to your app.
Now in addition to that,
there are certain parameters
where you can provide
your own vocabulary types.
For the Unicorn Fit application,
we're going to specify our
own unique workout names.
Some things that are unique
and unique to Unicorn Fit,
but known to all of our users.
Now I'm sure you all are very
avid users of Unicorn Fit,
but if you're not, now
we're going to learn
about the term Colorlicious.
It's a particular workout
name, and it describes a thing
where you run around
and you catch bits
of falling glitter
as fast as possible.
It's a great workout.
You should all give it a try.
It's a new craze.
So we're going to tell Siri
about the phrase "Colorlicious."
It's a unique term for our app.
Now, sometimes these phrases
are not the easiest things
to pronounce.
They don't look very obvious.
So you're also going
to give Siri some
pronunciation information.
And this is just a simple
phonetics spelling of the word
or phrase that you're
telling Siri about.
And finally, provide some
examples of how you're going
to use that phrase in your app.
"Do a Colorlicious in Unicorn
Fit", would be an example
of this particular
vocabulary entry.
The other type of vocabulary,
is user specific vocabulary.
These are words or
phrases that are unique
to individual users
of your application.
And because they're unique to
individual users of your app,
we're asking you to
provide them at runtime
from your main application.
We ask you to provide
these in the form
of an ordered set of terms.
Now you might be thinking,
and ordered set is kind
of an odd data structure choice.
Why would we be choosing
to use that?
Well I promise you there's
a really good reason for it.
We really, we really want you --
we'd encourage you to order
the vocabulary in a priority
for your users, with the most
important terms or the things
that you think users will
say the most up front.
Let's talk about that in
the context of Unicorn Pay,
where we have an address book
of everyone we can
send payments to.
Now we want you to give Siri
the names of all of the contacts
Now we want you to give Siri
the names of all of the contacts
in your application, so that
if the user says those names,
we'll be able to recognize them.
However, Unicorn Pay
keeps track of the people
that we've paid most recently.
So we're going to put
our recent contacts ahead
of all the other contacts
in our application.
In addition to that, Unicorn
Pay has a set of favorites,
which are the most
important people to the user.
So we're going to
prioritize those first,
and start off our ordered
set with our favorites,
followed by our recent
contacts, and then wrapping it
up with the other
contacts at the end.
And by prioritizing in this way,
your helping Siri better
understand your users
and make a better
experience for everyone
who uses your app with Siri.
There's a few types of
things that we support
with user vocabulary, and those
are things like contact names,
workout names, photo
albums or photo keywords.
And as an added benefit
to your application,
if you rely on the system iOS
address book, you don't need
to worry about this part
at all for contact names,
to worry about this part
at all for contact names,
because Siri has a
built in understanding
of the system address book.
So you only need to
provide contact names
if your app keeps its own
dedicated address book.
Now, you also should think about
what information you're giving
to Siri as part of this
vocabulary mechanism,
and only tell us
about information
that you your users
would expect Siri
to recognize and know about.
So we don't want you to give
us things like phone numbers,
or e-mail addresses in
your vocabulary data.
Just give us simple common
things like your contact names,
the things that users would
expect to see come from Siri.
Now it's important that
you update Siri promptly
as data changes, because users
are accustomed to saying things
to Siri the instant they
put them into their phone.
So for example, if the
user creates a new contact
in your app, make sure you
tell Siri about that as soon
as they hit the save button.
That way we can ingest the data,
and be ready to recognize it
as soon as users hold down the
home button and talk to Siri.
It's even more important
that you delete information
from your Siri vocabulary
promptly,
because your users
would be very unhappy
if they deleted something
from your application,
and then Siri was still
recognizing it a few days
or a few weeks later, because
you forgot to tell Siri
that the user deleted that
particular piece of information.
And finally, make sure
that you remove all
of your vocabulary data from
Siri if the user chooses
to reset your application,
or log out.
Now the other piece of SiriKit
has to do with permissions.
So like many other parts of iOS,
users have to approve
authorization
for your app to talk to Siri.
And we do this through
the standard TCC alerts
that you're used to
seeing for things
like the camera,
or the microphone.
Now your app is able to
provide a string that goes
into this alert to
tell users what kind
of information your app is
going to be sending to Siri.
This is also a fantastic
opportunity to teach your users
about what your app does,
and what cool new features
you've just created
for all of them.
We also are encouraging
you to request permission
from your app the first
time a user uses it
after they upgrade to iOS 10.
We want you to do that so
that users know about Siri
and have approved the use
of Siri before they make
their first request.
So as soon as they try talking
to Siri, and your application,
it's already approved, and
they have a fantastic first
launch experience.
So that's what's involved
in adding SiriKit
to your application.
These two pieces of extensions,
these two new extensions,
and two vocabulary mechanisms.
The last thing I would like to
talk about is how SiriKit fits
into the larger world of iOS.
Now Siri is an integral part
of our operating system.
It's always accessible to
the user via the home button,
via hey Siri, via
headset microphones
and many other ways
of triggering Siri.
And SiriKit is just as integral
to the operating
system as Siri itself.
to the operating
system as Siri itself.
Now Siri integration is part of
this larger intense framework
that we've created
this year, which is all
about communicating these common
and shared actions and intents
between your app and iOS.
And the intense framework
is going beyond just Siri
integration this year.
It's part of the
contacts integration,
where you can donate
information to the system,
and elevate your application
to be the default messaging
or telephoning mechanism
for certain contacts.
We've also done very deep
integration with CallKit,
to make a great VOIP experience.
The most, the biggest
piece of this is with maps.
Maps is adding ridesharing
to iOS 10,
where you can book a ride right
within the maps application.
Now we worked very
closely with the maps team
to create a common
extension point.
This intense extension and
ridesharing intents are used
for Siri, and maps, to
create the same experience.
for Siri, and maps, to
create the same experience.
So if you adopt SiriKit and the
intents extensions, you're going
to get the maps integration
for free.
We think this is a great
way to save you some time,
and make an even better
developer and user experience.
[ Applause ]
The last part I'd
like to talk about,
is how users talk
to your application.
Now it's very important
as part of our security
and privacy model of iOS
and making your customers
comfortable with your app
and your data, it's important
that they understand when a --
they're going to be talking
to your specific application
and when their data
is going to your app.
To make sure that all
of your users know that,
users are required to
say the name of your app
when making a request to Siri.
Specifically, we're
going to be looking
at your bundle display name.
This is what's shown
on the home screen,
underneath your app
icon, to your users.
Now Siri's all about
natural language
Now Siri's all about
natural language
and having a comfortable fluid
conversation with the user.
So to make sure that that's
part of how this works,
users are able to say your app
name in many parts of speech,
at the beginning, or the end
of an utterance, in the middle,
wherever it fits in
naturally, because Siri wants
to make a natural fluid
experience for talking
to your app, and to your users.
One of the common
things that users do
with SiriKit is what we
call verbing the app name,
or taking your app name
and using it as a verb
to describe a particular action.
As an example of
that, they might say,
"Unicorn Chat Celestra order
five rainbow rolls please."
And in this case, they're
using Unicorn Chat as the verb,
describing the action
they're asking us to take.
And this will, of course, work
fantastically well with Siri.
In fact, there's a multitude
of ways that users can talk
to your app using Siri.
The possibilities are endless.
So that's what's involved
in implementing a great Siri
experience for your users.
But there's more than just
engineering implantation
But there's more than just
engineering implantation
involved in creating a
good Siri experience.
And to tell you more
about designing a
great Siri experience,
I'd like to invite Buttercup,
my co-worker Corey, on stage.
[ Applause ]
>> Thank you.
Hello everyone.
I'm Corey Peterson, and
I'm a designer on Siri,
and I'm very excited
to talk to you
about creating magical
experiences with SiriKit.
I've got three things I
want to talk about today.
The first is some of the
things that Siri does for you
to help facilitate the
conversation with your users.
Next, we'll go through
some design guidelines
that you should keep in mind as
you start to work with SiriKit.
And then, I'll give a few
tips of what you can do
to polish the design, and take
your app to the next level.
So, let's get started
with user interface.
So, let's get started
with user interface.
Now when I say user interface,
you're probably thinking
about something like this.
A button. Part of
a visual interface
that appears on screen.
Now visual interfaces have
well understood behaviors
and patterns and
and everyone is pretty
familiar with using those.
It's part of an application,
and that application gives the
context of what you're trying --
the user is trying to do,
the options that
they have available,
and the details about
those elements.
And there's pretty much one
way to interact with a button.
But, as you start to work with
conversation things change,
and one of the biggest
differences is
that button is gone.
Now, it can be -- feel
strange not having information
on a screen.
So how do we solve this?
Because the user could
say whatever they want,
or they might not
even know what to say.
or they might not
even know what to say.
So we like to think of it as
a conversational interface.
The context of what's
happening is established
through the conversation
between the person,
Siri, and now your apps.
Now, without a visual reference,
the user, or the person has
to keep all this
information in their head.
So Siri works hard to prevent --
to present just the
relevant critical information
to get a task done, and
not overwhelm them with all
of the other information.
And, this interface is shaped by
the questions and the responses,
both from the person
and from Siri.
And it's important to think
about how Siri asks
and responds.
And your application
has the ability
to shape this conversation.
And we'll talk about
that a little bit later.
So let's go through an
example of how this can work.
So let's go through an
example of how this can work.
Let's say, someone
has created a message,
and Siri might say
something like,
>> Ready to send it?
>> Now people can say a lot
of different things here.
They might simply say "send it."
Or they might decide not to,
or they might ignore
the question entirely,
and say whatever they want.
And that's okay.
These are actually appropriate
responses in this situation.
And the great thing is, Siri's
going to take care of this
for you, and send the right
information back to your app.
And even better than that,
Siri will handle the thousands
of other ways that people
may say the same thing.
Now earlier, Robby talked
about all the different ways
that people can start a
conversation with Siri,
and I'm going to focus
on a couple of those.
Now everyone's familiar
with picking up your device,
pressing the home button, and
starting the conversation.
pressing the home button, and
starting the conversation.
But, another important mode that
we have is what call hands free,
where you don't have
to physically touch the
device to start talking.
So just by starting your request
with "Hey Siri" you're able
to work from across the room,
or -- or well, across the room,
if your hands are full, or if
you're doing something else.
Let me show you a
couple examples
of how the conversational
interface changes
to adapt to these situations.
So, here, we're going to say
I'm holding onto my phone
and pressing the button.
So I'll say, "Do I
have any reminders?"
>> You have five reminders.
>> So as you heard, Siri
presented a quick answer,
and then gave visual
information on the screen
since I'm holding it and
I can take a look at that.
Now, listen to how this changes
when I activate hands free.
Hey Siri, do I have
any reminders?
>> I found five reminders.
The first one is talk about
accessibility, then get Siri
to say hi to people
in Cupertino,
then go to the next slide.
Do you want me to
read you the rest?
>> No thanks Siri.
So as you heard, Siri
read out that information
and it gave a smaller amount
of information that's
easier for me to understand.
And at the end, Siri asks me
for the next step so that I'm
in control of the conversation
and what happens next.
Now we use these same principles
in other places as well.
In car play, where you need
to keep your eyes on the road.
And it's also very helpful for
people with accessibility needs.
Now Siri is great because you
can ask for it to perform a task
and its able to do
so, and you don't have
to find an application and
navigate through a visual UI
to find an application and
navigate through a visual UI
to get something done,
but Siri does more.
Siri understands when you have
devices connected, such as a set
of headphones, or a
Bluetooth hearing aid.
And it adapts what it --
oh, and in another case,
it can recognize settings
such as voice over,
which is Apple's
technology for reading
out the visual elements
on screen.
And it again, adapts
its behavior
to help these users
complete their tasks.
So, that's a little bit
about how Siri helps you.
Here's a couple things that
you can do to help Siri.
Make sure that you're testing,
as you develop your apps,
with many different requests.
Testing different
situations, and especially try
out things hands free.
And throughout the process,
make sure that your app is
providing information back
to Siri so that Siri can
represent it back to the person.
And that's especially critical
in these hands free situations.
All right, let's move on
to some design guidelines.
And I want you to think of this
as a conversation
with your users.
And as with any important
conversation,
you need to make sure that
you prepare, that you listen,
that you ask questions, and
that you respond to the request.
So, how do we prepare?
The first thing is to recognize
that your users aren't
exactly like you.
Now, what do I mean by that?
You've spent a huge
amount of time developing
and designing your application,
and you have an amount
of knowledge that's greater
than anyone else will ever have
with it, and that's going
to influence how you
would talk to Siri.
So, you need to go out and find
some real people, maybe friends,
So, you need to go out and find
some real people, maybe friends,
significant others, or maybe
even people in marketing
and ask them what
would you say to Siri?
So I'm going to go with
our theme from today,
and talk about an
application Unicorn Rides.
So for this fake
application, we went and talked
to some real users and they
came back with some responses.
Some people might say "Get
a ride", and in this case,
Siri is going to ask the user
which application they want
to use, and then can continue
through the rest of the task.
Some people may provide
more information,
and some people may provide
a whole lot more information,
some of which may
not be possible,
because as we all know,
it's really hard to fit more
than two people on a Pegasus.
Once you have all
these responses,
make sure that your
app is a good listener.
Take all of the information
that the user has provided
and use that, but they
may not provide all
of the information
that you need.
So, you need to find
ways to get it.
One way, is to pick
a good default,
so if I were requesting
a ride, and I didn't say
where to be picked up, Unicorn
Rides could pick a good default
that I probably want to
be picked up right here,
or maybe outside the
building down by the curb.
You can also make educated
guesses based on your knowledge
about the user, but
avoid bad surprises.
You don't want to pick
the most expensive option,
or some other default, or some
other guess that a user is going
to be unhappy with later.
So if you have questions
and you're not sure
if someone would be unhappy,
maybe the better answer is
to ask the question to the user.
But, with this, you need to
make sure you keep it simple.
Nobody wants to play a game of
20 questions when they're trying
to get something done.
So, only ask for the things
that are absolutely necessary.
So, only ask for the things
that are absolutely necessary.
And people may provide
information
that your app can't handle,
so you need to send
those errors back to Siri
so that Siri can
present it to the user,
and try to move forward
with the task.
There are two types of questions
that I want to go discuss more,
and Brandon talked
about them earlier.
The first is confirmation.
And this is great when you
have one item, that you want
to check with the user.
And this could happen because
you made an educated guess,
or if there's an item that
would have really big impact.
So if someone is used to
making small purchases,
or small payments
with your payment app,
and suddenly they make a payment
that's much, much larger,
you may want to confirm
that with them
to make sure it's
what they meant.
The next is disambiguation.
And this is great when
you have multiple options,
And this is great when
you have multiple options,
and you need the
user to pick one.
So this can be used when
you have a small list,
such as the available ride
types at a given time.
Or, if you have a much
larger list, say I was going
to send a message to Eric,
and I have a number of people
in my contacts, I
can filter that down
to just the possible
matches for Eric,
and let the user pick that.
Siri can present this list both
visually, and through speaking.
And the user can respond by
tapping or saying the response.
So before you perform
the task, Siri will check
with your extension, to make
sure everything's ready.
And it wants to get all
the information back
from your extension, so that it
can then present it to the user.
Now, again, Siri will
present this vocally,
Now, again, Siri will
present this vocally,
and it will provide a UI on
screen to show this information.
If the user decides to continue,
if the user decides to continue,
you should go perform
the action, and again,
return those results back to
Siri so that Siri can show those
to the user and tell the user.
And in the same way that
the user made the request,
they started in a conversation
and Siri can reply back
in the conversation as well.
The great thing about that,
is it works everywhere.
It works in car play, it
works hands free, and it works
when somebody's touching
the device.
And Siri will again, provide
a visual UI to help out.
But, if you want,
your application can provide
a custom UI to bring the feel
of your app into Siri.
I want you to exercise restraint
and limit what you show.
I want you to exercise restraint
and limit what you show.
The person is trying to get a
task done, and you don't want
to distract them
with information
that they don't need.
You also need to make sure
that you're designing this UI
for different screen sizes,
from the smallest phone
up to the biggest iPad.
Here's another example
of the custom UI
from our Unicorn
Chat application.
And as Brandon said,
you should use our API
to hide any redundant
information.
As you do so, you
need to make sure
that your custom UI
represents everything visually,
and that you send all the
information back to Siri
so that hands free
requests will still work
when they can't see this UI.
In a few cases, Siri will
open your application
to finish a request,
and we would do
that for a photo search request
where we can show the
results in your app.
And this is great because
your app can provide the
And this is great because
your app can provide the
follow-up actions.
Now anytime a user goes
from Siri to your app,
you don't want to surprise them.
Don't put any interstitial
information or alerts
in the way, and make sure
that you are only performing the
action that they have requested.
Okay, next, let's
do a couple tips
about polishing your design.
So, earlier Brandon talked about
the app specific vocabulary,
and the examples
that you can give.
And people can find that
information by going to Siri
and just asking,
"What can you do?"
Siri will show lots of
different things that it can do,
including your applications, and
the examples that you provide.
So I want to give you
some tips about how
to write good examples.
Make sure that your examples
include your app name,
Make sure that your examples
include your app name,
so that people can get to your
app for your functionality.
Keep your examples short, and
focused on what your app can do.
You should provide multiple
examples for each intent
that you support, and make sure
that you prioritize
both your intents
and the examples
within those intents.
If your app works in multiple
languages and locales,
make sure that you
provide localized examples
in the appropriate places.
And most importantly,
make sure that they work.
A couple things to avoid.
You don't need to address
Siri in your examples,
and you also don't need any
unnecessary pleasantries.
Just focus on the
functionality of your app.
And the last thing I
want to talk about,
is iterating on the experience.
You want to make sure that
you have time to explore
and to experiment in how
you shape the conversation.
and to experiment in how
you shape the conversation.
Make sure that you
talk to real people,
not just what they
would say at first,
but throughout the process
of getting something done.
And again, try it in different
situations, hands free,
or out and about, not
just testing at a desk.
And throughout, make sure
that you are asking,
"Does it feel right?"
If you need to, make a
change, and iterate again,
because we want to
get to a point
where you have a great Siri
experience for your users.
So, let's recap what
we've talked about today.
Robby started us out introducing
SiriKit, bringing the power
of Siri and your applications to
a conversation for your users.
Brandon talked about
creating extensions
and the way they communicate
with Siri through intents
and the way they communicate
with Siri through intents
and intent responses,
and the methods
of resolve, confirm, and handle.
And also that your apps
need to provide vocabulary
of the common things that
people will say to your app.
And then we talked about design.
Introducing the conversational
interaction,
the conversational interface,
and talking about the
options you can choose
as you're developing
your application
that will shape the
conversation for your users.
And last we talked about
iterating, so that we can get
to a Siri experience that
feels effortless, and magical.
We have more information
about this session on the web.
We have some related
sessions this week,
and I want to especially
highlight Extending Your Apps
with SiriKit, which is tomorrow,
where some of our colleagues
will go into more details
where some of our colleagues
will go into more details
about the API and
show you the code
for an example integration
with SiriKit.
So thank you very
much, and we can't wait
to see what you create.
[ Applause ]