Transcript
[ Music ]
[ Applause ]
>> Hi everybody.
My name's Ari and I'm really
excited to be here to tell you
today about some new updates to
Shortcuts in iOS 13.
In iOS 12 we extended SiriKit by
introducing the Shortcuts APIs.
Which let you expose actions
from your apps to Siri.
Shortcut actions exposed from
your apps can be used in three
different ways.
They can be used as suggestions.
They can be used through voice.
And they can be used as part of
multi-step shortcuts.
Now, with suggestions, Siri will
suggest your Shortcuts to the
user as just the right time in
search and on the lock screen to
accelerate things they do with
your apps every day.
With voice, customers can use
any capability of your apps just
by asking Siri.
This is really powerful, because
Shortcuts makes it super easy to
voice enable your app, which
lets people use your app in
totally new ways.
Whether they're on the go with
AirPods, in the car with CarPlay
or at home with HomePod, or just
from across the room with "Hey
Siri".
Lastly, users can build their
own shortcuts in the Shortcuts
app, which include multiple
actions, including those from
your apps.
Now, this year we're talking
Shortcuts APIs even further and
we have some really exciting API
updates to share that take voice
and multi-step shortcuts to the
next level.
So let's dive into what's new.
In iOS 13 the Shortcuts app is
redesigned, and it's built into
iOS, so it's easier than ever
for people to discover and use
Shortcuts.
Now, after we announced
Shortcuts last year, the biggest
request we heard from developers
was to add parameters.
So we're really excited to share
that in iOS 13 we're introducing
support for parameters and it
enabled something we call
conversational shortcuts.
This basically allows users to
have a conversation with your
app and you control the
conversation.
So you can prompt the user for
information you need and respond
dynamically based on the users
input and their current context.
This opens up really totally new
use cases for what people can do
with Siri and apps, and I'm
excited to tell you more about
it in just a second.
But next, we're adding new ways
for users to customize your
shortcut actions through a new
customization UI that acts on
the same parameters as the ones
that you define for the voice
interaction.
You can also enable advanced
scenarios like outputs and
variables.
Lastly, we have an updated Add
to Siri UI, which makes it easy
for people to get started with
Shortcuts inside of your app.
So, let's take a look at what
some of these new capabilities
mean to the voice experience
inside of Siri.
I have a great new Siri
shortcut.
It's called Add to Instacart,
and it adds items to my cart.
So, with conversational
shortcuts, when I say Add to
Instacart, Instacart can ask
what I want to add with custom
dialogue.
And I'll say bananas.
So, there are a few different
options for bananas.
Instacart will ask which
specific one I want by showing a
disambiguation.
I'll stay I want the organic
bananas.
So, if it's contextually
appropriate, the app can ask
additional questions based on
what I've said so far.
In this case, Instacart will ask
how many bananas I want, and
I'll say six.
Lastly, Siri can let me know
that the task is complete with
custom dialogue and custom
visuals from Instacart.
I also have another new
shortcut.
It's from the MLB At Bat app,
and it means that I can just
say, "Hey Siri, watch baseball."
and it will jump me right into
my favorite team's games.
So, the Philly's aren't actually
playing right now, but with
conversational shortcuts the MLB
app can detect that and recover
appropriately.
Letting me know that there isn't
a game on right now but giving
me some other options of things
I might want to watch instead.
So, that's just a glimpse of
what you can do in Siri, and in
a minute we'll dive into the
implementation.
But first, let's talk about some
updates to the Shortcuts app in
iOS 13.
The Shortcuts app is now built
in to iOS.
And you don't need to download
it from the App Store.
It's the one place where users
can manage all of their
shortcuts.
So everything that used to be in
settings in now here also.
In the My Shortcuts tab the user
can see all of the shortcuts
they have set up and they can
run them just by tapping on them
or by saying the name to Siri.
And there's no need to record a
phrase first.
In iOS 13 we also have a
brand-new Automation tab where
people can set up shortcuts that
show up or even run
automatically based on certain
criteria like when they arrive
at a particular location, when
they get in the car, or when
their alarm goes off in the
morning.
So, we all know Shortcuts is
really useful, but it can be
hard to start from a blank
slate.
So users can find hundreds of
pre-built shortcuts to get
started with in the Gallery tab.
And this year, we're updating
the gallery with a new section
called Shortcuts from Your Apps.
This section highlights relevant
actions from third-party apps
which includes your apps.
So this is a really great
opportunity to get your apps
actions in front of the user
even when they're not using your
app.
The shortcut editor has been
updated also with a new natural
language format.
Here's what it looks like in the
editor when the user sets up a
shortcut with the Soup Chef app.
Note how my app has exposed this
description, Order 1 Clam
Chowder.
And the user can tweak those
parameters at any time just by
tapping on one of the buttons.
Like tapping on 1 or Clam
Chowder.
The editor also has a new action
pane which features actions from
third-party apps that you can
drag or tap to add to your
shortcut.
Lastly, your apps actions can
now provide outputs, which means
they can be hooked up to other
actions to create even more
powerful shortcuts.
For example, the Soup Chef app
can share an ETA.
So you can add it to an Add New
Reminder actions to make a
shortcut that adds a reminder to
pick up your soup exactly when
it's ready.
So, that's what's new in the
Shortcuts app.
But we also have some updates to
Add to Siri.
Now, traditionally, voice
functionality can be hard for
users to discover, because they
don't know which features of
your app are available or what
to say in order to use them.
Add to Siri makes this easy
because people are already using
your app.
So you can add the Add to Siri
button in the relevant parts of
your app and it -- the user
actually wants to use it with
their voice.
In iOS 13 we've redesigned the
Add to Siri sheet.
The new UI lets the user set up
shortcuts without having to
record a phrase.
They can just type a phrase or
they can accept one that's
suggested by you.
Afterwards, saying the phrase to
Siri will run the shortcut.
And if you tap on the action
under Do, the user can tweak the
behavior of the shortcut just
like they can from inside the
Shortcuts app.
Now, let's dive into the
implementation.
First, we're going to talk about
how to enable your shortcuts to
be customized by the user and
then we'll cover parameter
resolution and how to use it to
implement a great voice
experience.
Then, we'll talk about how to
deal with parameters that have
dependencies on each other.
And lastly, we'll cover how
dynamic options and outputs can
help take your actions to the
next level in the shortcut
editor.
So, let's start by enabling
Shortcut Customization for an
app that I've been developing.
I'm really, really into soup and
so I've been working on an app
called Soup Chef, which lets
people order soups from their
favorite restaurants.
I want to enable shortcuts that
let the user order a soup and
choose how many soups they want
right from inside of the action.
So, in order to set this up
let's start in the Intent editor
in Xcode.
As you know, an Intent is a
specification of a task that the
user can complete in your app.
In this case, ordering soup.
You should define parameters for
every piece of information that
you need from the user in order
to complete that task.
The set of parameters that you
define here is used for both the
set of questions that Siri will
ask in conversational shortcuts
as well as the set of parameters
that you can edit in the
shortcuts editor.
From each parameter that you add
to your Intent, you can choose
from the list of supported types
that are provided by the system
or you can create an Enum, or
you can even in iOS 13 create
your own custom types.
Once you pick a type, you'll see
some additional settings that
let you customize that parameter
which are specific to that type.
For example, if I pick String,
I'll see some new options for
capitalization and autocorrect
behavior.
For this parameter I'm going to
choose a soup enum type that I
already created.
Now, since I'm updating an
existing parameter, I need to
check the User Facing checkbox
so that this parameter can be
exposed to Siri and the
Shortcuts app.
For new parameters that I add,
this will be checked by default.
Next, I'll need to fill in some
new information for my
parameter.
I'm going to add this Display
Name, because that's used to
show the parameter inside of the
shortcut editor.
Then, because this is an
existing Intent, I also need to
check the Intent is
user-configurable checkbox,
which makes the Intent as a
whole available to both Siri and
the Shortcuts app.
Now, lastly, I need to fill in a
parameter summary.
I want to include the parameters
quantity and soup in the
parameter summary so that
they're really easy for the user
to tweak in the shortcut editor.
So, I'll add a string, order
quantity soup, and I'll let
Xcode turn quantity and soup
into variables for me in Xcode.
So that's it.
That's actually all you have to
do to get your shortcuts to be
exposed like this in the
shortcut editor.
Now that the Intent is
configurable, and it shows up in
the Shortcuts app, it's
ready-to-use parameters.
So, the next thing we need to be
able to do is resolve the
parameters of the Intent.
Let's talk about how to do that.
In iOS 12, when the user invokes
a shortcut through Siri it goes
through two phases.
First Confirm and then Handle.
This year we're adding a third
phase which is called Resolve.
And it should be familiar to you
if you've already worked with
SiriKit APIs in the past.
In the Resolve phase Siri goes
through each parameter that
you've defined on your Intent
and asks your Intent handler to
resolve that parameter.
To decide whether or not to ask
the user a question.
For Soup Chef, Siri will first
call Resolve on the soup
parameter which might ask a
question like, "What soup do you
want?"
And then it will call Resolve on
quantity, which could ask a
question like, "How many soups
do you want?"
And lastly, it will ask it on
store location, which might say,
"Where do you want to order your
soup from?"
Once all the parameters are
resolved, Siri will invoke the
same Confirm and Handle methods
as the ones that we had in iOS
12.
So, let's take a look at what
this means in code.
Because Xcode automatically
generates an Intent handling
protocol for every Intent that
you define.
And you implement that protocol
in your Intent handler.
In iOS 12 Xcode generated these
two methods; Confirm and Handle,
which you would implement in
order to execute your Intent.
But this year, Xcode will also
generate Resolve methods for
each parameter that you've
marked as configurable.
Siri will call the resolve
methods and ask questions in the
order of the parameters that
you've defined in Xcode.
And you can drag and drop the
parameters in Xcode to change
the order of parameter
resolution.
When the Resolve method is
called, the Intent is passed as
input to the Resolve method,
including a value for the
parameter that's currently being
resolved.
So that value can come from a
couple of different places.
The first time the Resolve
method is called, when the
shortcut is being run, the value
is filled in from whatever the
user set on the shortcut when
they created it.
In this example, the soup field
is empty, maybe because the user
sometimes orders clam chowder
and sometimes orders tomato
soup.
And they want to set up a
shortcut that's going to ask
them every time they use it
which one they want.
So, when the Resolve method is
called the soup parameter will
be filled in as empty because
there was nothing filled out in
the shortcut and that's going to
be passed to the Intent that's
passed to Resolve.
The responsibility of the
Resolve method is to provide a
Resolution Result which
determines what Siri will do
next.
Now again, because the field is
empty, the Soup Chef app might
want to ask the user for a value
by passing the Needs Value
Resolution Result.
Passing Needs Value will cause
Siri to ask a question for this
parameter which the user can
respond to with their voice.
So Soup Chef might say, "What
type of soup do you want?"
And the user can respond with
something like clam chowder.
Now, the input that the user
says, clam chowder, will be
filled in to the Intent and the
same Resolve method will be
invoked again with that input
this time.
So the app can decide what to do
next.
In this case, clam chowder is a
valid soup and we have it in
stock.
So the Resolve method should
this time return success which
will cause Siri to move on to
the next parameter in the list.
Now, if the user had set up the
shortcut instead to have the
parameter preset to clam
chowder, because maybe they want
their shortcut to order clam
chowder every time without
asking them, then that would be
passed to Resolve instead of the
empty value and in this case,
the Resolve method can return
success right away and Siri
won't ask a question for that
parameter at all.
So, in the implementation of
your Resolve method, you need to
look at the current value of the
parameter on the Intent.
If there's no value, you might
need to ask the user for a value
by returning Needs Value.
And if the user has just -- has
filled out a value on the
shortcut when they set it up or
if they just said something to
Siri, there might be a value
there already which needs to be
validated and hopefully passed
as success.
So every time your Resolve
method is called, it needs to
supply this Resolution Result.
And I'm now going to hand it
over to Roman to tell you about
how you can provide great
Resolution Results to provide
robust voice experience in Siri.
Roman.
[ Applause ]
>> Thanks Ari.
Hello. When I'm building a voice
experience I want to think about
all of which cases that the user
could say and how to handle them
well.
For example, if the user asks to
order more soups than we have in
stock we should tell them
appropriately.
If the user asks to order soup
from a store that's far away, we
can give them options of stores
that are closer.
On the Resolve step, this is
your opportunity to influence
Siri's behavior and decide
whether or not Siri asks a
question.
And this is where you handle all
which cases of the user input.
In order to do that, you need to
provide the Resolution Result.
There are six Resolution Result
types that you can choose from.
Let's go through each one of
them starting with needsValue.
If the user didn't specify the
value at the shortcut
configuration time and you need
one in order to proceed, you can
tell Siri that you need a value.
You need to provide your own
custom prompt dialogue for each
parameter that you can resolve.
To do that, open the Intent
editor.
Select the parameter that you
would like to specify Siri
dialogue for and then type your
custom prompt in the prompt text
field.
If you return disambiguation
Siri will ask the user to pick
from a list of values.
This is a good idea if the value
in the parameter is ambiguous or
if you just have few possible
things for the user to choose
from.
You can customize the
disambiguation dialogue in the
Intent editor.
Return unsupported tells Siri
that your app doesn't support
the provided value.
For example, if customer tries
to order too many soups we can
respond appropriately by saying
that we don't have enough in
stock.
And after that, Siri will
re-prompt the user.
We can define the error messages
that can be shown to the user
under the Validation Error
section in Xcode.
Xcode automatically providers
default behavior for medium and
maximum values that you specify
in the Intent editor.
Return confirmationRequired will
cause Siri to ask the user to
confirm the parameter value.
Use this if it's unclear whether
or not the user actually wants
this value or you have a strong
guess, but still would like to
confirm the value with the user.
Again, you can customize the
Parameter Confirmation prompt in
the Intent editor.
Return success means that we
have a valid parameter value and
Siri should move on to the next
parameter.
Return notRequired means that
your app doesn't need a value
for this parameter at this time,
so Siri should skip it and move
on to the next one.
The Result methods will be
invoked at the shortcut
execution time in both Siri and
the Shortcuts app.
So you need to make sure you
implementation behaves correctly
in both environments.
So now let's take a look at what
the execution flow looks like.
This is the new Shortcuts app
and here in my Shortcuts app you
can see all of the shortcuts
that I created or added from the
Gallery.
To create a new shortcut, I'm
going to tap on the Create
Shortcut button.
In the shortcut editor I need to
add actions to my shortcut.
To do that, I'm going to tap on
the Add Action button.
I use the Soup Chef daily to
order delicious soups, so here's
already a suggestion for me to
order soup.
All I need to do is just give it
a name.
So here's my shortcut.
All I need to do is just give it
a name and save it to My
Shortcuts.
To do that, I'm going to tap on
the Next button.
Let's call this shortcut Order
Soup.
Now let's try invoking the
shortcut with voice and Siri.
"Hey Siri, order soup."
>> Which soup do you want?
We have three specials today.
Clam chowder, tomato soup, or
chicken noodle soup.
>> Clam chowder.
>> There are two restaurants
near you.
Which one do you want to order
from?
38801 S Sheridan Blvd or 7401 S
Lewis Ave?
>> The first one.
>> Ready to order?
>> Yes, please.
>> Okay, ordering.
Your total is $3.75.
Your clam chowder order will be
ready in 10 minutes.
>> And just like that, I ordered
soup with Shortcuts and Siri.
[ Applause ]
Now, let's try writing the same
shortcut from the Shortcuts app.
But before I do that, I would
like to customize the shortcut.
I'm going to tap on the soup
field and select the soup that I
usually order from Soup Chef.
I'm going to select clam
chowder.
Now I'm going to tap on the Done
button to save the shortcut.
Let's try writing the shortcut
from the Shortcuts app by simply
tapping on it.
So here I get presented with the
same disambiguation prompt to
choose a store, just like in
Siri.
I'm going to select the first
option.
And done. I just ordered another
soup.
I guess I'm really hungry today.
Now, let's talk about a few ways
to take our shortcuts even
further.
Wouldn't it be great if I never
have to leave my home to have
some soup?
Yes. And that's why I'm adding
something new in soup shop
delivery.
In the shortcut editor, I want
the user to be able to choose
between delivery and pickup
options.
If they choose delivery, let's
just show them a field that lets
them decide where it should be
delivered to.
For example, to their current
location.
If they choose pickup, we should
show them a field that lets them
decide which store to order
from.
To make that work, we need to
express parameter relationships
in the Intent editor.
So let's start by taking a look
at all of the parameters
involved in this.
We have the orderType parameter
which has two possible values;
delivery and pickup.
We also have parameters for
deliverLocation and
storeLocation.
Now, let's take a look at the
Relationship section.
Here for my deliverLocation
parameter I'd like to specify
the Parent Parameter to be the
orderType parameter.
I only want to show my
deliveryLocation parameter if
the parent parameter has the
value of delivery.
And exactly the same thing for
the storeLocation parameter.
I only want to show it if the
parent orderType parameter has
the value of pickup.
So now my users are able to
easily switch between different
order types.
Let's see how to do this in
Xcode.
So I have the Soup Chef project
open here.
I'm going to select the Intent
definition file.
This is where I define all my
Intents.
I already have Intent for order
soup.
Let's take a look at the
parameters.
We have soup, quantity,
toppings, store location.
Now let's scroll down to a new
section in Xcode that we have
called Shortcuts app.
Here you can see the -- all the
Support Parameter combinations
by the Shortcuts app and the
corresponding summaries.
You can also see the preview of
what it's going to look like in
the app.
So currently we have the
storeLocation and we have the
summary which says, order
quantity soup for pickup from
storeLocation.
I didn't put toppings in my
summary so it appears under More
Options.
Now, let's go up and add a
couple more parameters to
support delivery.
I'm going to click on the plus
(+) button and add a new
parameter called orderType.
I'm going to specify a display
name for this parameter.
So if I scroll down to the
Shortcuts app section you can
see that the Order Type
parameter now appears under More
Options and it was automatically
added to the Support Parameter
combination.
Let's scroll up.
Let's specify a new type for the
Order Type parameter.
I'm going to add a new enum.
Let's call this enum Order Type.
I will also specify the display
name for this type.
Now we add to add two cases;
delivery and pickup.
Let's start with delivery.
I'm going to click on a plus (+)
button.
Type delivery.
I will also specify display
name.
Let's add another case for
pickup.
Also provide display name.
Since my users can refer to
pickup as take out when they
speak to Siri, we can add this
as a speakable match.
So I'm going to click on a plus
(+) button and add take out.
Now let's go back to the Intent.
Now we need to add another
parameter called Delivery
Location.
Let's also give it a display
name, Delivery Location.
So now let's scroll down and see
what we have there.
We still have the single
parameter combination which
contains all of the parameters.
But it doesn't make sense to
have both Store Location and
Delivery Location at the same
time.
Let's fix this.
We're going to scroll up to the
parameters section.
Select the Store Location
parameter, expand the
Relationship section, and select
the parent parameter to be the
orderType.
I only want to show my Store
Location parameter if the parent
orderType parameter has the
value of pick up.
Let's do exactly the same thing
for the Delivery Location
parameter.
Let's select the parent
parameter to be the orderType.
Show if parent has exact value
of delivery.
Now, let's go down to the
Shortcuts app section and take a
look what we have there.
Xcode automatically created two
more parameter combinations for
us.
Let's fill out summaries for
each of them.
So will be the Order quantity,
soup for orderType.
Now let's imagine user taps on
the orderType field in the
Shortcuts app and they select
pickup.
In this case, they will see the
parameter combination that
contains storeLocation.
So I already have the summary
for this parameter combination,
but I had a hard coded value for
the pickup.
Let's change this to be a
parameter orderType.
And finally, we need to update
the last parameter combination,
which includes the
deliveryLocation parameter.
Let's add the summary.
Order quantity soup for
orderType to deliveryLocation.
Let's take a step back and take
a look at what we just did.
First, we identified parent and
child parameters in our Intent.
And then, we established
parameter relationship between
parent and child parameters.
And finally, we updated
summaries for each parameter
combination so they look great
in the Shortcuts app.
Now, let's talk about Dynamic
Options.
Most parameters such as String
or Number have obvious input
mechanisms, an input field in
the Shortcuts app.
We're asking a user to specify
some values here.
For some parameters you may want
to allow the user to pick from a
fixed set of values and have
those values be dynamically
provided by your app when the
user is setting up the shortcut.
For my Store Location parameter
I wanted the user to be able to
choose from a list of stores
close to them.
In order to accomplish that I'm
going to check the Dynamic
Options checkbox for my Store
Location parameter.
Once I do that, Xcode will
cogenerate two additional
intent handling protocol methods.
The first one provides store
location options.
And another one for providing a
default value for this
parameter.
In your provide options method
you need to return the current
set of possible values I the
completion handler.
You will also have the
opportunity to provide default
values for parameters with
Dynamic Options enabled.
When your parameter of Dynamic
Options is being resolved during
the shortcut execution time and
your return needs value in the
completion handler of your
Resolve method, Siri will
automatically use the list of
options you provided in the form
of a disambiguation prompt.
Lastly, I want users to be able
to make more shortcuts work
together with other actions.
For example, a user could set up
a shortcut that places an order
and then sends a message with
order details to their friend
who usually picks up their
order.
In Xcode 11 we are introducing
the ability to define your own
custom tabs that you can use in
both Intent and Intent
responses.
So I'm going to define a new
custom type called Order
Details, which contains proper
sets of information that I want
to pass on to another action.
In this example I'm going to
define two properties; Order
Estimated Time and Total.
As you might know an in turn
response is a logical result of
an execution of an attempt.
It contains properties that
convey the result to the user.
To add an output, I need to
define a new property of my
custom orderDetails type.
And then, I need to designate
this property as the output.
When the user taps on the Order
Details variable in the shortcut
editor, they will get presented
with an option to choose one of
the properties that I just
defined in my custom type.
It can also use custom members
of your custom types in response
templates for Siri.
So the same data that is passed
as the output can be spoken
aloud.
Today you learned how you can
allow users to customize your
shortcuts in the new built-in
Shortcuts app.
You also learned how you can
provide conversational shortcuts
in Siri.
Choose the right parameters for
each Intent so your app can
correctly respond to the user
each step of the way.
You can extend the power of your
shortcuts with other apps using
Outputs in the Shortcuts app.
We have another session on
Friday where my colleagues will
talk about design and building
great shortcuts.
You can also find us in the labs
throughout the week.
Thank you so much for coming and
enjoy the rest of your
conference.
[ Applause ]