Transcript
[ Music ]
[ Applause ]
>> Good morning.
Who here has an app on the store
that uses Apple Pencil?
All right, a few of you.
And who has an app on the store
with Apple Pencil features that
is not a drawing or a painting
app?
Oh, one, yes.
Well, today, not only are we
going to learn a few things
about how to make your app great
with Pencil, we're also going to
see that Pencil can be for so
much more than just drawing.
Pencil can do a lot of things
and it can be a bit daunting to
add Pencil interactions to your
app, especially if you've never
worked with it before.
Today, we're going to look at
seven properties of Pencil that,
when implemented, will add
entire new dimensions to your
app.
And to do that, we're going to
add Pencil interactions to an
existing app.
So, here's my app.
It's a fairly straightforward
sudoku game.
The puzzle screen shows the
current puzzle we're solving.
Tapping a cell brings up the
keyboard where the player can
enter a number, and there are a
number of buttons for showing or
hiding the timer, depending on
how competitive you are, for
getting a hint, and for changing
various settings about the
current game.
So, even though this is not a
drawing app, doesn't it seem
perfect for Pencil?
Let's make it happen.
And the first thing we're going
to look at is that Pencil makes
marks.
I don't know about you, but if I
pick up a pen and I try to write
with it and it doesn't write, I
toss it out.
We worked really hard on Apple
Pencil and we don't want people
tossing out their Apple Pencil.
So, don't require tapping a
button or entering a special
mode to draw with Pencil.
Pencil should make a mark the
moment it touches the screen.
When I put Pencil down, I can go
ahead and write my guess
immediately.
Pencil is familiar; it should
behave consistently with what
people using our app have come
to expect.
This means that tapping controls
should behave the same whether
you touch them with Pencil or
your finger.
Here, I can interact with the
various setting sliders using
Pencil just like I would with my
finger.
One thing we definitely never
want to do is have controls that
will have different
consequences, or worse, do
nothing when tapped with Pencil
rather than a finger.
This would be very unpredictable
and confusing.
Pencil has a number of sensors.
It can sense tilt, force, and
orientation, and these sensors
can add layers of depth and
expressivity to our
interactions, but there are a
few missteps to avoid.
A good way to use the sensors is
to make marks more expressive.
For example, the continuous
variation of pressure and tilt
can modulate the thickness of a
stroke.
Or the orientation to affect the
width of a marker.
We don't want to use sensors to
affect things not directly
related to Pencil input.
For example, here, depending on
how hard we press, we get a new
puzzle of varying difficulty.
That's just a confusing
behavior.
Sensors should affect the kind
of marks that Pencil makes, not
things disconnected from it.
Another way to handle sensor
data is discretely.
For instance, to modify the
input based on the pressure
exerted.
So, an idea for a game here is
to let people pencil in a
tentative guess, depending on
how hard they write.
Writing lightly lets you put in
a guess in lighter ink, and
writing with more pressure lets
you commit that answer in black
ink.
We can think about a sensor
reading such as pressor as a
continuum and treat it as such
to modulate a quality of a mark.
We can also split that continuum
binarily, as we just did for
this pencil-in feature.
In this case, we'll want to make
sure to give clear feedback as
to which side of the split we're
on.
We definitely don't want to
break that continuum into more
than two discrete parts though,
as it would become very hard to
differentiate.
People can tell the difference
between writing soft or writing
hard, but they can't tell the
difference between 5 or 8 or 15
different levels of pressure.
Handwriting is tremendously
expressive, and as a result, so
is Pencil input.
As we get our game for people to
try, we notice that people will
often write slightly outside of
the boundaries of a cell and
this is a totally normal side
effect of handwriting being so
personal and freeform, but it
means we need to be flexible
with how we handle input.
What we shouldn't do is impose
unnecessary constraints on
input, like rejecting it if it
doesn't fit perfectly within the
cell.
So, what we're going to do here
is assign the answer to the cell
it covers the most and we also
scale and move it to make it fit
neatly into the cell, which is a
nice little touch.
Now, an interesting question
comes up.
What should we do if the player
starts drawing or writing
outside of the main sudoku grid?
One thing we can try doing is
just nothing, but it doesn't
feel very good.
It goes back to our earlier
point about making a mark.
It breaks the impression of
fluidity and it might even lead
some people to think that their
Pencil is impaired, or the
battery is discharged.
Something we can try is to show
transient marks, marks that
disappear shortly after they've
been made, to communicate that
everything is working fine, just
the margins don't do anything
special.
Another option we can try is to
just let people draw and
annotate in the margins just for
fun.
And this could be particularly
great if later we add
multiplayer to our game.
Now, there is no right or wrong
answer here.
It is up to you and what you
want your app to be; but doing
something is always better than
doing nothing, even if that
something is just to communicate
that Pencil is working.
If you're not drawing with
Pencil, you're writing.
And because you're writing on an
iPad and not a piece of paper,
we have to ask ourselves
questions about whether we want
to process the input further.
For the case of our game, should
we convert handwriting to
digital or should we keep it
analog as it is?
One reason to convert it to
digital is to be consistent with
keyboard input.
And one reason to keep it analog
is that it's fun and personal.
Again, here there are no right
or wrong answers.
It will depend on your specific
circumstances.
You do want to keep it
consistent so that people using
your app know what to expect.
And if we do choose to keep the
input analog, it's always a good
idea to digitize it under the
hood to enable things like
searching for text or, in the
case of our game, so that we can
validate the final answer to the
puzzle the player came up with.
So far, we've been talking a lot
about Pencil and it can be easy
to forget that there is a human
holding it.
Different people will hold and
use Pencil differently.
For example, if I watch a
left-handed person use our game,
we'll notice that their hand and
arm gets in the way of the
controls on the left side of the
screen.
The positioning of these
controls wasn't much of an issue
when our app was used primarily
with touch, but it becomes an
issue with Pencil.
So, what we can do is move the
control to the top or bottom of
the screen or just make them
repositionable.
And, finally, Pencil is really
great when it provides
shortcuts.
That is, letting people do
things faster by drawing and
writing rather than tapping on
the screen.
So, how do we erase a cell in
our game?
It's a two-step process.
The first step is we tap the
cell we want to erase, and the
second step is we press the
backspace key.
So, can we turn this two-step
interaction into a one-step
interaction with Pencil?
Something that would feel pretty
natural is letting people cross
out their answer, so let's give
that a try.
Seems to work pretty well.
But, as I give my game for
people to try, I notice
something interesting when they
try to erase a cell.
Some people will strike it
through, others will scribble it
over, and yet others might draw
something like an X mark.
Handwriting is really
expressive; different people
will cross things out
differently.
So, in this case, it's probably
better not to ask people to
perform a specific mark they
would have to remember.
In addition to being hard to
remember, specific marks might
have different meanings in
different cultures.
For example, in the U.S., it's
pretty common to use a checkmark
to mean correct.
In other countries, that might
not mean anything or even mean
incorrect.
So, what we do is we just let
people erase a cell by
scribbling over it.
Whether they slash it out, draw
an X, scribble over it, draw a
straight line, it doesn't
matter.
All of it works and erases the
cell.
Let's look at another two-step
interaction, getting a hint.
First, we'd have to tap the hint
button and then we tap the cell
we want a hint for.
And this is another interaction
that we can bring down to one
step using Pencil.
Perhaps we can let the player
draw a question mark on the cell
they want an answer to.
And this feels pretty good.
And in this case, because the
question mark is mirrored in the
icon of our hint button, I feel
pretty good about using that
mark as a shortcut for hint.
Now, these seven properties are
by no means an exhaustive list
of all the ways Pencil can make
your app great, but I hope
they've shown you a side of
Pencil that perhaps you did not
expect and I hope they help you
start thinking about what adding
Pencil interactions would mean
for your app.
Thank you.
[ Applause ]