WWDC2014 Session 612

Transcript

X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
>> Good morning-ish.
My name is Andy, and I'm here
to talk about Motion Processing.
So welcome.
We're going to start
with some definitions.
We'll flesh out that
term "motion processing"
by giving you a description of
the features that comprise it.
We'll talk about how they work,
as well as how well they work.
We'll give you some ideas on how
to use it in your application.
We'll help answer some questions
about when should I use one
or the other, what
interface should I use,
and how can I use it with
other services from the OS?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and how can I use it with
other services from the OS?
From this very high level then
we'll dive into the details
of the interface,
we'll talk about usage,
we'll deconstruct
the data objects
so that you can know
better what to expect,
and finally we'll
reiterate the main points
with a coding exercise.
Now before I go any further,
I do want to apologize
because I've been standing
up here saying motion
processing, motion processing.
It's vague.
The title of this track
is "Motion Tracking",
and this session is not Gaming,
so there's probably a
huge number of you who are
in here thinking that you're
going to learn all about how
to track a person's
motion in a gaming context.
This is not that talk, so
nevertheless I hope you'll stick
around because there's a
lot of really neat things
that we're going to
reveal and hopefully even
if you weren't planning
to use this,
at least it'll spark some ideas
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
at least it'll spark some ideas
on how you can perhaps develop
the next great app with this.
So, with that, let's go on.
The biggest thing that happened
since the last time we talked,
Core Motion, has been the advent
of the M7 Motion Coprocessor.
This is an ultra-efficient
microcontroller that we use
to essentially manage all
of our sensory interaction.
It's really cool, but for
the purposes of this talk,
I'm really just going to focus
about how we combine the M7
with an electron
sipping accelerometer
to give you motion awareness,
constant motion awareness,
with no noticeable
impact on battery life.
How efficient is our
motion processing system?
You get 24 hours of
motion activity pedometer
for about three minutes
of a FaceTime call.
So what can you do
with motion activity?
I think that when you're talking
about a mobile application,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
I think that when you're talking
about a mobile application,
where you have limited
real estate,
you have limited bandwidth, and
the user is always in a hurry,
context is key, right?
It's the filter that
allows you to filter
out the nonessential information
so that the really important
stuff just comes to the top
and it allows the user
to connect the dots
so that they can bridge
missing information.
It allows you to
relate information
from disparate sources so that
you can recognize patterns,
right?
Now I don't have to convince
you of how important location is
as a context, it's intuitive and
you're all aware of that, right?
I do a search for
cafes in London,
I would expect different results
if I had initiated that search
in Ohio versus England, right?
Time is a context.
Activity is similarly very, very
powerful as a context, right?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Activity is similarly very, very
powerful as a context, right?
And that's what we're
here to give you,
the ability to know the
user's motion at any time
so that you can frame
the context.
So not all activities are going
to be detected in the same way,
there are going to be
differences, and I want you
to be aware of that
because that's going
to drive different use cases.
Different activities are
going to be better suited
for different ways or
different applications.
So please be aware.
And then when we talk about
performance we're really going
to evaluate on three things.
We're going to look at detection
accuracy, we're going to talk
about the latency of
detection and then, finally,
we're going to talk about the
robustness, how sensitive are we
to differences in, say,
the device location
or situational differences.
So, again, please
bear this in mind.
And when I start talking about
the performance in a little bit,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And when I start talking about
the performance in a little bit,
I'm going to stay
very qualitative.
I will give you some
numbers, but the intent is not
to give you precise
numbers and not
to compare the performance
to our competitors.
We are very proud that we have
industry leading performance,
but we do encourage you, I
encourage you to please go out
and test for yourself
and quantify
for yourself how well
this will work for you
when you get the final seed.
So, with that, let's go
ahead and get started.
The first activity
I'm going to talk
about is the walking activity.
This is something that
most people do every day,
it occurs quite often, so
you're going to see this a lot.
When we talk about
the performance
and how the performance
changes across location,
it's very, very robust.
When you're walking,
we're going to detect it,
and it doesn't really matter
if the device is in the pocket,
on the waist, on the
arm, or even in hand.
Now I do want to caution,
though, that when a person -
it is possible that a
person could hold a device
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
it is possible that a
person could hold a device
so steady while they're walking,
such as when they're trying
to read a text or so on, so that
you can fall out of walking,
but again they're
not going to do
that over the entire
walk so that's okay.
In terms of latency, it's
fairly low, 5 to 10 seconds.
If it's in my pocket
and I get up
and I start walking
it's going to detect
that in about 3 to 5 seconds.
It's going to take longer, as
I said, if it's in the hand
or the user is fiddling
with that.
So that should be
your expectation.
In terms of accuracy
it's very accurate.
You walk, we're going
to detect it.
And I'm going to say
it's accurate on average
because you can expect
that you're going
to intermittently fall into
and out of the walking state,
but again that's okay, right,
because that's actually
the situation
that the user finds himself in.
Let me explain what
I mean by that.
When you get up and
you walk a lot
of times you may get
distracted or you may stop, say,
to open a door, or
somebody stops you to chat.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And so as you stop and as you
start you're going to fall into
and out of the walking state,
and so the application,
going back later in time
to look at say the query,
the history of transitions
between the start of the walk
and the end of the walk you'll
see that it's mostly walk
and sometimes you'll be in a
non-walking state, but again,
that's okay and as long as
you expect that and deal
with that appropriately
it should be okay.
But what does that mean for
how you can use walking?
You should use walking,
if you're thinking
about using walking as a
context in the here and now,
you'll want to be
careful about that
because walking is not
a context here and now,
walking is a context over the
past say 30 seconds, right?
If I've been walking for the
past 30 seconds I'm walking,
but I could be instantaneously
stopped.
I think walking might be better
suited to use as a transition,
and I'll talk a little bit
later in the presentation,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and I'll talk a little bit
later in the presentation,
it'll be more apparent
what I mean by transition,
but walking is how we
go from A to B, right?
And so if you keep that
in mind it'll be a very,
very powerful tool for you
to help infer what the
situational context was.
All right, next is running.
Running is really great.
Again, in terms of robustness,
it doesn't matter
where you hold it.
We're going to detect
running, and it's going
to be pretty uniform
in the performance.
What's great about running is
that not only is the
signal very, very vivid,
in other words, it allows
us to quickly detect
that a person is running.
You take a handful of
steps, a handful of steps,
and we're going to
detect running.
So not only is it
very, very vivid,
but there is no situational
ambiguity.
If a person runs they
really mean it, right?
They're not going to
multitask while they run,
as like they do while
they're walking, right?
When they're running they're
not going to log into Facebook
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
When they're running they're
not going to log into Facebook
and change their
relationship status,
they're just going to run.
And when they stop it's not
because they saw something shiny
on the side of the road
and just got distracted.
No, they stop because
they meant to stop.
So as a context in the here
and now running is
very, very great, right?
Short in terms of latency;
very, very short in terms
of detection; very, very
accurate; and in terms
of robustness completely
robust, so think about that,
just think about how
you'd want to use that.
We've already talked
about the accuracy.
So let's go on to driving,
driving is different
from running and walking because
the motion signature is more
subtle, and because
it's more subtle.
The more coupled the device
is to the vehicle dynamics,
the more likely it
is that we'll be able
to detect the driving
state much more quickly.
So if the user were to, say, get
in the car and put the device
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So if the user were to, say, get
in the car and put the device
on the dash, in the cup holder,
perhaps mounted, that's fine.
As soon as they drive
off we're going
to be able to detect driving.
Conversely, if the user
were to, say, have the phone
in their back pocket, and
it would take a lot longer
to detect driving,
but again that's fine
because eventually
we'll detect driving,
and if you use driving as,
again, defining a transition
or if we were going to
use that with walking,
I'll show you again later in
the presentation how you want
to do that, you can make up
for some of these challenges
from observing things
strictly through motion.
Now the latency, again, is
variable, as I just described.
Now what's really cool is that
we're going to use information
from other sources as
it becomes available.
Without really going into
details what it really means is
that if you get in your
car, you plug your device in
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that if you get in your
car, you plug your device in
or you have Bluetooth
or so on and so forth,
it's going to be able to detect
your driving state pretty
quickly, so that's really cool.
Cycling is new, something
we introduced in iOS 8.
Cycling is very challenging,
and again you need the dynamics
and so it's going to be
very sensitive to location.
If it was mounted on the
upper arm the latency is going
to be fairly reasonable.
And if it's anywhere else, it's
going to take a lot longer.
So definitely I would not
suggest using cycling activity
classification as a hint for
the context here and now.
It's really something
that you'll want to use
in a retrospective manner for
a journaling app, for example.
This would be a perfect
use case for that, right?
So that's motion activity.
Now I want to talk a little
bit about the pedometer.
For the pedometer you have two
things, you have step counts
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
For the pedometer you have two
things, you have step counts
and you have the stride
estimate, how long is your step.
With step counting there is
a surprising amount of value
in just the user knowing
their step count, right?
There's a lot of research
already out there that suggests
that at least for sedentary,
for people with a very
sedentary lifestyle,
the best thing they can do for
themselves is to just to get up
and walk, just take steps.
And we know that, say, the
average American only takes
around 5,100 steps a day.
If you are from Switzerland,
Western Australia,
interestingly, you'll
take about 9,500 steps.
And so what you see here
is a great opportunity,
it's a great opportunity
to provide something
that will really help users
improve their quality of life,
just knowing that I want to
target 7,500 to 9,000 steps or,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
just knowing that I want to
target 7,500 to 9,000 steps or,
if I want to be more active,
try to get over 10,000,
and giving me constant feedback
about how I'm doing,
that would be great.
I think that would just
really make a whole world
of difference.
But if you wanted more than just
step counts and just wanted more
than knowing how active you are,
then stride estimation gives
you more quantitative results,
right?
With stride estimation we can
tell how far your walk was,
and so if you were to like,
say, initiate an activity,
go out and hike, go out and
run, we would be able to say
at the end of it that
you ran four miles
or that you hiked
three-and-a-half miles.
And then, also, if you were to
look at how we do, the user ran
or hiked in short bursts then
you can see the intensity
of the workout, and
again that is something
that is very interesting
and very powerful
to present to the user, right?
Now you can argue that you
could do this with GPS,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Now you can argue that you
could do this with GPS,
and while that's true,
the attendant power cost
in GPS is literally 100
times greater than the power
of the speedometer, the energy
cost of the speedometer.
So you can give, what if you
can give the same performance
in terms of the distance
without that power,
and what if you can give
the same performance
in that distance no matter,
say, the GPS conditions,
whether it's urban,
canyons, trail hiking,
or suburban streets, you
get the same performance,
wouldn't that be cool?
So talk a little bit then
about the performance
of our pedometer.
Similar to the activity
classification,
the step counting is very
consistent across body location,
and that's something that
we're very, very proud of.
Again, it doesn't matter
whether a device is,
here let's say, FitBit.
The device is here, it provides
very, very good clean signature
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The device is here, it provides
very, very good clean signature
of your motion, and so
they're able to get a very,
very accurate count
to your step.
We're able to provide
similar levels of accuracy,
but not only just, not
only here, but in the hand
and everywhere else,
so that's really cool.
Just talked about accuracy.
Another thing that we
really focus on is we try
to be robust to extraneous
inputs.
It is accelerometer-based,
and so you'll see things,
such as periodic inputs into the
device that will generate steps.
We try very hard to
eliminate most of that,
and we actually do
very, very well.
Compared to our competitors,
we're actually much,
much better.
So try this for yourself,
get a Fitbit, hop on a bus,
sit in the back of
the bus, more towards
over the wheel well, right?
Take it up and down the 280,
you'll see that you'll walk
out of there and will have
counted, say, 1,000 steps,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
out of there and will have
counted, say, 1,000 steps,
but again that's a
challenge of a pedometer
or an accelerometer solution.
So stride estimation, this is
again just like the step count,
we're very insensitive
to the body location,
but and this is the
really important thing,
we're also insensitive
across pace.
It doesn't matter if
you're running or jogging
or hiking you're going
to get the same accuracy,
and that same accuracy,
like I said,
is pretty good, very, very good.
And the other great thing
is that it's going to adapt
to the user across over time.
So the more they use it, the
more accurate it's going to get.
So, again, I encourage you when
the final release comes out,
do go try it out for yourself,
take the device, strap it on,
walk around the block,
maybe walk around
the block a few times
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
maybe walk around
the block a few times
and see how well we do.
Okay, so how can we
use motion activity?
I want to start by talking
about the different data models
that we use or the
different ways
that we can get the data to you.
There's a notion of
push versus pull.
In the push interface, we give
you events as they happen.
This is our lowest latency
path, so if latency is important
to you, you'll want
to use this interface.
What that means is that when
motion state transitions happen,
we'll let you know right away.
Every two-and-a-half seconds,
we'll give you the
new step count,
that's the lowest latency.
You can also use
the query interface,
and in the query interface you
will provide a range of times,
a start time and an end time.
And for motion activities,
we will give you basically every
transition that has happened
between the start and end time.
For the pedometer we'll just
give you the cumulative steps,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
For the pedometer we'll just
give you the cumulative steps,
so you'll get a single record
that provides a cumulative
pedometer activity
between the start and end time.
So you understand latency,
but then what is the value
of the query interface?
For that let me talk a little
bit about the architecture,
so we'll start with
the accelerometer.
And as you can probably surmise
we then operate in the data
at a periodic interval.
I mentioned two-and-a-half
seconds earlier
for the step count.
It's actually what cadences
that run everything.
Now I'm telling you this not so
that you set your watches by it
or that you design around this,
I'm really telling you this,
two-and-a-half seconds, so
that you can get an expectation
for the latencies, right?
And so when I say running, we
can detect that you're running
in three to five
seconds, that's really,
that three to five
seconds that's because of
that two-and-a-half second
batch processing that we do.
And really as soon as we just
look at data over that interval
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And really as soon as we just
look at data over that interval
and you're running, we're going
to know you're running
right away, right?
And so this information
as it happens gets pushed
to the client, and so
that's the push interface.
Now what's interesting is that
we also filter the information,
so we'll take all of these state
transitions in, and we'll try
to fuse them with data
from other sources,
and then we'll persist that.
And so if an application were to
come back later in time and ask
for the data, the information
is going to be more accurate
and it's going to be
less intermittent.
We talked about walk and how
you can start and stop, and stop
and start, and start and stop.
With the query interface,
many of those transitions
will be smoothed out for you.
So I think it might be
more tangible if I were
to motivate how we use
this in an application.
So let's say running, let's come
up with a running application.
I mentioned running as one
of those great activities
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
I mentioned running as one
of those great activities
that gives you the
context here and now.
I'm going to see
how we can use that.
So in a running application,
I'm going to be running,
so the last thing I
want to do is to mess
around with the interface or
do anything with it, right?
I just want to launch
it, and I just want it
to understand what I'm
doing and try to react
to me rather than
what I do to it.
And so I wanted to automatically
change this experience,
my experience as I go
through my exercise.
While I'm exercising I want it
to give me constant feedback
on how well I'm doing.
And then it'd be nice that
once I stopped it could
automatically, say,
summarize my results
and present that to me, right?
So, let's think about
how that could happen.
So let's say I've
got this running app,
I'm going to launch it,
I'm going to strap it on,
and then I'm just going to
go through my stretches.
Now music is a really
important part of my experience
when I exercise, and so
let's say that it's going
to start just playing some
kind of preprogrammed,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to start just playing some
kind of preprogrammed,
easy listening kind of playlist.
Now my stretch and my warm-up
routine, it's not constant,
so some days it might
be two minutes,
some days it might
be 10 minutes.
And so I don't want
to design a playlist
around my stretch workout,
I just want to put
something in there.
And then when I start
running it'd be nice
if the OS detected that,
and then the app were then
to quickly or just automatically
change out my playlist,
so to something that's
more uptempo, right?
Don't even have to think
about it, I just start running
and then, boom, the
music changes.
So I'm running, and now you can
imagine the Siri voice coming
out over my earphone, giving me
just this great feedback, "Andy,
you're doing a great
job, you're on pace
for your 15-minute mile."
So just stuff that gets
me really pumped up.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So just stuff that gets
me really pumped up.
And you can imagine I hit my
first mile and it tells me,
you know, what my
split time is, right?
So it's really real cool.
I'm getting this constant
feedback, and then I get
to the end of my
run and I slow down,
and that music is
really bothering me,
so it switches automatically
to a cooldown playlist,
something more suited
to my walking pace now.
And I don't even have to
think about it, right?
It just happens, right?
And then, let's say, I'm done
walking, I'm done with cooldown,
and automatically just, say,
maybe it uses a query interface
to loop back over how I've been
doing the past several days
and compares my performance
historically,
or it might summarize my
run in that particular day.
So all of that basically you
can get for very little cost
with the services that we're
providing you in Core Motion,
so that's pretty cool.
So that's an example of
a running application
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So that's an example of
a running application
where you rely primarily
on the push interface,
you're using context in the
here and now, but let's talk
about a journaling app,
where the requirements
might be different.
And so what would it - why
would I want a journaling app?
I think in a journaling app what
really matters to me is context
and correlation, right?
But I want the app
to do that for me.
I want it to be able to figure
out what I was doing and where,
so I'd want it to correlate
multiple sources of information.
And then ideally I
would like for it
to profile my physical activity,
let me know how active I've
been through the past day.
So, this is what my
typical day looks like,
and so let's say this
journaling app would start
by querying the visit monitoring
service, which is a service
from the location, Core
Location, and it gives me a list
of places that I've
visited throughout the day.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
of places that I've
visited throughout the day.
Typically I go from home,
sometimes I go to the gym,
and then of course I go to work,
and so it detects all that,
and so that provides essentially
the location context.
Now suppose the journaling
app also then queries
for the transport context
between these points?
And so looking at motion
activity it knows that, oh,
I drove from home to the
gym instead of cycling
and doing a solid for the
environment, just really lazy.
Got to the gym, and then from
the gym I walked to work.
And what if the app then
queried the pedometer,
and then I could look and
see, say, how far was the walk
between work and the gym and how
many steps that I took in each
of those different places.
And you can see here that this
presentation of these results is
so much more powerful
than to, say,
that I walked 5,000
steps, right?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that I walked 5,000
steps, right?
To say that I walked 5,000
steps, if I were to look back
at that data, five
days or 10 days
from now it would mean
very, very little to me.
But if I looked at that data
and it was presented something
like this I would know that, oh,
on that day I lost my
keys in the morning.
That's when I walked
around, all over the house.
All right, so that's
some examples of how
to use the services,
how to combine it
with other OS services,
and the difference
between a push and
a pull interface.
So now let's go a little
into the, basically the API.
Before I do that, I want
to talk about privacy.
It's very important
to us, privacy,
we treat it very,
very seriously.
And so, for motion,
just like location,
the user is going
to need to opt in.
Now the very first time you
launch your app we're going
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Now the very first time you
launch your app we're going
to request access on
your behalf, and a popup,
a dialogue will pop up that
looks something like this.
Your app name there, and then
the user would then choose
to opt in, and as
soon as the user opts
in then essentially you get data
up to seven days in the past,
no more than seven days,
just seven days in the past.
Now it's important to note
that at any time the user can
opt out, so when the user opts
out then you will no longer be
authorized to receive the data,
and then you'll have
to basically be able
to handle that case.
Sunny is going to
show you in a bit
with a coding example how
you can detect that condition
and how you can essentially
request for reauthorization.
And so that's a bit
about privacy.
Now let's go on into
actually the API.
So the very first thing that
we want to talk about is the -
for the activity classification
you'll want to deal
with the activity manager,
so you'll get a reference
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
with the activity manager,
so you'll get a reference
to the CM activity
manager object.
And the first thing
you want to do is
to check whether activity
classification is available.
It's pretty standard
stuff, there's not a lot
of mystery there, but I'm
just pointing that out
because it's just
good general hygiene.
Now two ways I mentioned
to get data,
there's the push and the pull.
For this one, this push
interface, what you want
to do is basically provide
a handler and a queue,
and then we'll call you
back on that handler
and we'll basically
call you back.
As soon as you call that start
method we'll call you back right
away, and what we're going to
give you then is the activity
that the user or the device
is in at that time, okay?
And then any subsequent changes
after that we will then
trigger an update, right?
For the push, for the
pull interface, again,
you provide a start time and
an end time in the query.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
you provide a start time and
an end time in the query.
Now the reason why you want
to provide a queue is because,
like I said before, we're going
to call you back not just once,
but we're going to call you back
for every activity transition
that has happened between
the start and the stop,
so you'll basically get a flood
or a storm of records, okay?
And it's also important to
note that the first record
that you get, you'll
actually get a time
that is before your requested
time, and that's okay
because it's that
current activity,
that's the very first activity
and its start time
might be before
when you asked for
it, all right?
Okay, so that's pretty
straightforward.
The start time and the activity,
and so the
CMMotionActivityManager,
you'll get a CMActivity object.
Again, the start date,
not that interesting.
This is what is very
interesting,
in addition to the start date
of the activity we'll
give you a confidence.
The confidence is what you can
use to trade off the difference
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
The confidence is what you can
use to trade off the difference
between accuracy - or trade
off accuracy with latency.
When you start walking, you'll
get walking, low confidence,
potentially medium confidence,
and then high confidence.
Sometimes when we're really,
really sure that you're
high confidence walking,
we might skip the
ones in between,
but typically the
transition is going to go
from low, medium, to high.
And so if you're
really, really interested
in latency you may
want to react as soon
as you get a low confidence
transition, but if you want
to be sure that you only react
to very high confidence
stuff then you want
to wait, all right?
Now I want to get to the actual
classifications, themselves.
The classifications are
presented as a bunch of BOOLs
that you're going to have
to query one at a time.
So, the very first one is
this notion of stationary.
Now stationary here is in the
context of motion stationary
and not location
stationary, right?
Here I'm on the stage, I
am location stationary,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Here I'm on the stage, I
am location stationary,
but I am not motion stationary,
so this thing is only going
to be true when the
device is say on the table.
These are the actual
classifications that we're going
to try to make a call on -
walking, running, automotive,
cycling, you already know that.
Unknown, that's an
interesting one.
What that represents is
essentially the state
in which we don't have enough
data in order to make a call.
When all of those BOOLs
are false it means
that we have plenty of data,
we're just not going
to make the call.
Unknown is, the unknown state
there is the unknown unknown,
and when they're all false
it is the known unknown.
Now I know you can't appreciate
this, but I've always wanted
to stand on stage and say that,
and so, check, bucket list.
All right, again, so to make
these points really salient,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
All right, again, so to make
these points really salient,
let's go through some
example scenarios.
So here we have the scenario
that the device is going to be
in and then all the BOOLs and
what they're going to represent.
So when the device is on
the table, like we said,
it's going to be stationary and,
obviously, it's not going to be
in any of those other states,
so a trivial example, right?
What happens when the device
is on the runner's upper arm?
In that case, running
is going to be true
and everything else is
going to be false, so,
again, not that interesting.
This is interesting, what if
it was in the dash of a vehicle
and it's idling at a stoplight?
In this case the
stationary flag is true,
as well as automotive
flag is true.
So stationary can be -
coexist alongside the other
classification states,
that's an important point.
And if it was in the
in-dash of a moving vehicle,
then stationary toggles
back to false,
automotive still stays true.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Immediately after the
reboot, this is the case
where we don't have enough data
yet to make a determination,
so we actually don't
know what the state is,
and so that's the only time
when - oh, I jumped ahead - oh,
so for this one,
this is interesting,
this is when the
passenger's checking e-mail.
This is because there's motion
here and it's not something
that we want to classify, the
passenger or the vehicle is
in the car, but the
motion suggests
that we actually don't know
whether it's in the vehicle
or not, so all of
this is a false.
We know that you, as a driver,
wouldn't be checking
your e-mail,
so that's why we
have that example.
Finally, the immediately
after the reboot was what I
was talking about earlier.
This is the only time when
the unknown flag will be true.
So similar to checking the
e-mail, what if there is motion
with a very strong
pattern, but like say Zumba?
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
with a very strong
pattern, but like say Zumba?
That one is tough
because it's periodic,
so in some ways it can look a
lot like walking and running,
but if it's on somebody that
is not well practiced in Zumba
or whose hips don't work
that way it could
look really awkward,
and so from a motion signature
standpoint it's something
that it's just going to stay
unknown for us for a while,
but that doesn't preclude us
from extending our
classifications in the future
to add activities that are
interesting, all right?
So, I talked earlier about
how we can use walking
as essentially a transition
and how that can help us,
so let me drive that point home.
Let's take this example, let's
say I'm walking to my car,
and so in that particular state,
walking is going to be true.
I get into my car
and now I sat down,
and so now walking is
no longer true and all
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and so now walking is
no longer true and all
of my flags are false.
I'm essentially in
the unknown state.
I start the engine and I start
driving, and I put it down
and start driving, put it
down, start the engine,
so it's completely stationary
now, so now stationary is true.
And now I start driving,
and so automotive is true
and stationary is false, right?
I stop at a light.
I'm still in automotive,
stationary is true.
Now you can imagine that you're
going to cycle back and forth
on this thing until you
actually pick the device up
and now it doesn't look like
we're in automotive anymore,
and then you'll go
into that empty set.
What's interesting is that
in the driving context you'll
basically limit cycle
among these four states
without actually being
automotive equals true.
You can think automotive equals
true as a hard classification,
and you can think of those other
two states essentially being an
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and you can think of those other
two states essentially being an
inference that you're still
driving, and that you're going
to stay driving until
you walk away, right?
And in this case, walking
serves as a boundary condition
that defines driving
in a larger context.
Does that make sense?
All right.
That's pretty cool.
All right, so now let's talk
about the pedometer API.
So, again, you'll want to
check whether it's available.
It's not going to be
available on all devices.
We're going to, and when
I say it's not available
on all devices, it's not
going to be available
on all M7 devices, we're only
going to support a subset
of them, so you'll want to check
which ones that are supported.
All right, so similar to the
activity, the push interface,
you provide a start call,
and then we'll start
giving you updates.
Now the one thing that I
didn't mention earlier is
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
Now the one thing that I
didn't mention earlier is
that we don't always
provide an update.
I said earlier we were going
to give you an update every
two-and-a-half seconds.
Well, that's not strictly true.
We'll give you an update every
two-and-a-half seconds as long
as you're taking steps.
As soon as you stop
taking steps,
we won't give you an update,
but again that makes sense.
So we'll give you an update
when something interesting
is happening.
What's interesting is that the
only time we violate that is
when your app gets backgrounded
and it gets resumed.
In that case, as soon
as you get resumed,
regardless of whether
you've taken any steps
or the user has taken
any steps you're going
to get an immediate
update, all right?
As far as the pull interface,
again, you're just going
to give us start and
end, and we're just going
to give you everything
that happened in between.
So there's really not a
lot to say about that.
Start and stop time, again,
that's going to mark the -
what's interesting here is that
for the pedometer, every record
that you get you'll have
the same start time,
and that start time is the time
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
and that start time is the time
when you call the
start update method.
That's really, really
important, so every record
that you get will have
the same start time,
you'll just have a
different end time, okay?
And that end time represents
basically the cumulative number
of steps taken between the
start time and the end time,
so you can think of every record
essentially just providing a
cumulative value that
is delta increment
over the previous value.
The number of steps is going
to be an integer number,
and one of the things
that we always get asked
about is what happens if
you provide several queries,
one query over say 24 hours, and
then 24 queries over an hour,
should you expect to get
the same number of steps?
And the answer is yes, there
might be slight differences due
to rounding issues or
interpolation issues,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
to rounding issues or
interpolation issues,
but by and large the number
of steps should be the same.
And the reason why you'd
want a large query,
in addition to smaller
queries, again,
it depends on your use case.
If, say, for example, you know
that a person left the
particular location
at a certain time and ended
in a location then you want
to just provide a query over
that time interval, right?
All right, total distance,
again, similar to the number
of steps, except distance is
going to be a fractional number
and that's going to be
represented in meters.
Again, that is the total
distance that you travel,
it could be from start to stop,
so it's not the actual
stride themselves.
All right, so that is pretty
briefly the Motion Processing
services that we provide.
The APIs are pretty
straightforward, pretty simple,
but you'll find that the
things that you can do
with them can potentially
be very, very powerful.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
with them can potentially
be very, very powerful.
The stride estimate, I think can
basically facilitate a whole new
set of applications
around health and fitness.
And you can imagine now
using health or the things
that you achieve in fitness
as an additional currency.
If you've not seen the HealthKit
session, I encourage you
to watch that on video because
that perhaps will give you some
ideas on how you can use this
in a social media
context, all right?
So that's it, all that I have.
I'm going to invite
Sunny up to actually go
over the demo exercise.
[ Applause ]
>> Thank you, Andy.
Hello, everyone.
My name is Sunny.
I'm a Developer on
the Core Motion Team.
So the topics I want to be
covering today are mainly
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
So the topics I want to be
covering today are mainly
about CMMotionActivityManager,
CMPedometer, as Andy
just covered.
We'll be taking a
deeper look at both
of these classes during
the coding exercise.
Now as we're going
through coding exercise,
at times there might be a lot
of code that's displayed
all at once.
Don't worry too much
about following every
last detail, though.
This coding exercise has already
been made publicly available.
The important parts, though,
I will be sure to highlight,
and I think as long
as you follow those parts
it'll make your experience
with the Motion Activity
APIs that much smoother.
So let's go ahead and start
with the coding exercise.
Here you go.
So just some context - so this
coding exercise eventually
compiles into a UI application,
and the code that we're
about to implement right
now makes up the data model
that the UI application
will use.
The very first thing
that we want to do
with this data model is, as
Andy mentioned, we want to do -
we want to check for
proper availability
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
we want to check for
proper availability
of these APIs before
we start using any
of the queries or
the live updates.
So let's go ahead and
go implement that.
Get rid of that typo
there, and the check
for availability looks
something like this.
And you look at the
parts I have highlighted.
Here it's as simple as checking
for activity that's available
for CMMotionActivityManager
and isStepCountingAvailable
for CMPedometer?
It's important to check for
both because, as Andy mentioned,
not all platforms support both.
Now that we've done our due
diligence it's time for us
to create our instances
of CMMotionActivityManager
and CMPedometer.
Something to keep in mind with
these classes is that a lot
of our APIs are largely
callback-based,
so you want to keep in mind
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
so you want to keep in mind
about the lifetime
of these objects.
For my class, because
I'm expecting
or will be expecting
live updates
at any point during my
program, I've decided
to tie really the
lifetime of these objects
with my data model, in
essence tie the lifetime
of my data model
with my application.
A pattern I've seen commonly
with code that has come
across my table is people will
very commonly create an instance
and assign it to an object
that only lives on the stack,
and so what happens
when that object perhaps
in a stack frame
goes out of scope?
Well, that object is released,
and no more callbacks
are called.
So it's important
that you keep in mind
about lifetimes of
these objects.
You just want to make sure
that these objects are still
available when callbacks,
when you want callbacks
to be involved.
The next thing we want to do
is check for authorization.
This is - we handle
most of the details
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
This is - we handle
most of the details
of authorization for you, right?
Every time you make a call
that accesses motion activity,
we'll do the proper popups.
The only time you'll want to
do something more than that is
to check to make sure that your
app does have authorization
to access activity
because when you don't,
you want to show a dialogue
box saying, informing the user
of his mistake and
allowing your app a chance
to correct that behavior.
So the query looks like this.
As Andy mentioned,
there's - well,
the authorization really
is just a simple query.
We don't really care about
the results that come
from this query, we
just care about sort
of whether it returns
an error or not.
This is why the "from" dates
and "to" dates are
both equal to now.
Now on the callback we
just want to make sure
that there's no errors, and
if there is an error code
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that there's no errors, and
if there is an error code
that it's not equal to this
particular error message.
And if you've done that
then you've made sure
that your app is, indeed,
ready to start querying
for motion activity data.
There's one more
thing to notice here,
I've chosen to use CMPedometer
to check for this authorization,
and because my
authorizationCheck
CompletionHandler may possibly
invoke some UI dialogue
informing the user that
motion activity was denied,
you will want to make sure
that you dispatch that block
onto the main queue because
blocks that are passed
to CMPedometer will be
invoked on an arbitrary queue
that might not necessarily
be your main queue.
Okay, so now that we've done
our check for authorization,
now it's time for us to start
querying data, and this is kind
of the magical part of our
API because it allows your app
to give the illusion that you've
been collecting data all along.
So the queries, for
this data model,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
this data model is
responsible for collecting data
from start date to end date
for both motion activity data
and step counting
data, and when all
of that data has been
collected then it's time
to invoke some sort
of completion handler
allowing the UI to be updated
with all of that data.
So the motion query looks
something like this.
There's a start date, there's an
end date, there's a main queue,
and there's a completion
handler.
Be sure to check for errors
because at any point during your
program's operation the user can
still go ahead to the security
screen or the privacy pane
and disable authorization
on your app.
After that, you can
start playing
around with the activities
that are passed back
when this call back is invoked.
Now a word about the
types of data segments
that are passed back
after this query,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that are passed back
after this query,
very often you might
not be interested
in all the data segments
that are passed back.
For example, you might not be
interested in all the periods
of time for which your
device stays static or that,
if you're a fitness app you
might want to consider a trip,
a jog through a shopping mall
as one bit contiguous walking
segment as opposed to a walking,
stopping, walking,
stopping segment.
And so you'll want to do some
sort of app-specific filtering
on this activity data to kind of
clean up and make it appropriate
and more sort of easier to
understand for the user.
And this is what,
this additional processing
method is doing,
it's doing some filtering that's
specific for the demo app.
In the demo app, we'll do some
filtering based on confidence,
we'll do some filtering based
on getting rid of data seconds
that are short enough
and have no sort
of classification behind them.
But I'll leave the sort of
implementation details up to you
as an exercise for you
to check out later.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
as an exercise for you
to check out later.
Now that we've done that,
we have activities ready,
they're filtered, they're almost
- we're almost ready to push up
and let the UI know that
all of our data is ready,
but as I mentioned we're also
interested in step count data.
The question that
comes up now is, well,
how do I synchronize my
motion activity results
with my step counting results,
and what I like to do is
to solve that problem
and nest the query
so that one successful query
results in another query,
and at the end of all
the successful queries is
when you tell the UI is
ready to display the data.
The step count query
looks like this.
There's a start date,
there's an end date.
Note again the absence of an
operation queue parameter,
this means that if you were
to invoke any UI modification
behavior inside your completion
block, that it's wise to
dispatch your first domain.
And let's go ahead and
invoke the completion block
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And let's go ahead and
invoke the completion block
that ultimately invokes the UI.
And that's it, at this point
my app has all the data
that was accumulated on behalf
of my app while my app
was in the background.
So in order to further
the user experience
so that you can see live updates
as the user is using the device,
let's go ahead and start
using the live updates parts
of our API.
So we'll start with step
updates, it's as simple
as calling starts, and
for people who are used
to seeing step counter
you might have noticed
that we have a new
parameter here and a state,
this is a convenience for
you so that in case you want
to start step counting, but you
want some offset applied to it,
say you're a daily journaling
app, you want step counts
to be counted from the beginning
of today, not from right now.
You can now specify that date,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
You can now specify that date,
that saves you an
additional query and having
to combine information together.
Again, we did not specify
an NS operation queue
because this makes the API
simpler, but if you want
to do any UI modification
behaviors,
be sure to dispatch this back
to the main queue before you
run your completion handler.
I know I'm sounding like a -
I'm repeating myself a couple
of times, but this is something
that I've found something
to be problematic in
my own app development.
Stopping step updates is
simple, you just call stop.
Let's go ahead and start
motion updates, as well.
Starting motion updates is
very simple, specify queue,
specify a completion block, and
they'll all be kind of handled
for you, and you'll have
indications every time you get a
new motion activity.
And stopping motion updates is
also as simple as calling stop.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
And stopping motion updates is
also as simple as calling stop.
And there you go.
At this point you have a
fully functional data model
that you can use in your UI
app to get motion activity data
for when your app is in
the background and for
when your app is
in the foreground.
So let's go ahead
and take a look
at what this eventually
compiles into.
So here we have the
motion activity demo app,
and in the very first
view you can see a listing
of the past seven days, for
which we've collected data,
and if you click on any of
these days you'll see sort
of some transformations
that we've done on the set
of activity data segments.
You'll see the total
durations for walking,
running, driving, and moving.
And because I've clicked
on the Today view you can
also see my current activity,
and because this device is in
my hand it says it's moving.
If I set this device on
the floor or on the table,
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
I'm sorry, it will now
say it's not moving.
It will also show you a
set of live step counts.
What's more interesting
or what's really cool
about this is once you do the
application-specific filtering,
you can get a lot of very useful
contextual information, as well.
And this is sort of the filtered
history of my day so far.
You can see when I walked
to my car in the morning,
from 8:15 to 8:20, and you
can see my commute from home
to work, from 8:20 to 8:36.
Now I actually have
a real time span
for how long it took
me to get to work.
You can see me walk from my
car to the crosswalk in front
of aisle two, and I took 50
steps, and there's a little bit
of a running segment
here because I was trying
to beat the Ferrari that was
stopped and I didn't want
to hold up the exec
in that Ferrari.
After that, I walked to get
some breakfast, and you can kind
of unfurl all of that, and
you can see when I make
that big long drive from
aisle two to Moscone Center.
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
that big long drive from
aisle two to Moscone Center.
So, cool, so that's the demo
app, and I've talked a lot
about sort of the steps
that are necessary
for accessing motion
activity data.
I've talked about
checking for availability.
I've talked about
initialization.
I've talked about authorization.
And I've talked about
historical queries.
I've talked about live updates.
However, nothing beats
hands-on experience,
so I highly encourage
all of you to go ahead
and download this coding
exercise and compile it,
deploy it in your phones,
and take a look at the data
that has already been
collected for you,
I think you'll be
pleasantly surprised.
So for any questions
that you might have
or for more information about
our Motion Activity Frameworks,
feel free to contact Allan or
after WWDC feel free to interact
on the Developer Forums.
As Andy mentioned,
motion activity is just
one context, right?
However, when you
start combining this
with other contextual
queues from HealthKit,
from Core Location, it can
help you build a very powerful
X-TIMESTAMP-MAP=MPEGTS:181083,LOCAL:00:00:00.000
from Core Location, it can
help you build a very powerful
picture of what the
user has been doing
and what the user is now doing.
So we highly encourage you to
check out the sessions related
to HealthKit and Core
Location, as well.
So I thank you all for
attending this session.
We really appreciate
you taking the time
to understand the Motion
Activity APIs a little bit more.
We hope to see some of you
at the session, and for those
of you who we don't, we hope to
see some of your work on that,
and we can't wait to see
what you will do with these -
what you'll come up
next with these APIs.
Thank you so much.
[ Applause ]