Transcript
>> Hello, everyone.
My name is John Blackwell, and
I'm an engineer on the Core
Motion framework.
Today, we're going to be talking
about creating immersive
applications with the Core
Motion framework; and we're also
going to discuss a number of
best practices along the way.
So today, we're going to be
covering a number of things.
The first is a brief overview of
what Core Motion provides.
Next, we're going to talk about
authorization, a frequent pain
point for many of our
developers.
Then, we're going to look at
some new additions to Historic
Accelerometer; and then, we're
going to dig into DeviceMotion,
the sensor fusion that we
provide in Core Motion.
And finally, we're going to look
at Badger with Attitude, a game
where we'll take some of the
concepts that we've talked about
in DeviceMotion and put them
into practice.
And with that, let's take a
closer look at the Core Motion
framework.
The Core Motion framework
provides access to the
accelerometer, the gyroscope,
the magnetometer and the
altimeter; and through the
framework, you can access raw
forms of the sensor data, as
well as process forms and use
these in your applications.
Now, there are a number of
interfaces available through
Core Motion.
The first is CMMotionManager,
which provides access to the raw
sensor data, as well as the
sensor fusion in the form of
DeviceMotion.
CMAltimeter provides access to
relative altitude updates for
the device.
CMPedometer provides access to
step counts as step-related
information, as well as
work-out, pause and resume
events.
MotionActivityManager provides
access to the activity context
of the device; for example,
walking, running or automotive,
etc. And CMSensorRecorder
provides access to historic
accelerometer data.
So that's a brief overview of
what the Core Motion framework
provides.
Next, let's take a closer look
at authorization.
So of the APIs that I just
mentioned, these following four
are sensitive due to the nature
of the private information that
they expose about the user.
So to handle this, we have a
prompt that appears the first
time you invoke one of the
sensitive APIs.
Keep in mind that the first time
you invoke the sensitive APIs, a
prompt will appear for users;
but after that first time, your
users will need to go into
Settings to change the
authorization state for your
applications.
Now, let's take a look at what
it looks like the first time you
call one of these APIs.
At this point, your users will
need to decide if they want to
grant access to the Motion
Fitness data; or deny access.
And as a developer, you're going
to want to handle the case where
they deny access.
Now, in the past, we've asked
you to do something like this
where you call any of our APIs
that are sensitive.
In this case we're calling
queryPedometerData; and then you
ignore the result that you get
back from the API and only look
at the error code.
And if the error code is not
authorized, at this point, you
know your application's been
denied access to that motion and
fitness data.
Now, we realize that this is
less than ideal.
You need to jump through hoops
to get access to your
authorization state, and once
you get it, you can't tell
exactly why your app has been
denied access.
And that's why this year, we're
providing an Authorization
Status API.
This API is available in the
same four classes that I
mentioned before, and it's
available on iOS and watchOS.
Let's take a closer look at the
CMAuthorizationStatus value that
you'll get back from this API.
The first state, notDetermined,
represents the state before the
user has been asked about
authorization in your
application.
Restricted represents the state
where the user is unable to
change the authorization state
for your application
specifically; and this can
happen when motion and fitness
is disabled in privacy.
The next state -- denied --
means what it sounds like.
Your application's been
specifically denied access by
the users.
And authorized means your app is
ready to access the user's
motion and fitness data.
Now, let's come back to the
authorization check that we were
looking at before.
One of the first best practices
that I want to talk about is
making sure that you check for
the availability of a given API
first before you ask for the
authorization status.
In this case, we're asking for a
StepCountingAvailable.
And the next thing that you're
going to want to do is query for
the authorization status.
And at this point, it's up to
you as a developer to decide how
you want to handle the
authorization that you get back
from the framework.
You now have a simple mechanism
for doing this on both iOS and
watchOS.
I encourage you to consider how
you can use the authorization
status app -- authorization
status API in your applications
-- and you should definitely use
it.
Next, we're going to talk about
Historical Accelerometer.
Historical Accelerometer, also
known as CMSensorRecorder,
provide 50 hertz accelerometer
data; and we can record this for
your applications while your
apps are in the background.
You can request up to 36 hours
of accelerometer data, and this
data will be stored on your
behalf for up to three days.
Now, Historical Accelerometer is
currently available on Apple
Watch, and today, I'm excited to
announce that it's now available
on iPhone 7 and 7 Plus.
This opens up a whole new set of
use cases for your applications,
and to get you thinking about
how you can use Historical
Accelerometer on iPhone, let's
walk through one sample
application now.
So let's say you're a big
automotive enthusiast, and you
want to build an application to
enable you to track your car's
performance over a long-track
day.
How would we go about building
this?
Well, the first thing that we
want to figure out is when the
user is driving, and for that,
we can use Motion Activity.
Motion Activity provides an
automotive state, and we can use
this to determine the periods in
which the user is driving.
Now, I want to take a brief
minute to talk about the
automotive detection.
In iOS 11, the automotive
detection has received special
attention to ensure
best-in-class performance.
This automotive state in Motion
Activity is the same state
that's being used to power Do
Not Disturb While Driving; and
it's also available for you to
use in your applications.
If, say, you wanted to
personalize your application's
UI while the user is driving.
Now, coming back to our
automotive performance tracking
application, the next thing we
want to do, is we want to gather
the accelerometer data.
And for that, we can use
SensorRecorder.
We can gather the accelerometer
data for the periods in which
the user is driving, and
translate that into metrics for
the user, such as how many Gs
they were pulling through a
turn, or maximum lateral Gs or
longitudinal Gs.
Now, there are many different
ways you could consider building
this kind of application, but by
using the Motion APIs, you can
provide your users with a
low-power, all-day experience.
Now, there are a number of best
practices to keep in mind with
Historical Accelerometer.
The first is that you want to
choose the minimum duration that
makes sense for your
applications.
So for automotive performance
tracking application, we may not
need the full 36 hours of data
-- something more like 8 to 12
hours would make more sense.
And the next best practice to
keep in mind is to consider
decimating or dropping samples
if you don't need the full 50
hertz accelerometer data.
These two suggestions, these two
best practices, will reduce the
amount of processing that your
application is doing, and in
turn, save the user's battery
life.
Now, I encourage you to consider
how you can use Historical
Accelerometer on iPhone.
Next, let's talk about
DeviceMotion.
DeviceMotion is the name for the
sensor fusion algorithm that we
provide in the Core Motion
framework.
Now, there are a number of
things that go into
DeviceMotion.
The first sensor that we use in
DeviceMotion is the
accelerometer, and the
accelerometer enables us to
measure the accelerations
imparted by the user, as well as
the acceleration imparted by
gravity.
The gyroscope enables us to
precisely measure the rotation
rate of the device.
And the magnetometer allows us
to measure local fields around
the device as well as the
Earth's magnetic field.
Now, when dealing with the raw
sensors, there's a number of
challenges to keep in mind.
With the accelerometer, it can
be difficult to distinguish from
the accelerations imparted by
the user; and from those
imparted by the force of
gravity.
With the gyroscope, we can have
bias in the measurements over
time, and with the magnetometer,
it can be difficult to
distinguish between the local
fields and the Earth's magnetic
field.
And this is where DeviceMotion
comes in.
DeviceMotion provides 3D
attitude tracking while the
device is undergoing free-space
motion.
And it does this by fusing
together all of the sensors to
relay the advantages of each,
while minimizing the
disadvantages.
And as a developer, what this
really means is it enables you
to focus on how you want to use
the motion data rather than the
mechanics of trying to get the
best from the sensors.
Now, we've talked about
DeviceMotion in a number of
previous sessions.
I encourage you to check them
out.
We go into details about the
sensors and the DeviceMotion
algorithms.
But today, we're going to think
about how we can use certain
aspects of DeviceMotion to
create immersive applications
for your users.
So as a developer, when you're
first getting started with
DeviceMotion, the first thing
that you need to consider are
the reference frames.
The reference frames determine
which sensors are used in the
fusion, and how attitude is
calculated for your
applications.
The first reference frame,
xArbitraryZVertical, fuses
together the accelerometer and
the gyroscope but does not use
the magnetometer.
And the last three reference
frames, xArbitraryCorrected,
xMagneticNorth, and
xTrueNorthZVertical, use all
three sensors.
Now, let's talk about the first
reference frame in more detail.
Let's say you've got an awesome
racing game where you're
currently using touch controls
to allow your users to steer in
the game.
This is great, but we could
provide a more immersive
experience using motion.
What we want to do is we want to
enable users to steer using
their device; so when they tilt
their car -- tilt their device
to the left, the car will turn
left.
Now, as long as the device is
relatively static, we can use
the accelerometer to estimate
the force of gravity; and once
we have gravity, we can use that
to determine the tilt, or the
angle offset from gravity.
One thing to keep in mind,
though, is that if you were
using the accelerometer on its
own, certain gestures can be
ambiguous.
Tilting the device to the right
can look the same to the
accelerometer as sliding the
device to the left.
Now, one way you could consider
dealing with this is by
averaging over the accelerometer
signal.
In doing so, you would reduce
the short-term effects from the
user and leave only the
long-term effects, such as the
force of gravity.
However, in doing this, you
would notice your application
would now respond more slowly.
And this is where DeviceMotion
comes in.
DeviceMotion means that your
application doesn't need to
build up filtering logic to get
great performance from sensors.
With the xArbitraryZVertical
reference frame, this is the
default reference frame that
your application will receive if
you don't explicitly specify one
when you start DeviceMotion
updates.
And this reference frame is
great for use cases where you
want to track the tip and the
tilt of the device.
And for our use case in the
game, the accelerometer and the
gyroscope are going to fuse
together to allow us to more
quickly and precisely track
gravity.
And once we have gravity, we can
translate that into tilt for our
game.
Now, to walk through exactly how
to do this, my co-worker Ahmad
is going to come to the stage a
bit later and show us.
This reference frame is also
great for use cases where you
want to track gestures from the
user.
I encourage you to check out our
sample application, SwingWatch.
SwingWatch is an app that runs
on the watch that uses
DeviceMotion to track when
you've made a forehand or a
backhand gesture during a tennis
game.
Both the sample code and the
session are online, and I
encourage you to check them out.
So, let's say you've got another
game.
Let's say it's a shooter game
where you allow the user to aim
using virtual thumb-sticks.
This is great, but we could
provide a more immersive
experience using motion.
And what we want to do is we
want to determine where the user
is pointing their device and
have that translate into aiming
in the game; and for that, we
want to use attitude.
Attitude provides the rotation
from the reference frame, fixed
when you first start
DeviceMotion updates, to where
the user is currently holding
the device in 3D space.
Now, one way you could consider
getting attitude is by taking
the integral of a raw gyroscope
signal.
Your sway would determine the
attitude.
However, this method for
determining attitude would drift
over time due to bias in the
gyroscope, and that's where the
xArbitraryCorrectedZVertical
reference frame comes in.
This reference frame uses the
magnetometer to improve the
horizontal attitude estimation
that we provide; and as a
developer, what this reference
frame means to you is it
provides reliable attitude
performance with a fixed center
reference.
So in the game, your users can
move the device around and then
come back to a known center
location.
Now, there are many other ways
you could consider using this
reference frame.
Let's say you wanted to build a
360-degree photo and video
player application.
Your users would be able to move
their device around and then
bring it back towards the center
-- towards the dock overlooking
the lake.
Now, another way you could
consider using this reference
frame is for something like a
virtual-reality real estate
application where you want to
allow your users to look around
at different parts of a room
just by moving their device.
Now, let's say you build this
application out, and your users
love it; but they want to get a
better sense of in which
direction the windows face, or
in which direction they can
expect the sun to rise.
And for that, we want a world
reference.
Now, you could consider using
the magnetometer for this.
This would provide a world
reference and enable your
applications to orient with
respect to the world.
However, using the raw
magnetometer, you would find the
output susceptible to
disturbances.
Some of these are inherent to
the device, and some of these
are external to the device.
This is where the final two
reference frames come in.
These reference frames use the
magnetometer to orient the
device with respect to the
world, and these reference
frames handle magnetic
device-level effects; as well as
in challenging magnetometer
situations, we can stabilize the
output.
Now, choosing between these two
reference frames is going to be
based on the needs of your
application.
If you've already got data
that's referencing true north --
for example, map data -- it
would make sense to use the
xTrueNorthZVertical reference
frame.
Now, how could we consider using
these reference frames?
Well, one example is things like
stargazing applications, where
you want to allow your users to
point their device at a star in
the sky and identify it.
Another way you could
considering using these
reference frames are for things
like augmented reality
applications where you want to
fuse the camera data with a
world reference.
And for that, we encourage you
to check out ARKit.
ARKit's session follows
immediately after this one.
Now, let's come back to our
virtual-reality real estate
application.
Let's say you built the
application out, and your users
love it, but you want to provide
them with more features.
Let's say you want to put
landmarks on the horizon in the
direction that they actually
are.
How would we go about doing
this?
Well, what we want is heading.
Heading gives us the direction
the device is pointing with
respect to north.
So when the device is pointing
straight towards north, heading
would return 0 degrees and as
the user rotates their device
around, the heading angle would
update.
Now, one way you could consider
getting heading is by using
CoreLocation.
CoreLocation provides a start
updating heading API that you
could then use the heading from
and fuse that in with the data
that you're already getting from
DeviceMotion.
One thing to keep in mind,
though, is that CoreLocation's
heading can fuse course.
Course is the direction of
travel for the device.
Now, this may make sense for
things like turn-by-turn
navigation applications, but for
things like augmented reality,
this may make less sense.
I encourage you to check out
CoreLocation's section on
Thursday for more details.
Now, another way you could
consider getting heading is by
calculating it from the attitude
estimation that we provide in
DeviceMotion.
However, getting this right in
all circumstances is
non-trivial; and this is why
this year, we're adding heading
as a first-class property in
DeviceMotion.
Heading fuses together the
accelerometer, the gyroscope and
most importantly, the
magnetometer, to give us the
direction the device is pointing
with respect to north.
Keep in mind that heading is
only available on iOS, where the
magnetic field property is
available.
Now, let's take a closer look at
the API.
Heading is only valid for the
xMagneticNorth and xTrueNorth
reference frames.
For the other two reference
frames, you'll receive a
negative value back from
heading, and heading will give
you from 0 to 359 degrees from
the X axis that you've chosen in
your application, either
MagneticNorth or TrueNorth.
Now that we have heading, we
could use that to overlay the
Golden Gate Bridge on the
horizon in the direction that it
actually is.
Now, with DeviceMotion, there
are a number of best practices
to keep in mind.
The first is that you want to
make sure that you check for the
availability of a given
reference frame before you start
updates.
And to do that, you can use the
availableAttitudeReference
Frame's API.
This will return you a
CMAttitudeReferenceFrame bitmask
that you can then and, with the
reference frame that you are
interested in, to determine
availability.
And the next thing to keep in
mind is that the choice of
reference frame is key for your
applications.
This is going to determine how
attitude is calculated in your
applications, as well as which
sensors go into the sensor
fusion.
Now, we've talked about a number
of things in DeviceMotion, but
let's get a little bit more
practical.
Let's take some of the concepts
that we've talked about in
DeviceMotion and put them into
practice using a game.
Badger with Attitude is a game
where we're going to apply
DeviceMotion to translate into
controls for the game, and for
that, I'd like to invite my
co-worker, Ahmad to the stage to
walk us through this.
[ Applause ]
>> Thank you, John.
Hello, and welcome.
Today, I'm going to be taking
some of those concepts that John
just talked about and help you
put them through a practical
example.
My name is Ahmad, and I'm an
engineer on the Core Motion
team.
I'll be using the Badger app,
which has been developed by our
colleagues over at SceneKit.
You may have seen this app in
last year's session.
In it, you play a cute little
badger called Bob.
He sits in a mining cart rolling
down some tracks, selecting gems
and power-ups along the way.
So we're going to take this app
with existing swipe controls and
use DeviceMotion to transform it
to motion-based gestures.
Here's what we've got in store
for you today: First, we're
going to talk about the existing
controls and what we're trying
to accomplish.
Then we'll show you some of the
basics of using DeviceMotion.
And finally, the algorithms we
use to detect those gestures.
So the Badger app allows you to
swipe your finger up across the
screen to make the badger jump,
and if you swipe your finger
down, the badger ducks and hides
inside the card to avoid the
obstacles.
And swiping left or right will
make the badger swing to reach
out for those gems.
If you've worked on beautiful
graphics like these, it's a
shame to have to hide them
behind your finger while you
play the game.
So let's use the phone as our
controller here and have the
user fully immersed in the
experience that we've built for
them.
So first, we'll detect if the
user has rotated the device
towards them to make the badger
jump up.
A slight bump in the device
downwards will make the badger
squat and hide in the cart.
And tilting the device left or
right will make the badger lean
either way.
Now I want you to focus on those
couple of points and think about
them when you're looking at the
sample code later, or thinking
about incorporating motion into
your application.
As John has mentioned earlier,
Core Motion allows you to
interact directly with the
sensors.
Let's take the accelerometer,
for example.
That input may look fine as long
as the user is semi-stationary,
but if you start walking around
or get on a bus, then you're
going to have to account for
these additional accelerations.
So with DeviceMotion, we've
taken other sensors like the
gyro and the magnetometer that
complement the accelerometer
really well, and we fuse those
inputs for you so we can
minimize those environmental
factors and let you focus on
capturing those motion controls
rather than how to process the
sensor input.
The DeviceMotion APIs allow you
to query for samples in two
different ways, the push and the
pull mechanisms.
Let's take a close look at those
two.
The push mechanism is a great
way for you to detect a discrete
gesture across a small window of
time.
In the SwingWatch app from last
year's session, we used the push
mechanism so that the framework
will push to us updates as they
are available at a fixed
interval.
Then we would capture those
updates and detect if the user
swung their arm to perform a
backhand or a forehand.
This is what the API looks like.
You use the CMMotionManager's
start DeviceMotion updates
function; you provide a
reference frame of interest,
provide an operation cue for
your motionHandler to run on as
soon as those samples are ready.
However, if you want to know
what the current state of the
device is, then you want to use
the pull mechanism here.
As we'll show you later on in
the Badger app for the tilt
gesture, we want to make the
Badger lean at the same angle
that the phone is in.
So we ensure we provide the
responsive and smooth experience
for our graphical application.
The API for the pull mechanism
is a bit simpler.
You call
startDeviceMotionUpdates, and
you provide the reference frame;
and whenever you're ready, you
can pull for the latest
DeviceMotion sample from the
framework.
We'll be releasing the sample
code for use, so you'll be able
to take a look at it later, but
let's focus on the Core Motion
parts for now.
So to get start with them for
the Core Motion framework and
then instantiate an instance of
the CMMotionManager.
Then we'll check if DeviceMotion
updates are available on this
platform.
And if you recall from John's
talk, the tip and tilt gestures
that we're interested in track
where gravity is in the device
frame.
So we'll be using the
xArbitraryZVertical reference
frame, and we'll check if it's
available on the platform.
You may have noticed I did not
check for the authorization
here, and that's because I'm
using the MotionManager API,
which does not access sensitive
data.
For the rest of the talk, we'll
assume the device is in the
[inaudible] orientation, but in
the sample code, we'll show you
how to detect and handle other
device orientations.
For the first gesture, when
rotating the device towards you
makes the badger jump up, we
want to capture the magnitude of
that rotation rate along the
horizontal axis of the phone.
In this case, it's the Y-axis.
So we'll be looking at the
rotation rate property from the
DeviceMotion object.
And we chose that specifically
because we're not interested in
the current angle the device is
making, but rather a quick
change in that angle.
So if we use the rotation rate,
we'll be able to detect a quick
pulse and make the badger jump
accordingly.
This is a gesture that we're
detecting across a small period
of time, so we're going to be
using the push mechanism for it.
Let's see how that will look in
the code.
To start off, we'll set the
update interval to 50 hertz.
And you want to be careful when
you set that.
You want the samples to be
coming in fast enough so that
you can capture that gesture;
but not too fast that you're
increasing your computational
and memory requirements.
Then we'll start DeviceMotion
updates using the push
mechanism, provide our
xArbitraryZVertical reference
frame, a queue, and I'm using a
standard operation cue here, and
finally, our motionHandler.
This is what our motionHandler
will look like.
It will get called as soon as
samples are ready.
The first thing we'll do is
check for any errors and then
grab that rotation rate from the
DeviceMotion object, and look at
the Y-component that we were
interested in.
Then we store that in a buffer,
and I'm using the circular
buffer here, so as soon as the
samples are coming in, we'll
accumulate more of them.
Since we're using the SceneKit
renderer for this application, I
chose the renderer's
updateAtTime function.
This will get called just before
you render a new scene, and this
is the ideal place for me to
check for the state of that
buffer and then update the game.
And I'll leave it up to you to
find out where in the
application is the best place to
do that.
And then I simply check if the
mean of that buffer has crossed
a certain threshold and make the
badger jump.
Keep in mind this threshold is
tunable and adjustable to your
app's specific needs.
Next, we'll take a look at the
second control, where bumping
the device downwards will make
the badger duck.
For this one, we want to measure
the user acceleration along the
gravity vector.
So we'll be looking at the user
acceleration property, and here
we chose that because even if
the device is slightly tilted or
rotated at an angle, the user
acceleration is going to still
look the same regardless of the
attitude.
So this is one of the gestures
again that we're detecting over
a window of time, so we'll use
the push mechanism as well.
Since we're already set up for
the push mechanism, we'll head
back to our motionHandler, where
we were storing those rotation
rates previously.
But this time, we'll put out the
gravity property out of the
DeviceMotion object and the user
acceleration as well.
We compute the magnitude of that
user acceleration along the
gravity vector and store that in
a buffer.
And once the mean of that buffer
crosses a certain threshold,
then we'll make the badger duck
down and hide in the cart.
So we've taken a look at two
controls where we were
monitoring motion across a small
window of time and we used the
push mechanism for that.
Let's take a look at our final
control, where our requirements
are slightly different.
For the tilt control, in the
simple case where the device is
held in a vertical position, you
can break down the gravity
vector into its x-component and
its y-component; and by applying
simple trigonometry, you can
arrive at the tilt.
But since we want this gesture
to be slightly more flexible,
we're going to break gravity
into its y-component; and its
component in the x-z plane of
the device.
This will allow us to tilt the
phone even if the device is
slightly rotated at an angle.
What's different about this
control is that we want to know
the current state of the device
and not a discrete motion that
occurred.
This will allow us to make the
badger lean at the same angle
the device is leaning in and
provide a very responsive
experience.
And for those reasons, we'll be
using the pull mechanism for
this control.
Since we're already set up for
the push mechanism, the
framework is ready for us to
pull for samples at any point in
time.
So we'll go back to our renderer
function.
If you recall, this is getting
called just before you render a
new scene.
That makes it a perfect place
for me to pull for the latest
sample, make a calculation, and
then update the graphics.
So we'll ask the MotionManager
for the latest DeviceMotion
sample, grab the gravity
property out of it, compute the
vector and the x-z plane, and
arrive at the tilt as the atan
between that x-z-component and
the y-component.
Don't forget to let the
framework know that you're no
longer interested in
DeviceMotion updates if they
were previously active.
This is a great thing to do when
your game is paused or has ended
to make sure you're not wasting
more battery than you need.
So here you see the results.
The user is able to play the
game by tilting the device from
side to side to make the badger
lean either way, and tipping the
device towards them makes the
badger jump up, and finally, a
small push of the device
downwards will make the badger
squat and hide inside the cart.
It's pretty cool, we've taken an
app with swipe controls and used
DeviceMotion to replace them
with motion-based ones.
I'm excited to see how far you
can push the DeviceMotion APIs.
[ Applause ]
So to wrap up some of the key
points that we talked about
today, we encourage you to look
at the authorization API and
check for your app's
authorization if you're using
one of the sensitive APIs.
While you might have your use
case for using the sensor data
directly, we encourage you to
look at the DeviceMotion APIs,
because our sensor fusion
algorithms handle the most
common case.
It eliminates those
environmental factors so that
you are able to focus on the
motion controls of your user.
The APIs provide you a smooth
and consistent experience across
all our supported devices, and
we've made enhancements this
release to make sure that when
you query for those updates, you
do that in a very
power-efficient manner.
Remember the two different ways
to query for updates: Use the
push when you want to detect a
gesture across a small window of
time, and use the pull when
responsiveness is key and you
want to know what the current
state of the device is.
Here are a couple of the
sessions that we think you might
be interested in.
Please check them out, and if
you're interested in learning
more about the Badger app, check
out last year's SceneKit
session.
If you follow this link, you'll
be able to get at the sample
code and other Core Motion
documentation.
We thank you all for being here,
and have a wonderful evening.