Transcript
[ Music ]
>> Hello. My name is Tanya,
and I want to welcome you
to the second session about
developing CarPlay systems.
The video to Part 1
is also available.
So if you haven't watched
it, definitely check it out.
In this session, we'll
cover more detail
on how CarPlay is integrated
into a typical automotive
infotainment system.
We'll start with a system
overview, then talk about volume
and resource management,
and finish by discussing
application state management.
Now, let's dive into an
overview of the system.
If you watched the first video,
you will recall the
software components you need
to implement a CarPlay receiver.
In summary, you need an IP-based
link to the head unit either
over USB or over Wi-Fi, IP
cline for data exchange,
an instance of the CarPlay
Communication Plug-in
implementing the control
protocol, audio framework
to play or record sound,
and a video framework
to render the video stream.
However, those components are
just one of the sub-systems
in a typical automotive
head unit.
In addition to the
CarPlay functionality,
your unit has its
own native interface,
audio sources, and logic.
Both together build
the complete head unit.
Then you add all hardware
resources within the vehicle:
Microphone, speakers, displays,
sensors, user input elements,
and you have the
complete system.
OK, so we have the full system.
Now the question is how do those
seemingly competing sub-systems
work with each other?
Let's understand it.
First, let's look at how
sensor data is exchanged.
And we will take location
data as an example.
Location information includes
data from a GPS receiver
or sensor data like
wheel speed or yield.
The data reaches the
device through iAP2,
must be always available,
and is requested
by the phone when needed.
Next, let's look
at the instrument cluster
or heads-up display.
These secondary displays
are used
to show metadata
provided by the device.
Such metadata could include the
currently-playing music track,
the active phone call,
or starting with iOS 10,
the next turn-by-turn direction.
Again, the data is fired
through the IP protocol
and exchanged on demand.
Now let's see what happens
on the primary display.
Of course, to operate the
content shown on the display,
the user must either
use a touchscreen,
an app controller, or both.
So we can consider
the availability
of the user input devices
together with the display.
When the CarPlay UI is active,
the unit is rendering the video
stream on the primary display.
And both the screen and the user
input are used by the device.
But if the native user interface
is shown on the screen,
then both the display and
the user input are routed
to the native sub-system.
However, additional user
controls on the steering wheel,
like the Voice Recognition
button, Next/Previous Track,
may be linked to CarPlay even
when the native UI is
shown on the display.
That way, Siri can
be always launched
That way, Siri can
be always launched
or the next audio track played.
OK, now that we looked at the
user interface integration,
let's talk about audio.
Audio in the car can be roughly
separated into three categories.
The first one provides access to
the speakers and the microphone,
and it's used for phone
calls or voice recognition.
Then we have audio playback
for any media content
like radio or music.
And last, we have the alerts
category which is used
for navigation prompts or any
other higher-priority alerts.
Within the native sub-system,
these three categories
are to be found again.
Within CarPlay, we have
a Main audio channel
which includes phone
call, voice recognition
and audio playback audio.
And an Alternate audio
channel which is reserved
for higher-priority
prompts and sounds
such as navigation prompts
or new message notifications.
So the media channel can either
play music coming from the phone
or media playback from one
of the native audio sources.
OK, now that we have
looked at media playback,
let's see what happens
when there is a phone
call through CarPlay.
The phone call needs the audio
channel which provides access
to both the microphone
and the speakers.
So media playback is interrupted
and the channels are switched.
And audio switched over to the
channel, which provides access
to both the microphone
and the speakers.
In addition to switching
to audio channel,
In addition to switching
to audio channel,
the screen switches to CarPlay
to show the phone call UI.
And metadata information
on the instrument cluster
is updated as well.
Of course, if the native system
also supports phone calls
outside of CarPlay, then that
phone call is played and shown
on the in-car displays.
But while on a native
phone call,
the user can be using
Apple Maps for navigation.
So each time there is
an approaching turn,
Apple Maps will play a sound
through the announcement
channel to notify the user.
The phone call continues to
be routed through the speakers
of the car, but the turn-by-turn
notifications are also displayed
as they are mixed
with the call audio.
Of course, in such situation,
we would not want the user
to hear the phone navigation
prompt, but just an alert sound
to hear the phone navigation
prompt, but just an alert sound
that an upcoming turn is coming.
And if the user wants to
see where the next turn is,
the main display
may show Apple Maps.
And last, after the phone call
ends, music continues to play
through the native sub-system
as that was what the
user was listening
to prior to the phone call.
Now, the navigation announcement
will include the full-spoken
direction as the user is
no longer on a phone call.
All right.
As you see, integrating CarPlay
is rather a complex task,
especially in regards to
sharing hardware resources.
But before we get
into those details,
let's look at volume controls.
Volume management.
As you might know, a usual
automotive head unit shows the
As you might know, a usual
automotive head unit shows the
volume indicator while the user
is operating the volume knob,
and that volume knob
operates the volume
of the currently-playing audio.
The same principle
applies for CarPlay audio,
but there is a different
volume level setting for each
of the major CarPlay
applications.
Let's look through them.
During Siri playback, the volume
knob sets the volume specific
to voice interactions.
For an incoming phone call,
the volume of the
ringtone is controlled.
When the user is
on a phone call,
the phone call audio
is controlled.
And if a next turn
is approaching,
navigation prompts are played.
Then, the user can change
the navigation volume.
And remember, media could
be playing in the background
but its volume will be
lowered during those prompts
and cannot be adjusted.
Once the announcement is over
and music volume is ramped up,
the button controls
the media volume level.
All right, now let's look
at resource management.
We are going to take a look at
which resources are managed,
talk about how they are managed,
and go over a couple
of typical examples.
So we saw that there are
multiple hardware resources,
but let's understand which
one of them are managed.
And there's only two:
The mainScreen representing
the main display in the car,
and mainAudio as the
resource giving access
to the car's audio system.
Those resources can either
be taken or borrowed.
When you take a resource,
it belongs to you
for unlimited amount of time.
It's basically yours.
When you borrow a resource,
you may use it for a while,
but you have to give it
back after you're finished.
but you have to give it
back after you're finished.
OK, so let's have the
native user interface take
over the display permanently.
The user can be using native
navigation, or listening
to FM radio, or adjusting
some car settings.
What triggers such permanent
switch to the native UI?
The native UI can
take the screen
when the user presses a hard
key or switches to the native UI
from the entry point
within the CarPlay UI,
or by using the native
voice recognizer to call
out a specific native
application.
Now let's say the user
selects the Apple CarPlay icon
from the main menu.
In this case, CarPlay
takes over the display
as the user explicitly
requested it.
Again, what can trigger
such permanent transition?
Any hard key linked to CarPlay.
Or, as we saw, through
any CarPlay button
within the native UI.
Or through Siri, for example,
by saying, "Open Maps."
Now, there are other
applications
that need to borrow a resource.
Remember, they just need it for
a while and will give it back.
Which applications
are we talking about?
That's phone calls,
voice interactions,
notifications, or alerts.
Let's look at an example.
The native UI has taken
the Main Screen resource
and is permanently
showing on the display.
Now the user gets a
call through CarPlay.
The CarPlay UI borrows
the screen and is shown
for the duration of the call.
Once the call ends, we go back
to the native UI and stay there
until further user action.
Then, let's look at the
Main Audio resource.
Main Audio can be split
in four major types.
Each type is used for a
different CarPlay application
as it provides access to a
different hardware resource.
Media is audio output only and
is used for any media playback.
Alert is also only output
and plays ringtones
and timer alerts.
speechRecognition
is used for Siri
as it adds access
to the microphone.
Same for the telephony type,
which is used for phone calls.
And same for the default, which
is used for undefined audio.
But don't forget the
second audio channel used
for navigation announcement.
alternateAudio is not managed.
It is basically always
available, so there is no need
to take it or borrow it.
alternateAudio is mixed
with any of the audio types
within the mainAudio channel,
and it is always accessible.
And with this, I'm going
to hand off to Tom to talk
about the resource manager.
>> Thanks, Tanya.
Hi. I'm Tom, and I also work on
the CarPlay Engineering Team.
So now that you've learned which
resources need to be managed
and you know you can either
take or borrow those resources,
let's talk about how you
go about managing them.
To be able to distribute
the resources
between the two sub-systems, we
need some type of arbitrator.
And we call such an
arbitrator a resource manager.
So what does the
resource manager do?
It has three main tasks.
First, it holds the current
state of the whole system.
Second, it follows a set
of strict rules to decide
Second, it follows a set
of strict rules to decide
which system will
get the resource.
Third, based on the current
state and the set of the rules,
it assigns the resource
to one party or the other.
So let's see how this
works in practice.
Let's say both the
native UI and CarPlay need
to show something on the screen.
So both send a request to
have ownership of the display.
Then, the resource manager
checks the internal state
and decides which of the
two should get access
to the display.
Let's say, in this case,
the native UI request
had a higher priority
and the screen is
assigned to it.
The resource manager
sends a notification
that the screen can now
be used by the native UI.
Only then the native UI can
show content on the screen.
It's important to note that
this state is not changed
until the resource
manager has sent an update.
So where does the
resource manager live?
Is it part of the native
system or of CarPlay?
Is it part of the native
system or of CarPlay?
Well, when we designed CarPlay,
we asked ourselves
the same question.
We considered the complexity
of exchanging detailed
information about the resources.
We considered the future.
What, if any, of those
rules needed to be adjusted?
What if new applications became
available through CarPlay
that the vehicle had no
existing designs to handle?
What solution would give us the
greatest flexibility five years
down the road?
So we decided to implement a
system where the complexity
on the native system is lower
and it's easier to update
after the vehicle has been
in the customer's
hands for a while.
Hence, the resource manager
is implemented within iOS.
And because the resource
manager is the component
with which the native
system interacts,
we refer to the iPhone
as the controller,
and the head unit
as the accessory.
But don't forget, all
of the iPhone applications
request the same resources
in the same way as
the native UI would.
Next. What commands
can you use to interact
Next. What commands
can you use to interact
with the resource manager?
It's simple.
You only need two commands.
The changeModes command
is used to request
or release a resource, and the
modesChanged command is used
to describe the current states.
changeModes is the notification
sent by the head unit,
the accessory in this case,
to the resource manager,
the controller.
This changeModes command
states what the accessory wants
to do with the resource.
It declares why it needs
it and who can take
or borrow the resource
after it is transferred.
The modesChanged is
the notification sent
by the controller
back to the head unit.
modesChanged provides
the current state
which describes who's the owner
of the resources of the system.
It sends so the accessory knows
if a resource has
been transferred
and the owner has changed.
Now we'll talk about how
resource management works
in more detail.
We will start with
the simple action
of switching audio
from iOS audio to FM.
Then, we'll talk about
native voice recognition,
then how to handle
a backup camera case
where you would not
want iPhone apps
to interrupt you unless
we walk through an example
where Siri triggers
music playback.
So let's look at
playing FM radio.
Let's say iPhone
music is playing
through the car speakers, then
the user wants to play FM radio
through the native system
and it's the Radio button.
The unit sends a changeModes
request to have ownership
of the speakers,
and it's taking it
as the user might continue
listening to FM radio
for a prolonged period of time.
The controller assigns
the audio to the head unit
and it sends a modesChanged
notification.
And the head unit is the new
owner of the audio resource,
so it can now start
playing FM radio.
The mainAudio resource
is permanently assigned
to the head unit at this point.
So, in summary, this
example showed us
when to take a resource and that
the native system should not use
that resource before it is
the owner of that resource.
Let's look at the next example
using native voice recognition.
FM radio is still playing
from our last example.
Let's see what happens
if the user triggers the
native voice recognizer.
The unit requests to temporarily
own the display and audio,
so the transferType
should be set
to borrow the audio and screen.
Both the mainScreen and the
mainAudio are transferred
to the accessory.
And the native voice
recognizer starts.
Once the voice dialogue
is finished,
the head unit returns
the borrowed resources
by sending an unborrow command.
And since the unit was
playing FM radio before,
the resource is again
assigned back to the head unit
and FM radio can resume.
Now you may be wondering,
"Why would the accessory need
to borrow the resource if
it was already the owner?"
Well, that's a great question.
Whenever the resource manager
needs to evaluate a request
to change the ownership
of a resource,
it must know what the current
state of the system is.
The resource manager needs
to know why you're
using the resources
so that they can take
the right decision
if somebody else needs
the resource later on.
Let's move on to
our next example,
showing the backup camera.
Let's say CarPlay
audio is playing
and the native UI is
showing on the screen.
When the user switches
into reverse gear,
the unit borrows the screen.
However, notice it also
tells the resource manager
that the screen cannot
be borrowed again.
The resource manager assigns
the display to the native UI
The resource manager assigns
the display to the native UI
and also notes the limitation.
So the backup camera
is now shown.
CarPlay audio continues to play,
but now the user
gets a phone call.
The iPhone cannot show
any content on the screen
as the head unit has
constrained the access,
but the ringtone plays
through the speakers.
So how did that happen?
With the changeModes command,
the unit told the resource
manager which rules to apply
by the resources
in its possession.
By setting the borrowConstraint
to anytime,
the unit allows the
application to be able
to borrow the resource.
By setting it to user-initiated
only events triggered
by the user can borrow.
With never, no application
can use the resource,
as with this example.
As you see, the constraints
have a major impact
on the system behavior,
so it's very important
that they are used properly.
Only restrict access
if there's a need
for immediate user attention.
Do not restrict permanent
ownership
when CarPlay is connected.
So that was using backup camera.
And the last example we want
to look at is what happens
when Siri plays music.
Again, we start with
FM radio playing.
The user triggers Siri.
Siri is launched on the phone
and the resource manager
tells the native system
that the resources have
been assigned for Siri.
Siri shows up and the user can
ask, "Siri, play some music."
Siri returns the display
and audio resources,
but the music app now needs the
audio resource to play music.
So the resource manager notifies
the unit that mainAudio is now
with the phone, but the screen
returns to the accessory.
The head unit does not
continue to play FM radio,
and so iPhone audio
is heard in the car.
So what did we see here?
On two occasions, the resource
manager changed the state
without a request
by the head unit.
And this is completely
fine because the apps
And this is completely
fine because the apps
on the device request
resources as well.
And if they get them, the
unit is notified about it.
So, how to handle this?
First, after each modesChanged,
you check if you're the owner.
If you are, resume
your activity.
Otherwise, you should wait
for CarPlay to take action.
Second, do not ignore
resource switches to the phone
when they are triggered
by the phone.
OK, so far, we've looked
at examples showing how
resources are being exchanged.
But there is another
situation where both sides want
to accomplish the same task.
So let's look at how
applications are managed.
We need to manage native
and CarPlay applications
which have similar features.
But what are those applications?
They are active route
guidance, phone calls,
and voice recognition.
We achieve those
using appStates.
But what is an appState?
We use different appStates to
record who is currently engaged
in the shared application
that can only be active
on the accessory or the
controller at any given time.
So the three different
appStates are TurnByTurn,
PhoneCall, and Speech.
So let's have a look at how
that route guidance
appState is managed.
When turn-by-turn navigation
is started on the accessory,
it uses a changeModes command
to update the TurnByTurn
appState to be active.
With this example, it is
a last in win situation.
So the controller grants
the TurnByTurn appState
to the accessory, returning a
modesChanged command showing the
accessory as currently
running route guidance.
Now let's look what happens
when the user sets a new
destination using Apple Maps
while the native system
is already guiding.
The user says, "Siri, take me
to the closest coffee shop."
Now, the iPhone has
started route guidance.
So the controller updates
the TurnByTurn appState.
This is the instruction
for the accessory
This is the instruction
for the accessory
to stop its own native
turn-by-turn guidance.
Now, the iPhone has
started route guidance,
so the controller updates
the TurnByTurn appState.
This is the instruction
for the accessory
to stop its own native
turn-by-turn guidance.
So now, the Apple Maps
turn-by-turn directions are
shown on the display and
announcement played over audio.
And there will be no conflict
with the native guidance system.
So now when the Apple Maps
turn-by-turn directions are
shown on the display and
the announcement is played,
there will be no conflict with
the native guidance system.
So that was route guidance.
Now let's look at a phone call.
If the unit supports
a second phone
over Bluetooth Hands-Free
Profile
and there was an
ongoing phone call,
the call uses mainAudio
exclusively.
The phone call cannot
be interrupted.
So what happens if the
user gets a phone call
on the CarPlay device?
The CarPlay call rings
only on the device
as it should not interrupt
the ongoing conversation.
as it should not interrupt
the ongoing conversation.
This is managed using
the PhoneCall appState.
And lastly, let's look
at voice interactions.
If the unit has a native voice
recognizer when it's running,
it has borrowed the resources
and is exclusively using
the speaker microphone
and screen resources.
But it can be interrupted at any
time by user-initiated events.
So if Siri is triggered
by the user,
native voice recognition
ends and Siri continues
on the in-car display.
The Speech appState is
used to achieve this.
So, in summary, CarPlay
relies on the same resources
as your native system and
is designed to coexist
with your native
user experience.
For a great CarPlay experience,
consider resource handling
for each use case and follow the
CarPlay design recommendations.
The CarPlay specifications
are available
through the MFi Program.
If you haven't watched
Developing CarPlay Systems,
Part 1 yet, I'd encourage
you to go and check that out.
For more information
on this talk,
For more information
on this talk,
please visit the
URL on the screen.