Transcript
[ Music ]
>> Hi, this is Tom from the Car
Experience Team at Apple and
welcome to our talk on Advances
in CarPlay Systems.
As you know, CarPlay is the
smarter, safer way to use your
iPhone in the car.
iOS 13 has a ton of new updates
to CarPlay including the all-new
CarPlay Dashboard for a simple
view of your tunes, turns and
Smart Siri suggestions.
In this session we will
introduce you to vehicle
integration features newly
available in iOS 13.
All of the features we will talk
about today will require the
vehicle system to adopt new APIs
available in the CarPlay
Communication plug-in R15.
Today we will introduce new
display and audio features in
CarPlay that you can take
advantage of in future vehicles.
First up, let's look at how
CarPlay supports irregularly
shaped displays.
The vehicle declares the
physical size and resolution of
the display and iPhone will
setup an H.264 video stream of
that size which will be rendered
to the display.
Prior to iOS 13, the CarPlay UI
would be drawn up to each edge
of the video screen.
However, some car displays are
fitted with pieces of trim that
may obscure parts of the CarPlay
user interface, restricting
access to some touchable
controls and preventing the user
from being able to see certain
parts of the UI.
With iOS 13, CarPlay now
supports irregularly shaped
displays.
First, your system defines a
view area.
This represents the rectangle
that CarPlay can draw a user
interface image into.
A view area can also contain a
safe area.
The safe area is also a
rectangular area and must be a
subset of the view area.
The safe area represents the
portion of the display that will
contain interactive content.
This includes user interface
elements such as buttons and
lists and important status
information like time, signal
strength and route guidance
information.
iPhone will draw interactive
content within the boundaries of
the safe area.
Modern car displays can come in
many different shapes and sizes.
Displays themselves may be
non-rectangular or portions of
the display may be occluded by
other elements.
With iOS 13, CarPlay can use
safe areas to better support
non-rectangular shaped displays.
First, the system defines a view
area.
The view area rectangle in this
case would be made up of the
widest and tallest points of the
display.
This allows the CarPlay image to
render to all parts of the
display.
Then a safe area rectangle will
be declared for the part of the
display where a CarPlay UI is
guaranteed to be visible and
interactable to the user.
And that was our update on
irregularly shaped displays.
Next up, let's talk about second
screen support in CarPlay.
Prior to iOS 13, iPhone provided
a single H.264 video screen to
display the CarPlay UI on the
center console display.
This provides a great way for
users to set a navigation
destination, interact with music
apps and more.
However, this limits the user's
CarPlay experience to the
vehicle's main display.
Many vehicles have rich displays
directly in front of the driver
to show important information in
the form of an instrument
cluster.
In iOS 13, we now support
multiple H.264 video streams to
display CarPlay content in the
vehicle's instrument cluster.
This is a great way to display
additional information from
CarPlay apps directly in the
instrument cluster.
And different types of
information can be displayed in
different areas of the
instrument cluster using more
than one video stream.
In iOS 13, CarPlay will support
a maneuver instruction card
provided by Apple Maps, as well
as a map view which can be
rendered in parallel with the
instruction card in two separate
video streams.
Each video stream will have an
independent night mode, allowing
the vehicle system to specify an
appearance mode for different
content types.
Also, the vehicle chooses what
type of content is shown in each
stream.
For instrument clusters, car
play will only draw the type of
content the vehicle has
selected.
But please note for heads-up
displays, information from iAP2
Route Guidance will continue to
be used.
So with this in mind, let's look
at how to best utilize the
different available protocols to
integrate CarPlay into a vehicle
instrument cluster.
Users can be provided with
simple status information from
apps in CarPlay such as
navigation, audio and phone
calls.
Firstly, map or turn-by-turn
information provided as a video
from iPhone can be displayed.
Additional information about the
navigation route can also be
provided as metadata using the
iAP2 Route Guidance feature.
This, like other iAP2 features,
requires the vehicle to draw the
metadata in its own native UI.
The currently playing track or
podcasts can be rendered using
iAP2 Now Playing along with any
information about ongoing phone
calls using the iAP2
Communications feature.
Control of those phone calls can
be supported using iAP2 Call
Controls, allowing the system to
offer specific user actions
during phone calls like mute,
swap call and end call.
Displaying CarPlay content in
the instrument cluster is
another part of the vehicle
system that benefits from using
safe areas and view areas.
In this example, the view area
would be the rectangle between
and behind the virtual
tachometers.
While aesthetically it may be
desirable for CarPlay content to
fill this region, some of it
would be obscured by the virtual
tachometers.
So a safe area would be declared
in the area between the two
tachometers and CarPlay will
position the important
information where it will be
visible to the user.
And multiple streams can be set
up to display different types of
content from CarPlay within the
instrument cluster.
For example, the second stream
could be a map view and the
third stream an instruction
card.
Displaying content within a
virtual tachometer will also
make use of view areas and safe
areas.
The rectangle containing the
entire circle would be the view
area, and the rectangle
contained entirely within the
circle is the safe area.
Again, CarPlay will position the
important information where it
will be visible to the user.
So that was our update on
support for second screens.
Now let's take a look at the
last of our new features that
affects displays in CarPlay,
dynamic screen resizing.
Prior to iOS 13, the size of the
CarPlay image remained fixed
while the CarPlay session was
active.
Now the size of the CarPlay UI
can change on the fly at the
user's request by declaring
multiple view areas for that
display.
This is ideal for systems that
support configurable horizontal
split screens as shown here, and
larger portrait screens that are
normally divided into distinct
context-specific regions.
CarPlay can provide an Always
Available button for the user to
trigger a resize request from
within the CarPlay UI.
This is useful to show CarPlay
in a full-screen configuration
without reserving space in the
native UI for dedicated control
for the user to exit that
configuration.
Resizing is not only supported
on the center console display
but also on the instrument
cluster as well.
So let's use the instrument
cluster from the previous
section as an example of how
dynamic resizing could be used
in CarPlay.
Let's say this instrument
cluster offers the user a way to
move the virtual tachometers
into one of two preferred
positions.
For this to work well, the
CarPlay image needs to be able
to adjust its size based on
where the virtual tachometers
are positioned.
Multiple view areas can be
declared for this display to
make this happen.
Let's declare the first view
area for the widest
configuration and include a safe
area as discussed earlier.
For the narrower configuration,
a second view area would be used
with its own narrower safe area.
The vehicle system can also
specify how long the transition
between those configurations
should take, allowing the
CarPlay content to transition
sizes in the same timeframe.
Let's take a look at how this
transition would work.
The user starts in the wider
configuration and uses some
native UI control, possibly a
steering wheel button, to begin
the transition.
During this three-second
transition, CarPlay is telling
the native system the size of
the encoded image in the H.264
frame, allowing the vehicle
system to synchronize the
position of its own tachometer
UI elements.
So in summary, dynamic resizing
is supported on center console
and instrument cluster displays.
A UI resize can be triggered by
the user either through the
native UI or a button within the
CarPlay UI.
The vehicle system declares how
long transitions between view
areas should take.
And CarPlay provides an update
on how far the view area
transition it is so the vehicle
system can synchronize
transition animations in its own
UI.
So that's all for our new
display-related features.
Thanks for listening so far.
Now over to Tanya for an update
on Siri in CarPlay.
>> Thanks, Tom.
My name is Tanya Kancheva and
I'll talk to you about enabling
Hey Siri support in future
vehicles.
So what do we mean when we talk
about supporting Hey Siri in
your car?
First, of course, the driver can
launch Siri completely
hands-free without needing to
press a button or take their
eyes off the road.
And since Hey Siri is detected
using the car's microphone,
iPhone can be out of sight, left
in a pocket or even in the
trunk.
Siri will still be available.
We also updated audio handling
so that activating Siri through
a button press is instantaneous.
Just hold the Siri button on the
steering wheel and say your
request without waiting for an
audible trigger to speak.
Siri will be back with your
answer immediately.
Lastly, we looked at how Siri
and Music interplay in the car
and enable the smoother and more
familiar experience.
Music continues to play in the
background while interacting
with Siri.
Here is a system overview for
audio handling in CarPlay.
Some of the components should
already be familiar to you, but
let's go ahead and look into the
details.
To support Hey Siri, the in-car
microphone needs to be always
active and undergo continuous
voice processing.
The echo cancellation module
runs all the time and removes
any audio playing through the
car's speakers as well as
reduces background driving
noise.
The process microphone data is
sent to an audio ring buffer
within the CarPlay communication
plug-in which holds a couple of
seconds of historical audio.
This buffer is stored within the
car until there is a trigger to
send it to iPhone.
So what is this trigger?
In fact, there are two modules
which can lead to audio exchange
with iPhone.
Both constantly analyze the
process microphone data.
One can trigger when the drive
says Siri.
That's the keyword detector.
The other one triggers when the
drive starts talking.
That's the voice activity
detector.
Both detectors must be available
in the car as iPhone determines
which one is used for a
particular scenario.
Now let's say the keyword
detector triggered so the car
notifies iPhone through the
communication plug-in.
iPhone opens a new audio stream
called Aux In, requesting some
historical audio data to be
sent.
The received audio is reanalyzed
by a voice trigger detector on
iPhone which verifies that the
user said, "Hey, Siri."
The second pass detector uses
offline keyword detection on
iPhone similar to how Hey Siri
is detected when the user is not
driving.
Only after the second pass
detector on iPhone triggers Siri
is activated to process the data
and respond to the user.
Now, let's look at how this
architecture is helping us
improve triggering Siri with a
button press.
As we already saw, the mic, the
ECNR unit and the audio ring
buffer are processing audio
within the car system all the
time.
Now the user presses the Siri
button on the steering wheel
that is the trigger for iPhone
to request audio data from the
time when the user pressed the
button.
Since this is the button press,
Siri activates immediately.
And as the buffered audio is
sent faster than real-time, Siri
can be ready with a response
right after the user stops
talking.
Okay, now that we have discussed
the three ways in which Siri can
trigger from the car through a
detected keyword, voice activity
or a button press, let's see
what happens when Siri is active
and helping the driver.
Typically, music will be playing
in the car.
In this case, let's say that
music is coming from iPhone.
The car will receive the audio,
mix it and play it through the
speakers.
In the past, when Siri started
in CarPlay, music had to stop
and there was a bit of silence
before Siri prompts played.
With this new architecture,
media playback is separated from
Siri prompts so the music can
keep playing in the background.
When Siri launches, an
additional audio stream
dedicated to Siri output, Aux
Out, will be opened.
The car will then mix the two
streams, reduce the volume of
the music as needed and play
Siri prompts quickly and without
big interruptions.
But please don't forget that in
addition to Siri prompts, there
might be active route guidance,
so the mixer has to be able to
mix three parallel audio streams
coming from iPhone.
All right.
Now that we have gone through
the details of the audio
architecture, let's quickly
summarize what is needed within
the vehicle system.
First, a microphone which is
always processing audio in your
system.
Then, an echo canceller and
noise reduction module to clean
up microphone input.
A detector that triggers when
the driver is talking.
A detector that triggers when
the Siri keyword is heard.
The ability to mix three audio
streams from CarPlay.
And that's how our update on Hey
Siri.
Thank you for your time.
For more information, check out
developer.apple.com.
And enjoy your drive with
CarPlay.