Transcript
[ Music ]
>> Hi everyone.
My name is Jeremy and I'm an
engineer on the tvOS team.
Today I'm really excited to
speak with you about what's new
in TVMLKit that we've added in
tvOS 12.
For the uninitiated, TVMLKit is
Apple's high level framework for
quickly building content based
applications on tvOS.
Out of the box, it is compliant
with our human interface
guidelines.
So your applications look
beautiful and feel correct.
It uses JavaScript to drive
application logic and a special
XML based markup language which
you define that renders into
user interfaces on the screen.
In fact, some of the apps you're
familiar with and use today are
built using TVMLKit.
Along with that, there are
thousands more in the app store
that you write.
So let's get started with the
enhancements we've made to
TVMLKit, beginning with three
things we want to talk about
today.
First, we have some enhancements
to Web Inspector that allow you
to further debug your
applications and introspect them
at a deeper level.
Since last year, we've added a
bunch of features and
enhancements to the framework
and I'm going to discuss three
of them with you today.
We're going to talk about new
Web Inspector enhancements for
better debugging, new data
binding constructs that are
added to make it more powerful,
and finally, a new way to
customize the playback
experience on TVMLKit.
Let's get started with Web
Inspector.
tvOS 11 introduced more support
for Web Inspector that allow
better introspection into your
TVMLKit apps.
Since then, we've enhanced this
support even further.
In tvOS 11.3, in addition to
showing install event listeners
on an element, you can now
temporarily disable them.
This vastly helps with debugging
as you can now toggle the event
handlers off and on at whim.
In the network tab, joining at
[inaudible] in document
resources are image resources.
This allows you to see the
images that are being loaded as
well as information about how
long it took and where time was
spent.
If you are interested in seeing
the actual image that comes off
the wire, this is now also an
option you have at your
disposal.
However, please note that this
only works after you've
connected the Web Inspector.
It does not show what has
already been loaded.
Finally, my most favorite
feature of all, the inspect
button.
Clicking this will show you the
approximately element that
represents the view that's
currently in focus.
If your elementary is collapsed,
Web Inspector will expand it to
the exact element and highlight
it.
To use Web Inspector, either
download the latest version of
macOS and install it, or use
Safari technology preview.
More information about how to
use Web Inspector can be seen in
our talk given last year using
Web Inspector with tvOS apps.
With that, let's talk about data
binding.
And before we jump into the new
features of data binding itself,
let me give you an overview of
what it is.
Data binding helps you easily
transform your data to user
interface elements through
expressions in your template
markup.
This is really good because it
allows you to decouple your data
and your lot -- and your layout
logic as well as your
application logic itself.
Because of this, data bound
templates can reduce the amount
of JavaScript code you actually
have to write to convert your
data to TVML documents, since
the framework does this on your
behalf.
In fact, by authoring your
documents on your behalf, it is
able to do it in the most
performing way so you don't have
to worry about the right APIs to
use.
Let's look at a practical
example to explain this.
Say you want to generate a
banner with a title and a
description.
And this is how you would
typically do it without data
binding itself.
First, you will fetch the
relevant data that's supposed to
be shown to the user in this
case, a title and a description.
And upon fetching, you pass it
onto a part of the JavaScript
code you've already written that
processes this data and
generates the actual final
document itself.
Now, with data bindings, you can
take out JavaScript processing
step and provide binding
specifications in your template
itself, and TVMLKit will
populate the final document on
your behalf as directed.
Effectively, your application
just needs to worry about
fetching and massaging data, and
not worry about dom [phonetic]
editing at all.
In a nutshell, that's how data
binding works and helps reduce
the amount of code you write.
Last year, we introduced the
concept of data binding with
some core concepts that include
binding of attributes of an
element, binding the text
content of an element, and of
course, binding items of a
section in either a shelf, grid,
or list.
Let's quickly recap on these
concepts with another example.
The corresponding data bound
template representation of this
image element would contain a
binding expression with the 'at'
symbol, followed by the
attribute name and the property
you want to bind it to.
Next, let's seek the example of
generating text content of an
element.
Like in this case, a title
element that has corresponding
data of what it should actually
be filled with.
The data bound template
representation for this title
element will contain a text
content binding and the property
it maps to.
Finally let's talk a little bit
about items binding.
It is a slightly different
binding and involves a group of
data pieces you want to show.
It also only works with sections
on a shelf, list, or grid.
In this example, we have some
data with the tree list items as
an array.
And the final representation
should be a section of tree list
item lockups.
The corresponding data bound
template for this section will
contain two things; an item's
binding and the property it maps
to, and also a prototype to
convert each data object
contained in the array.
In this case, it would be a data
bound template representation of
a list item lockup.
So that's the tree binding
constructs we've introduced in
tvOS 11.
For more information, please
look at our talk, Advances in
TVMLKit from last year's WWDC.
This year, we've extended that
vocabulary.
To begin, we've added a more
generic way of binding child
elements of an element using the
children binding.
And to help better organize your
DOM, we've added a couple of
special elements, namely
fragment and rules.
We'll go into this in detail,
starting with children binding.
Children binding is a more
generic form of items binding.
Items binding is optimized for
use cases where you have a
section of a shelf, grid, or
list in order to work
efficiently with large data
sets.
They can be used outside these
elements.
For everything else, use
children's binding.
And it's because of a very
simple reason.
It works by generating children
of a target element itself.
And it behaves the same way as
items binding does.
You require the use of
prototypes to define the
elements that data should be
converted into and this will be
used as a template when
generating the final DOM itself.
Let's take an example to explain
how this works.
I am going back to the same
example of tree items in an
array.
We have this very same piece of
data but in this particular
situation, we have three
different menu items.
And this would be used in a menu
bar.
This is the final representation
we expect to see.
Basically a menu bar tree, menu
bar items.
And this is a very simple way to
specify the template for it.
As you can see, it's similar to
how you would do items binding.
It has prototypes that's used to
map data to elements and it has
a binding expression.
The only difference is that the
element is expressed on
[inaudible] section.
Children binding works with any
element.
Now this works really well as
long as you want to generate all
the child elements itself.
There might be cases where you
want to only generate some of
those children.
Take, for example, an auto
streaming app with a now playing
menu item.
The now playing menu item is a
special menu item that should
always be in the menu bar but
would only appear if background
audio is currently playing.
However in this case, we still
want the menu bar items to be
data bound.
In order for this work, we need
to compartmentalize the data
driven and non-data driven parts
of the menu bar.
And that's where fragments comes
into play.
So what are fragments?
Fragments are invisible elements
that the renderer doesn't see
and it helps you organize your
DOM.
But what's special about
fragments are that its children
are visible.
And because it is an element and
children binding works with any
element, the fragments itself
works with children binding.
So let's go back to our data in
the final form we want and we
have something like this.
You have the menu bar items.
Nice to encapsulate in a
fragment and this is great
because it allows us to do
children binding like so.
Now all we have done is moved
the data around [inaudible] into
fragment while keeping the menu
bar intact with the now playing
menu item.
And because the renderer only
sees the children of the
fragment, this still renders as
a properly formed menu bar.
On the very subject of data
itself, that you can use to map
to user interface elements, this
is another likely scenario where
you have data that some do not
change, and some change all the
time.
For example in this particular
piece of data itself, it is for
a video that has a poster image,
a title, and a playback
progress.
In some situations, we want to
show different user interfaces
based on that information
itself.
In the case where playback
progress hasn't started, it's
naturally zero and we are
interested in showing the poster
image and the title for the
video itself.
But when we start watching the
video, progress will naturally
be greater than zero.
For that, we want to show the
same thing that never change,
the poster image and its title.
However to make things obvious
this video's being played back,
we want to show a progress bar
that's filled through the
playback progress' percentage
itself.
In the very same vein, we have
now two use cases where data is
different and we also want to
show two different looks of what
it is.
In the first case, we -- all we
have is an image and a title in
the lock up.
And in the second case, we have
the addition of an overlay and a
progress bar.
Typically you would have
application logic that outputs
different x amount based on that
data itself but with rules, you
can have a single statically
defined template that would give
you either representations of
the final document.
So what are the rules?
Well, rules used data states to
redefine your final document
which, in turns, refines your
UI.
It is an invisible element.
So the renderer doesn't see it
but it affects how the document
is authored.
Any operations within those
rules happen on sibling elements
that the rules are a part of.
And the best way to show this to
you is in another example of how
this can be setup.
So let's look at the rules that
we would use to structure the
very prototype that we want --
that we've shown as an example.
We will start by defining the
prototype as the lowest common
denominator of what our user
interface should look like.
And in this case, we have an
image and a title.
However, you would notice that
we also have a place holder for
the progress bar.
Placeholders are also special
elements that are invisible to
the renderer and in this case,
would be used by the rules as a
target for replacement when data
states match.
Now let's fill in our rules.
A group of rules that act on
sibling elements are
encapsulated in the rules tag.
Individual rules that match a
data state are encapsulated as
specialized elements, and
specialized elements become
active when it matches certain
data states.
And the way that that's matched
is based on a query in the state
attribute itself.
When data state matches, the
children of the special element
-- specialized element are the
list of operations that act on
the sibling elements of rules.
In this case, we want the
placeholder to be replaced with
an overlay element and its
children.
TVMLKit matches the elements to
be replaced by looking at the
tag attribute on any element.
It effectively does replacement
by first matching on that tag
and then comparing the element
name.
If the element name is
different, it would replace that
element hole and in this case,
placeholder becomes an overlay.
However, if the element name
matches, whatever that's new
would be appended to what's
already existing.
And so we have a single template
with rules that will generate
two different output based on
the state of the data that's
provided.
Effectively you can move your
application logic to deal with
how things are displayed into a
statically specific template
that exists within the context
of elements that's transformed
into user interfaces.
Now let's switch gears and talk
about playback in TVMLKit.
TVMLKit has long provided
extension points where you need
more customization of your user
interfaces, be it individual
views or even whole templates.
In tvOS 12, we are extending
this to our playback pipeline,
giving you control over playback
as well as its associated user
experience.
This experience works with all
the different playback styles we
have, whether it is embedded or
in a full screen.
You do this by providing a
TVPlayer object and its
associated user interface as a
UIViewController.
These have close analogues to
our JavaScript APIs.
So that would be less confusion
in talking to your JavaScript
developers.
And finally there's a limited
JavaScript Bridge that's
exposed.
It allows communications between
your native code and JavaScript
itself.
Let's talk about TVPlayer and
that's the base where you can
get your customized playback
experience working.
TVPlayer is a public AVPlayer
adaptor to the Playback
Pipeline.
What this means is that TVPlayer
effectively translates regular
AVPlayer callbacks into what
JavaScript expects.
TVPlayer is also the object that
you can use to dispatch custom
events to JavaScript and by
default, it already handles
everything that AVPlayer has as
playback events.
So anything extra is on you to
dispatch.
Changes that JavaScript makes to
the player are KVO observable.
So you know when things are
changed by your JavaScript
developers.
Finally the TVPlayer object
plays media in a sequential
fashion from the very first
[inaudible] item all the way to
the very last in its playlist.
Whenever a player is needed,
TVApplicationControllerDelegate
will ask for a TVPlayer and you
will have to return an instance
in order to participate in the
Playback Pipeline.
The next step of the Playback
Pipeline is to actually show
Playback on the screen itself in
the form of a user interface.
This can happen anytime whether
it's full screen playback or
embedded playback.
It is entirely up to you to
create your own user interface.
When TVMLKit needs a user
interfact, the TVInterface
[inaudible] will ask its
delegate for a view controller.
It will also pass it a reference
to TVPlayer that is responsible
for playing media in that view.
With everything, there are
several caveats to using
TVPlayer and its associated user
interface.
The very first thing is you
should handle any 'should'
events yourself.
These are usually tied to
interstitials and since that's
largely user interface, this
should be handled by the native
code that you write.
If you use FairPlay encryption
for your video playback, you
need to use AVContentKeySession
for secure key loading.
For more information about
AVContentKeySession, look at our
talks from last year, Advances
in HTTP Live Streaming, and a
talk from this year about
AVContentKeySession Best
Practices.
Finally, if your JavaScript
developers use overlay and
interactive overlays, this will
not work out of the box.
They are user interfaces and
because you are building your
own, you will have to handle it
yourself.
In summary, the changes we've
made to TVMLKit and tvOS 12 are
the following two: One is the
data binding is now even more
powerful, enabling you to build
data driven templates in any
form.
We highly encourage you to check
this out.
And finally, if you've been long
waiting to customize the
playback experience itself, you
can do it today by implementing
your own native playback
experience.
For more information about this
talk, please look at the
following url.
And thank you for attending WWDC
2018.
Thank you.