Transcript
>> Hello, and welcome to part 2 of Delivering
Audio and Video Using Web Standards.
My name is Jer Noble.
I'm a Safari and WebKit engineer, and I'm just going to pick
up right where Vicki left off from part 1, specifically,
last part of the session we covered basic
element playback in your webpage,
specifically how to add using the video
tag, a element to your site
and a few JavaScript functions to
control playback of your media.
And from there, we're going to take all of the things
she talked about, build upon them and add a whole lot
of more additional functionality to
the element in your webpage.
So, where we left off?
Just a quick recap for people who may have missed the last
session, the adding of element to your site is
as easy as the video tag or the single source attribute.
With this alone, you get HTML5
video embedded in your webpage.
But for additional functionality, if you want to let your
browsers choose between the most appropriate video content,
adding multiple source tags to your
video tag will allow you to do that.
And finally, controlling the video playback
is as easy as a Play and Pause of it,
or Play and Pause function on your element.
So, that's where we left off.
The last demo was a video with a big Play/Pause button
and, while, that's cool and all, we're going to take all
of those stuff and pump up to awesome,
the video playback experience.
So we're going to add or we're going to fill out
the custom JavaScript and HTML and CSS controllers
so it reproduces most, if not all of the
functionality of the built-in controller.
Once that's done, we're going to add some
subtitles and closed captions to your video.
We're going to cover how to add bumper clips to your video
within playback, and we're also going to talk about how
to take the video from your website
and embed it someone else's website.
So step 1, so we have a big Play/Pause button.
Something that exists in the built-in controllers
that you might want to add through JavaScript
and CSS is a progress meter, so you might
be asking, "What's a progress meter?"
Well, you have your big Play button,
you have your piece of video.
The progress meter would be something
like this, that when the user hits Play,
[background sound] it starts playing
the video and the time continues.
I know it's really cute, you'll be seeing a lot of it.
Anyway, so, to implement this JavaScript based
playback meter, you'll need a few pieces of API.
First of all, there's a timeupdate event.
Now this is fired occasionally, 4 times
a second or so, during regular playback,
when you're playing back at a constant rate.
Also, it gets fired more continuously as the user perhaps
changed the time with the built-in timeline controller.
Once you catch this timeupdate event, you'll
need to query the currentTime attribute
to pull the current time out of the video.
And to calculate a percentage of the way through the
video that your playback has currently progressed,
it will pull the duration attribute
out of the video as well.
Now, there's an additional attribute called
startTime that for almost all cases will be zero.
So for normal playback, you got a
media file on a server somewhere.
You put it in a element, the startTime will be zero.
The cases where this is not the case is in HTTP live
streaming, because the startTime is going to change based
on when the user starts playing their content.
And as Vicki described in the first session, there's the
durationchange event that fires when the browser notices
that the duration of the video has changed.
Again, in most cases, the durationchange
event will only get fired once.
The cases where it doesn't or it gets fired
mobile times is again, like HTTP live streaming.
OK, so to implement this in JavaScript, first
we're going to make an updateProgress function.
We're going to ignore the start time and
assume it 0, and calculate the percentage based
on dividing the current time of the video by its duration.
And again, we're going to take the simplest case of
representing the progress meter by stuffing that percentage
in the style attribute of a div called progressDiv.
Now this is the most, like again, the simplest
case, but you could represent the current time
as a string placed anywhere on your website.
You could do it as a pie chart, a bar
chart, the possibilities are endless here.
So, we'll catch the timeupdate event from the
element and call updateProgress occasionally.
And to cover our bases, we'll also
listen to the durationchange event.
And when that changes, recalculate the
width of the div, and it's simple as that.
So you have working progress meter.
Something else that's in the built-in controls that you
might want to add to your custom controls is the way
of showing how much of the video has loaded.
In part 1, there was an indeterminate
progress meter spinning,
but we're going to show you how to add a determinate one.
So-- but why should you care?
Again, Vicki kind of covered this in the first part of this.
But if you don't display "what" ...
to the user the state of their movie,
whether it's loading or not, what it's doing.
They might get bored waiting for a video to start playing
if they're on a slow connection and it's a big movie.
So, you can add an indeterminate state, but
we're going to show you a way of doing a--
showing how much of the video as a percentage is loaded.
And again, if you're a good internet citizen and you
have a page full of videos, and you don't want them all
to start loading at once to-- so that the first one the
user clicks on loads faster, you will have like a Click
To Play button like we showed you last
time, and it might look something like this.
User clicks the Click To Play, the movie
starts loading and at some point in the future,
the user says, "Oh, I can play, awesome!"
[background sound] and the movie plays, awesome.
You know, I just can't get enough of that.
[Laughter] Anyway, OK.
So, how would you implement this
script to your other script?
Well, an event that's fired occasionally
by the video element is the progress event.
It's fired no more often than about 1/3 a second
and no less often than every byte that's loaded.
Hopefully, you're not getting progress
events for every byte that's loaded.
It indicates your user is on a very slow connection.
But you'll catch this event and then query
the buffered attribute of the video element.
Now the buffered attribute is implemented
in terms of this type called TimeRanges.
It is a single object even though it's a plural name.
It is like an array but not quite an array.
It has a linked attribute and a couple of functions
called start and end that allow you to pull out of
that TimeRanges object the start and end points
of a range of time that the browser has loaded
and has a data available for immediate playback.
Now these ranges could be discontinuous.
The user could have jumped ahead of the downloading stream
and the user or the browser started downloading a new area
of the movie, but they'll always be
nonoverlapping and they'll always increase in time.
You can be guaranteed that the last
item in the timeranges object will be
after the first item, and every item in between.
So, to implement this piece of JavaScript, we're
going to have a function called updateLoadState.
And for our purposes, again the very simple case,
we just want to know what the maximum allowed time that's
been loaded, and so we'll check to see if there is any area
of the film that has currently been buffered,
grab the last item, ask it for its end time,
and use that as the maximum loaded value.
And just like in the progress example,
we'll calculate a simple percentage
and set that as the width of a
div somewhere else in our webpage.
OK, listen for the progress event, and every time the
browser tells us that it's loaded a little bit more
of the movie, we'll update the load
state of our custom controller.
OK, so you have a progress meter, you have a load state.
Something else that's in the built-in
controls that you might want
to re-implement is a Scrubber a Scrubbing
slider, and why do you need a Scrubber?
Well if-- like most people, they'll have favorite
parts of your movie and you might want your users
to jump to any random point in the timeline.
And as long as your server supports byte range requests, the
browser can immediately pick up where your user requested
at a current time and start loading times past that.
So, just like in the progress event, you'll need--
or the progress meter, you'll need the currentTime
attribute this time to set the current time
and cause the browser to jump to a specific point.
There's also the duration attribute which like in the
first example, we'll need to calculate a percentage.
And there's another attribute called seekable.
Now, when buffered, it tells you what parts of the
video are currently loaded, but there's no restriction
on jumping past the end of the loaded state.
The browser will notice, it will start downloading a new
portion of the movie, except for the seekable attribute.
For most cases, seekable will define
the entire range of the video.
There'll be no restrictions on where the user can jump.
In special cases like HTTP streaming,
this attribute will contain a range--
or a set of ranges of time which define where
the available data lies in your video content.
Now-- and when you're live streaming for example,
30 seconds in the future hasn't been filmed yet,
so there's no way for the user to
jump past like what is considered now.
And perhaps the user has thrown away all the data from 30
seconds ago and won't let the users scrub back to there.
In that case, the seekable attribute will
define a very narrow range of time that you want
to make your slider the min and
maximum amounts on that slider.
Now, it's reasonably difficult to implement
a slider with straight HTML and CSS.
However, new in HTML5 is a new type of input
that is of the type range normally rendered
as like a normal slider from whatever your OS is.
However, in Safari and WebKit, this input element
is completely styleable through CSS.
So we're going to use the input range element
to allow the users to control the portion
of the movie that they're currently playing back.
So for this element, it takes a minimum and a maximum.
Because we're working at percentages already,
we're just going to set min to 0 and max to 1
and use the value attribute of the input element
to extract what the current state of the input is
and, as the time changes, to shove a new value in
and have the input element reflect that change.
And as the user is dragging the input slider
around, it will generate these change events
and that's what we'll use to set the current time.
So this is what it would look like in JavaScript.
On the scrubber side, we'll listen for the change event.
Its value is something between 0 and 1, an easy percentage.
So we will just multiply it by the duration,
set with the current time, and Bob's your uncle,
you've got a working slider in your ... in your HTML video.
And on the-- from the other direction,
as the video's time changes,
we'll want to occasionally update the
value of the slider with its new position.
And again, since it's a percentage arranged from 0 to 1,
it's as easy as dividing the current time by the duration
and setting that to the value of the slider.
And in order to demonstrate these techniques,
all of the last 3 I'd like to call Eric
up, and he has got a few demos to show you.
[ Applause ]
>> So, we have got our adorable baby here with a controller.
We're going to be logging the events
over on the right-hand side.
We click Play, we click Pause,
we see the events coming through.
You can see the progress indicator here.
We have got a thumb, we drag it back and forth.
You can see that we're getting seek events
and timeupdate events just as you expect.
One more thing that we added just for a little
bit of extra bling, is as we move the mouse off
of the video, we show and hide the controller.
And that's as easy because we have CSS.
That's as easy as setting a couple of properties on the CSS.
So you can see in the video controls
attribute, we initially have the--
we've initially set the opacity of the controls to
0, and we've set the Y translate to 100 percent,
which pushes it off the-- pushes
it off the bottom of the screen.
And we've also set WebKit transitions
on both opacity and WebKit transform.
So that when the user hovers the mouse over the player,
when we change the opacity to 1 to make it visible,
and translate Y to 0 to bring it back up,
that transition will take a half a second.
So again, we move the mouse over it, it shows.
We move the mouse off and it hides.
And it's literally as simple as adding
just that, that little bit of CSS.
[ Applause ]
>> Alright.
Thanks, Eric.
OK, so we have most of the basic functionality
of the built-in controller now done
through JavaScript, accessible through CSS.
Style it however you want.
For the next step, we're going to push
past what the built-in controller can do
and add functionality that it never
could in the first place.
So subtitles, why should you care about subtitles?
For one reason, perhaps you have Americans with
Disabilities Act requirements that require your video
and your entire website to be accessible to
users for whom the audio is not very useful.
So you might need to present closed captions on top of
the video describing what is happening in the scene.
So that's something called descriptive audio.
You might also have video content, the audio which isn't
in the same language as your user's native language
and you want to have translations
appear on top of your video.
In this case, she doesn't speak any English
[background sound], we'll need to translate.
That's mama, and mama again.
OK, so how are we going to do this?
Well, before we start, I'd like to point out that this is
an active area of development in the "what" working group.
And they're currently writing the spec that
will define how, in the future, these subtitles
and its closed captions will be built in to future browsers.
Now, however, even though this is a future part of
the spec that's not even very well defined right now,
all of this is perfectly capable with the HTML and video
element as it exists today in Safari and other browsers.
So to start off, if you have a piece of video content
that already has closed caption tracks built in,
there are few methods you can use to detect that there
is closed caption tracks and enable them or disable them
in the video itself, and that's this webkitHasClosedCaptions
and webkitClosedCaptionsVisible
attributes of the video element.
Now this allows you to make visible a
single closed caption track in your video.
And that's great for people who have
existing content with closed captions.
However, it's not exactly dynamic.
It allows you to turn on or off
a single closed caption track.
And if you want to support multiple languages of
closed captions, you will need multiple videos.
So we're going to show you a technique that will work with
video that does not have anything built in and will work
for any language you care to insert through this method.
So meanwhile, we're going to embed
the subtitles directly in the HTML.
For the purposes of, you know, very
simple demo, this is the quickest way
to get your subtitles built in to your video content.
But there is no reason why you have
to embed the subtitles in HTML.
You could pull them down through JSON.
You could use XML HTTP requests.
And as we saw yesterday in the Safari state of the
union, someone from being implemented in pure HTML5,
a way of doing live translations of selected text,
there is no reason why you couldn't have translated
the subtitles available completely dynamically
through an API provided by a language translation service.
But for our case, we're just going to
embed it in HTML, and we're going to listen
for the same timeupdate event we
use to track the progress meter.
And as that event is fired, we'll check the current
time, figure out what subtitle is supposed
to be visible at that time and mark it as display.
For a long, long list of subtitles, that might
be something of an expensive proposition.
So if you have a set of subtitles, you can use the built-in
setTimeout and clearTimeout functions to schedule a subtitle
to be visible some time in the future and use the built-in
properties of the video element to figure out when
that subtitle will need to be displayed to the
screen, making things slightly more performing.
Again for a simple case, we only have a few subtitles,
so we're just going to calculate the correct
subtitle every time we catch the timeupdate event.
So here are some basic HTML code, you have a
element, and a few divs that have start
and end attributes defining when they should be
made visible through the playback of the video.
In this case, a 1-second subtitle, and then
another 1-second subtitle 1 second later.
For the CSS part of your HTML code, well by default,
make these subtitles display absolute
nothing, the bottom center of the screen.
But again, because this is completely dynamic done in HTML
and CSS, you are not limited to overlaying your video.
You can have the subtitles appear in
a different part of the same page,
or even in a pop-up page maybe movable around by the user.
For the JavaScript portion, we're going
to listen to the timeupdate event.
And just like in the progress meter example, plot
the current time, the time considered to be now.
And then iterate through all of
the elements marked as subtitles.
And for each one of them pull out their start
and their end attributes, compare them to now.
And when we find a subtitle whose start and end times
{current time}, set the display rule for its CSS style
to be inherent overwriting the
display none in our CSS style sheet.
Otherwise, we'll set it back to display
none and that subtitle will hide.
And to demonstrate this technique, I'd like
to call Eric back up to do another demo.
[ Applause ]
>> OK, this time there is not much to show.
You have seen the code, but let's see it in action.
We'll start our video playing.
There she says mama, and she coos at the end.
Because we're changing the display of these
through the events that are emitted by the element,
it works the same when we're playing
it as when we drag on the thumb.
Because the element as you can see from the
events on the side, emits the events so we know
when to show it and we know when to hide it.
It makes your coding really simple.
You just listen for the events to be emitted by
that element and you take action based on that.
Jer?
[ Applause ]
>> Thanks, Eric.
So, to reiterate, it's an entirely flexible technique
that allows you to do closed captions, subtitles,
apply styles as you saw on the example, to
make your subtitles look however you like.
And it works not only for subtitles, this could--
the same technique could work to cause events
to fire as you progress through the movie.
Put overlays on top of your video at specific times.
Cause different parts of your page
to change as the movie progresses.
We can use this technique for just about any
dynamic action you want to occur on your webpage.
So, let's recap.
We've got working JavaScript controller, we have subtitles.
Something else you might want to do with
your video playback is add a bumper clip.
And you might be asking yourself, what is he talking about?
Well, a bumper clip is a piece of video
industry jargon that refers to a secondary piece
of video inserted in between two other video clips.
Now, we might know this more colloquially as advertising.
So, a lot of video playback sites are doing this now.
YouTube, specifically I just saw the other day, inserting
advertisement at the beginning of video playback.
And you might want to know how to do this
if through HTML5 video on your own site.
So, there's a few different techniques to
getting your advertisements or your bumpers
or completely non-advertising these
things into your video playback.
The easiest approach is just to burn in the
advertising into your video content directly.
Do that QuickTime Player Pro or if
you have Final Cut Pro or even iMovie.
You can embed one video in another with
some nice transitions and do it that way.
That's great as far as it stands.
It's a little static, so we're going to show
you a couple of techniques to use if you want
to do a more dynamic way of inserting video.
So, if you're already doing HTTP Live Streaming, you can
insert some video content into your live stream directly.
And for this technique, I advise you to stick
around for the session following this one
that talks more about HTTP Live Streaming.
If you just want something to work completely client
side with no service added requirements whatsoever,
we're going to show you a technique that involves
switching sources of the video content during playback
to have the videos chained together one after the other.
It would require APIs like this one.
So first of all, there's an event inspired by the video
element as the playback reaches the end of your video,
and that's the ended event, named very well, I might say.
So, the source attribute is not only-- it's read/write,
and what we're going to do is shove a new source
into the video once the last video ended.
And then as the video sources change,
'cause the video act to play,
and if you're on a fast connection you'll see a very
seamless transition from video clip to video clip.
So, it looks something like this.
We have an array of sources, 4 or 3 different movies that we
want to play in a row, and an index into that array pointing
to the current playing movie, and the function which
advances those self sources from one to the other.
It simply increments the index, pulls the new source
out, sets it to the video and then cause play to occur.
And in this case, we're going listen to
the ended event and cause the advance
to occur only when the last movie ends playback.
There's no reason why you're limited to this.
You can have the next video source
come in the middle of your playback.
You could have something played in
the beginning and when the advertising
or whatever your bumper is finishes
playing, move on to the next video.
You can even have different links in the same
webpage cause different sources to be loaded
without taking the user out of the current page.
And to demonstrate this technique one more
time, we're going to have Eric come up.
[ Applause ]
>> In this case, we're just using the default
controls so that it's easy to see what's happening.
So, if you watch the current time thumb on the default
controls, watch what happens as we play through.
So we start our first movie.
It reaches the end.
If you watch fast, you can see
that the ended event was fired.
Our next movie loads.
It gets to the end and it goes back to another movie.
Now, you may want to customize the experience, make it
look to the user like they're just watching a single movie.
In that case, you would want to use a default controller,
but that would just use the same techniques using
for the custom controls up to this point.
Let's take a look here.
We'll see the same thing on the iPad.
Not too much of a surprise.
We start the movie, it reaches the
end, it goes to the next movie.
It plays through.
Isn't she cute?
And it goes to the last movie.
Now, if you're paying attention in the last session, you'll
recall that Vicki said that playback was strictly under--
playback is strictly under user control on iOS.
That is a script is not allowed to start or stop
a movie unless it's called from a user gesture.
And yet, what we have here is when the
ended event fires, our function is called.
We change the source to the next source.
We call load on the element and we call play.
The ended event isn't the user event and so you
would think that this wouldn't work but it does
because the user initiated playback on
this element when we first started to play.
So from that point on, the scripts are in fact allowed
to start and stop playback without user control
because the user initiated that initial playback.
So, that's one important point
that you want to keep in mind.
Jer?
[ Applause ]
>> Awesome.
Thanks, Eric.
OK, so we have a working JavaScript controller.
We have subtitles.
We have learned how to embed different clips and--
or not how to embed, we're getting to that,
how to combine different clips together.
The next step you might want to consider is if you have a
video page or if you have a webpage with some video elements
in it and your users come to that site and they see
something that it's really awesome and they want
to embed that same clip into their own site.
This is called embedding.
And originally, it referred to inserting a plug-in into
a webpage that was running native code but was hosted
in the interpretative code of the webpage.
However, it also allows you to insert one page into
another, and this is the technique we're going to use
to embed HTML5 video into a page that was
maybe not originally coded to HTML5 standards.
Because you embedded one webpage into another, it should
allow your users to copy and paste a very short line of code
to get that video into their page instead
of having to copy and paste like, you know,
the entirety of your page web and
your CSS and your JavaScript.
We'll get this with just one single small line.
So, the technique we're going to
use for this is the tag.
And the tag takes a source, and to make it
slightly more dynamic, we're going pass the argument
of the video we would like the iframe to
play back in the fragment portion of the URL.
That's the one after the octothorpe, the
hash mark, the sharp, the pound, that thing.
And on the client side, it might look something like this.
They have an existing HTML page.
They've got a body, they've got a bunch of content, and
they want to put your video right in the middle of that,
and they're going to do that with the tag.
As a source attribute, in this case, an
embed-- a link to an embedded specific website.
And in our case, we're going to pass the argument as
the entirety of the URL after the fragment portion.
If you're interested in doing server side scripting, you can
pass any number of normal query arguments in that same URL.
If you're still doing it, like, all
through JavaScript, you can still do arguments
after the fragment and parse them out yourself.
And this is the technique, actually this is
the most simple technique in the slides here
but Eric has a slightly more complicated technique that's
going to involve passing arguments in this fragment.
However, on the HTML code on your side, if you're doing
a very simple video tag embed, you'll have a video tag
with no source content and an onload handler for the body.
In this case, init, that will set everything up to
the way the user who embedded your URL requested.
So, the JavaScript to get this to work that's hosted on
your site, the init function might look something like this.
We're going to pull out the video element by its ID and, in
the simplest case, it will be as easy as setting the source
to the substring of the fragment,
everything after the hash mark.
And this alone will allow users to link to an
embedded HTML5 video page to get a video playback.
And to show you a slightly more
complicated piece of sample code,
Eric is going to-- has a great embedded video demo.
[ Applause ]
>> First, what we have is our--
the page that we've been using.
And what we've done is we've added a little bit
of JavaScript that queries the movie in the page,
pulls out the URL, pulls out the poster frame, if there
is one, gets the width and the height of the video
and puts it into this text area in a page.
And we've done that so that a user can come to the page.
For example, a user with the site
cutebabyfaces.com, which is nice,
but it's not very cute without the face of the baby.
Go to our blog, we'll open up our demo, our blog, and we'll
just paste in the iframe that we pulled out of the webpage.
We can reload it and there's our cute little baby.
Now, this is fine but this doesn't work
so well with the layout of the page.
The video is a bit wide for the layout that we have here.
But because the width and the height of the iframe are
parts of the iframe element, we can easily come in here
and change this, down here, scale it by 75 percent
in this case, reload our page and there she is.
It works like it always did.
In this case again, we're showing the
default controls to make it slightly simpler
but it would work just as well with custom controls.
Let's look at it on the iPhone and we'll reload the page.
And you can see, again, it works just
the same as it does on the desktop.
So, allowing somebody to come to your site and use
your content on their own site without actually having
to understand how all the technology works.
[ Applause ]
>> Awesome.
Thanks, Eric.
OK, while you're still up here, however, I'd like
you to give the example we came up with that took all
of these different techniques and pulled them on
together to something that's slightly more refined
than the examples that we've been using so far.
As much as I like that baby, there's a limited
audience, so something a little bit more well polished.
>> So, what we've done here is we have the same Dartmoor video that Vicki showed in the first example,
and we have put our own custom controller on it.
It's slightly different than the one that we saw last time.
We have a button.
We've added a button that allows us
to scale it up and to scale it back.
And as Vicki noted in the first
session, Safari 5 has full screen.
So if we wanted to, we could take this element.
We could ask WebKit, we could ask Safari
to fill the entire screen with this.
But when that happens, we take just the
video element full screen and, in this case,
we actually want to have-- we want
to be able to show titles on it.
And if we were to show that in full screen, because only
that video element shows that those subtitles wouldn't work.
We'll start it playing.
Subtitles are helpful.
Maybe we want to see them in a different language.
Notice that because this is HTML5 and the
video element is just another element,
we have flipped the element around using the CSS transform.
We have another element on the back with our subtitles
controls and it is not a hundred percent opaque
so the video is showing through and
you'll notice that it actually is--
the video is actually playing backwards
because we've flipped it around.
And that's not because QuickTime knows
how to play this thing backwards,
it's because it's a video, it's just another element.
And so of course when you flip it around,
it behaves the way that it should.
Back to English.
[ Laughter ]
[ Pause ]
[ Laughter ]
>> I don't know, maybe you guys
can tell me what that thing was.
It's not a fossilized pig face.
So this happens to be a fairly high data rate movie.
It's 1280 by 544 pixels at 1875
kilobit, and look at the performance.
It's really stunning.
I take it big and small.
You see playback is smooth.
It's really spectacular.
[ Applause ]
>> Please contact Vicki Murley.
She's our Safari Technologies Evangelist,
as well as check the documentation
on the apple.com developer page for Safari.
Specifically, there's a whole section on HTML5 audio
and video in the Safari HTML5 audio and video guide.
As well as the, what working webpage-- the W3C also has a
working draft of the HTML5 spec which you can check out.
And the brand new developer forums for
Safari are available at devforums.apple.com.
So, in the session directly following this one, it
will be a whole talk on advances in HTTP streaming.
And I highly encourage you to stick around for that one.
Later today, there are-- as a two part session on CSS
effects, we'll talk about all the special CSS3 stylings
that we put into these webpages to make them super awesome.
[ Applause ]