A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.

To request an account, ask an autoconfirmed user on Chat (such as one of these permanent autoconfirmed members).

Video Overlay

From WHATWG Wiki
Revision as of 13:52, 28 November 2009 by Philipj (talk | contribs) (cleanup)
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

There are several types of content which should be displayed overlayed on <video> elements, including subtitles/captions from various sources, scripted controls and more advanced scripted content such as karaoke or timed annotations. <overlay> suggested below tries to provide a simple markup to enable all such uses.

Use Case Description

There are several distinct use cases addressed by this proposal:

  • Linking <video> with external captions/subtitles for native fetching/decoding/syncing by the UA.
  • Styling captions/subtitles with CSS, regardless of their source.
  • Allowing scripts to operate on captions/subtitles in a uniform manner, regardless of their source.

Possible sources of captions/subtitles include in-band (e.g. embedded in an MPEG-4 or Ogg stream), external (e.g. SRT or DXFP) or scripted (e.g. extracted from an on-page transcript) captions/subtitles.

Current Limitations

HTML5 currently lacks convenient markup and/or interfaces to handle at least these things:

  • Syncing and styling external subtitles/captions with <video>
  • Styling in-band subtitles/captions from media resources
  • Rendering scripted controls on top of <video> and positioning them to bottom.
  • Callbacks at specific times for scripted subtitles/captions (previously possible with "cue ranges")
  • Allowing any overlay (controls/captions/etc) to be retained in fullscreen mode.

Current Usage and Workarounds

Currently no browser supports rendering in-band subtitles, so there are no workarounds for styling them. Fullscreen support is still immature, but there is no possible workaround for having scripted captions or controls to appear in fullscreen display.

Scripted Captions

In Silvia's <itext> demo external SRT subtitles are fetched with XHR, parsed with JavaScript and finally synced in the timeupdate event. Using the timeupdate event is sub-optimal because it isn't guaranteed to fire any more often than every 250 ms, which isn't enough for fast-paced dialog.

Scripted Controls

In order to overlay scripted controls on top of <video>, a wrapping <div> and some CSS is needed:

<div style="position:relative;width:400px;height:300px">
  <video src="video.ogv" style="width:100%;height:100%"></video>
  <div class="controls" style="position:absolute;bottom:0;left:0;right:0">
    <!-- actual controls here -->
  </div>
</div>

This isn't terrible, but requires the size of the video to be known or be fixed to a certain size as above.

Benefits

<overlay> provides a single container for styling for all kinds of overlay content. The alternative would be to have one markup for in-band and external captions/subtitles (e.g. <itext>) and another solution for scripted captions/subtitles/controls/annotations, even though the problem solved is mostly exactly the same.

Requests for this Feature

  • <overlay> suggested by Philip Jägenstedt (Opera)
TODO: Find the many mails related to some of the features addressed by <overlay>

Proposed Solutions

<overlay>

The <overlay> element is used as a child of <video>. It can optionally refer to an external source, which should be in a format supported by the UA. Example:

<video src="video.ogv">
  <overlay src="captions.srt">
</video>

Possibly, one could allow <overlay> to have <source> element children, similar to <video>. The purpose would be group resources which are mutually exclusive, e.g. subtitles in different languages:

<video src="video.ogv">
  <overlay>
    <source src="captions-english.srt" lang="en"></source>
    <source src="captions-simplified-chinese.srt" lang="zh-Hans"></source>
  </overlay>
</video>

If necessary, one could also <source> to provide the same resource in multiple formats for fallback purposes:

<video src="video.ogv">
  <overlay>
    <source src="captions.srt" type="text/x-srt"></source>
    <source src="captions.xml" type="application/ttaf+xml"></source>
  </overlay>
</video>

When <overlay> does not point to an external resource, its content should instead be displayed. By updating the content with scripts, the possibilities are many:

<video src="video.ogv">
  <overlay><!-- content goes here --></overlay>
</video>
<script>
  var v = document.querySelector("video");
  var ol = v.querySelector("overlay");
  v.ontimeupdate = function() {
    ol.textContent = someInterestingText();
  }
</script>

Processing Model

Resource Selection

If the <overlay> element is allowed to reference external resource using <source>, a resource selection algorithm must be defined. Other proposals have included variations on the theme of letting the UA to automatically select the language and type of timed text most suitable for the user (e.g. French subtitles for French-reading users and captions for hard-of-hearing users). Unlike resource selection for media elements, this would require the <source> candidates to be evaluated in another order than strict document order. There are certain complications to this, which may or may not be justified:

  • Requiring UAs to keep (or act as if they keep) a priority-sorted list of the candidates and keep that in sync with DOM modifications, so that they can fall back to the next best if a resource is unavailable or undecodable.
  • Relying on the Accept-Language setting which is often wrong or adding new language preferences to the UA which are likely to fail in the same ways Accept-Language has.

This proposal does not include a solution, implementor experience on this is probably the best way of finding out what makes sense and not.

Styling

It needs to be defined exactly what <overlay> is to <video>. The simplest may be acting as if <video> were the containing block for <overlay> elements, so that e.g. applying position:absolute;bottom:0;left:0;right:0 on <overlay> causes it to be position at the bottom of the parent <video>.

Should <overlay> could be a block or an inline element?

DOM API

TODO: cue ranges

Limitations

Cases not covered by this solution in relation to the problem description; other problems with this solution, if any.

Implementation

Description of how and why browser vendors would take advantage of this feature.

Adoption

Reasons why page authors would use this solution.

References

Silvia Pfeiffer's blog posts:

Silvia Pfeiffer's <itext> proposals:

Mailing lists: