WHATWG Wiki - User contributions [en]

Timed tracks

2010-04-26T13:31:27Z

Rescator: /* Streaming */ Web radio possibility.

This page contains notes for the development of the first version of timed track features in HTML.

See also [[use cases for timed tracks rendered over video by the UA]], [[use cases for API-level access to timed tracks]].

== Requirements ==

=== Subtitle/Caption/Karaoke File Format ===

* per-cue in/out times
** relative timings would be useful while editing, but may not be necessary in the published format
* inline time cues for karaoke
* bidi, newlines, ruby, italics, bold [there's been no evidence provided that there's any need for more fine-grained control at a per-cue level]
* voice selection (so that e.g. sfx descriptions and each character can be a different colour)
* per cue vertical position: % of vertical video height (default 100%)
* per cue horizontal position: % of horizontal video width (default 50%)
* per cue direction: horizontal/vertical (default horizontal)
* per cue width/height: % (default is remaining space on line given alignment)
* per cue text alignment: start/middle/end (default middle for subtitles, start for captions)
* multiple cues placed in adjacent places (e.g. from different voices or with slightly different times) would need to automatically stack so they don't overlap
** but should support multiple cues from multiple voices on the same "line", e.g. when two people both utter something at the same time (need an example of this).

(Percentage positions would work like background-position in CSS.)

==== Track-wide formatting ====
* color of background/text/outline is needed for readability on different types of video, unless UA default has clear contrasting outlines or an opaque background.
* webfonts is needed to provide high quality subtitles in some non-Latin languages (e.g. Chinese where a suitable font is unlikely to be available even on Chinese computer systems).
* providing a pseudo-element to style each voice would likely be sufficient for authors who want overall formatting control (this would also allow user overrides conveniently)

=== HTML ===

* an API and UI for exposing what timed tracks exist and selectively enabling/disabling them

* format for external subtitles/captions
* format for external audio descriptions
* some mechanism for text in the page to be used instead of external files, for subtitles/captions or audio description
* an API to allow a segment to be dynamically inserted into the rendering on the fly

* an API for exposing what the currently relevant segments of each timed track are
* a way to hook into this mechanism to advance slides

* native rendering of subtitles
* native rendering of audio descriptions
* native rendering of multiple audio or video tracks, to allow pre-recorded audio descriptions to be mixed in and sign language video to be overlaid
* a way to hook into this to manually render timed tracks

== Architecture ==

<img src="http://docs.google.com/drawings/pub?id=1GR6Pzq0GY2n1sx_ZjDfuICM2LnXxLVxzvyl4kuQy-48&w=640&h=480">

The caption format should be reasonable for either a web engine or a media engine to render, since implementation strategies may differ.

=== Declaring timed tracks ===

Each timed track is either:

*enabled, in which case it is downloaded, triggers events, and if appropriate is rendered by the user agent; or
*disabled, in which case it does nothing

The enabled/disabled state is by default based on user preferences and the kind of timed track as described below, but can be overridden on a per-track basis.

Each timed track has a kind which is one of:
* for visual display (subtitles, captions, translations), enabled based on user preferences, shows in video playback area
* for audio playback (text audio descriptions), enabled based on user preferences, renders as audio
* for navigation (chapter titles), enabled by default, shows in UA UI
* for off-video display (lyrics), disabled by default in this version, not shown by UA
* for metadata (slide timings, annotation data for app-rendered annotations), enabled by default, not shown by UA

Tracks that are for visual display or audio playback have additionally a user-facing label and a language.

Tracks that are for visual display have an additional boolean indicating if they include sound effects and speaker identification (intended for the deaf, hard of hearing, or people with sound muted) or not (i.e. translations intended for people with audio enabled but who cannot understand the language, or karaoke lyrics).

Each timed track associated with a media resource, like the media resource itself, can have multiple sources.

Each source for a timed track has:
* URL
* type (if there are multiple sources)
* media

The media resource can also imply certain timed tracks based on data in the media resource.

The script can also add "virtual" timed tracks dynamically.

==== Markup ====

<track src="" enabled="true" kind="" label="" lang=""></track>

<track enabled="true" kind="" label="" lang="">
<source src="" type="" media="">
...
</track>

enabled="" is true or false.

Values for kind="":

* subtitles (includes karaoke) - default
* captions
* description (text audio descriptions)
* chapters
* lyrics
* metadata

Questions:

* Should we use lang="", hreflang="", srclang=""?

* Is there a better solution to enabled=false for disabling tracks by default? Do we ever need to disable a track that might be enabled by default?

=== Visual titles ===

==== File format ====

Should be backwards-compatible with an existing format, ideally SRT given the huge volume of subtitles available in SRT format on the Web today.

...

==== Processing model ====

...

==== CSS extensions ====

Cues are rendered as block boxes with inline boxes. Cues have a voice (identified by a keyword or a number). Cues can have a part that is before the current time and a part after the current time.

The block box is matched by the pseudo-element ::cue on the media element (<video>).
Only visible cues are matched (those on tracks enabled and shown by the UA whose start/end time range contains the current time).
The ::cue pseudo takes an optional argument that is the voice of cues that it is to match.

video::cue { color: white; background: rgba(0,0,0,0.5); font: 900 sans-serif; text-transform: uppercase; }
video::cue(narrator) { color: white; font-style: italics; }
video::cue(1) { color: yellow; }
video::cue(2) { color: lime; }

The ::cue-fragment pseudo matches all inner-most inline boxes in the cue of the element. Its last argument is one of "i", "b", "ruby", "rt", "before", "after". Its first argument, if it has two, is a voice. If it has arguments, the pseudo-element only matches inline boxes that correspond to "i", "b", "ruby", or "rt" fragments (for those keywords) or that correspond to fragments before/after the current time (for those keywords).

video::cue-fragment(i) { font-style: italic; }
video::cue-fragment(narrator, i) { font-style: bold; }
video::cue-fragment(b) { font-size: larger; }

[This isn't great. Any better ideas?]

=== DOM API ===

HTMLMediaElement
attribute MediaTrack[] tracks;
MutableMediaTrack addTrack(label, kind, language);

MediaTrack
readonly attribute DOMString label;
readonly attribute DOMString kind; // subtitles, captions, descriptions, chapters, lyrics, metadata
readonly attribute DOMString language;
readonly attribute unsigned short mode;
const unsigned short TRACK_OFF = 0; // not firing events, may not even be downloaded yet
const unsigned short TRACK_HIDDEN = 1; // firing events but otherwise ignored by UA - intended for scripts
const unsigned short TRACK_SHOWING = 2; // browser is handling it
readonly attribute MediaCue[] cues; // sorted in startTime order
readonly attribute MediaCue[] activeCues; // sorted in endTime order?
readonly attribute Function onentercue; // fires CueEvent
readonly attribute Function onexitcue; // fires CueEvent

MutableMediaTrack: MediaTrack
void addCue(cue); // throws if cue.track != null
void removeCue(cue); // throws if cue isn't in this track

MediaCue
readonly attribute MediaTrack track; // null if newly created and not yet added to a track
readonly attribute DOMString id; // empty string if not applicable
readonly attribute float startTime;
readonly attribute float endTime;
readonly attribute unsigned short horizontalPosition;
readonly attribute unsigned short verticalPosition;
readonly attribute unsigned short size;
readonly attribute DOMString direction; // horizontal, vertical
readonly attribute DOMString alignment; // start, middle, end
readonly attribute DOMString voice; // for styling purposes
DocumentFragment getCueAsHTML(); // returns a copy of the cue as HTML, with the current position in the case of karaoke lyrics annotated using a ProcessingInstruction or some such

Constructor for MediaCue: new MediaCue(id, startTime, endTime, hPos, vPos, size, dir, align, voice, text); // text gets parsed like the cues in the main format, whatever that ends up being

CueEvent
readonly attribute MediaCue cue;

HTMLTrackElement
readonly attribute MediaTrack track;

=== Other minor things ===

We need to make sure that media playback is paused until all enabled timed tracks are locally available.

We need to block cross-origin tracks (eventually blocking only those that aren't CORS-enabled).

== Open issues ==

=== Synchronised media ===

For now, sign-language and alternate or additive audio tracks (e.g. audio description tracks) have to be in-band, because UA vendors are refusing to implement synchronisation of external media tracks for now.

However, we should bear it in mind. Adding that kind of thing to the API is going to be non-trivial. The simplest way is probably to just to require that the authors use multiple <video>/<audio> elements and we link them somehow; with one designated as the "sync clock" with them all syncing to it, rather than having each <video> element expose multiple "buffered" "seekable" etc.

=== Streaming ===

Do we need to handle live transcription and streaming titles in external files? If so, how?

For now, it's not clear if there are any use case for streaming external timed track resources.

Although web based radio might benefit from serving a live audio stream with song title and other details like a artist url maybe.
This would broaden the listener base to those that do not have a shoutcast able player.

Feature Proposals

2010-03-24T19:57:45Z

Rescator: /* Document Markup */

This document contains a list of the problems for which feature requests have been made. Linked problem pages contain the document of the problem and their relevant solutions. Obviously, we want to keep HTML as simple as possible. That means not everyone will get what they want. Having good documentation for the problems at hand will help all of us work out what is most important.

== Guidelines ==

Before proposing a feature, please read [http://blog.whatwg.org/proposing-features Proposing features]. If you want to add a feature request, start by copying the [[Problem Solving]] template page onto a new page and fill out as much information as you can.

You don't have to provide detailed answers for everything straight away. The most important information to provide at first is the problem description. Once we have detailed descriptions, use cases and an understanding of the limitations with existing markup, we can then begin to discuss the best way in which to address the problems and work out more of the more technical details.

== Document Markup ==
* [[Image Caption]]
* [[Layout tables]]
* [[Drag'n'Drop Uploads]]
* [[New Vocabularies]], [[Constraints for New Vocabularies]]
* [[Generic Metadata Mechanisms]]
* [[Link Icons]]
* [[Page thumbnail]]
* [[Video Overlay]]
* [[Meta element href]]

== DOM Scripting ==
* [[Text in Canvas]]

== Security ==
* [[Link Hashes]]
* [[Digital Signatures]]

== Other ==

This list of features needs to be sorted out. They've come from all the [http://del.icio.us/lachlan.hunt/WHATWG feedback provided on blogs] over the past few weeks.

* Don't render quotation marks around <code>q</code> elements.
* Make form validation easier
** <code>required</code> attribute
** <code>maxlength</code> attribute for textarea
** <code>pattern</code> attribute
** <code>min</code>
** <code>max</code>
* New Form Controls
** Search fields
** Combo boxes
** Date/Time
** E-mail
** Int
** Long
** Unsigned
** Float
** Number
** Currency
** URL
* WYSIWIG Editor (<code>contentEditable</code>)
* Placeholder attribute
* Captions for images
* Bring back the <code>start</code> and <code>value</code> attributes for ordered lists.
* Bring back the <code>menu</code> element
* Require XHTML-link syntax for HTML
* Caption/label/list header for lists
* Include the <code>role</code> attribute.
* Allow <code>href</code> on all elements
* Make it easier to mark up blocks of code
* Allow block level elements inside paragraphs
* “a tag that could hold "bad grammar" and not have any effect on the validation (sort of like a document.write from JavaScript) and would terminate all unclosed items at the end of the element (like TDs tend to do).”
* <code>blink</code>!
* Fix the <code>object</code> element.
* Unify <code>img</code>, <code>object</code>, </code>embed</code> and <code>iframe</code> into a single element
* Headers and footers
* A mechanism to include content from an external source (e.g. <code>include</code>, perhaps like XInclude)
* A <code>corner</code> element (presumably for making rounded corners)
* Markup for advertisements
* Easier column layouts
* A <code>foot</code> element for containing scripts at the bottom of the page, or something to help deal with cross-browser load events.
* Key Generation/Certificate management (The <code>keygen</code> element)

[[Category:Feature Request]]

Meta element href

2010-03-20T12:54:22Z

Rescator: /* My Solution */

:Extending the Meta element to support a href attribute that references a current document element id attribute.

== Use Case Description ==
A new href attribute would allow referencing element id's within the document, who's content would replace the meta content.

=== Current Limitations ===
The current meta element causes redundant content in many cases.
And it is hidden from view usually.''

=== Current Usage and Workarounds ===
No known current usage of meta href, no workarounds currently known/possible.

=== Benefits ===

Avoiding redundancy, meta content is no longer hidden from view, consistent and more varied meta content, humanly readable.

=== Requests for this Feature ===

* <cite>[http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-March/025575.html] WhatWG Mailing list</cite>

== Proposed Solutions ==

=== My Solution (Roger Hågensen) ===
:Allow use of href attribute with meta element, enabling id referencing to replace content attribute with other document content.
<pre>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="description" href="#description" />
<meta name="author" href="#author" />
<meta name="date" href="#date" />
<meta name="copyright" href="#copyright" />
<meta name="keywords" href="#keywords" />
<title>href extending the meta tags.</title>
</head>
<body>
<article>
<header>Section header.</header>
<p id="#description">This is the first paragraph in the document or an aside or some other content perhaps.</p>
<p>More content here.</p>
<footer>Author: <a href="example.com/author/url/" id="#author">Roger Hågensen</a> on <time datetime="2010-03-18T08:00:00" id="#date">18th March 2010 at 8 o'clock.</time><br />
<span id="#copyright">© Roger Hågensen 2010</span><br />
Keywords: <span id="#keywords"><a href="http://example.com/tag/Example/">Example</a>, <a href="http://example.com/tag/Meta/">Meta</a>, <a href="http://example.com/tag/HTML5/">HTML5</a></span></footer>
</article>
</body>
</html>
</pre>

==== Processing Model ====
:Validators would check if there is a content or href attribute define for the meta element, content and href are mutually exclusive and validators should show a warning. Browsers and parsers should if both are present, ignore content attribute and use href instead. Parsers/browsers need to look through the all elements of the current document for an id attribute that matches the meta href, if found the content of the element need to have it's tags and any presentational markup stripped so the pure content is available. This stripped content will now be used as if it was the meta content attribute. If the id is not found, or if after stripping the tags of the referenced element the content is empty, then it should be treated the same as if the meta description was either missing or empty.

==== Limitations ====
:Only local to the document, url's is beyond this spec's current scope and should no be allowed.

==== Implementation ====
:Browsers, screenreaders and parsers will benefit from this, but more importantly indexers and search engines will benefit even more. As the meta href references an element in the document, which is humanly readable like say the first paragraph, it ensures that the meta description is actually proper page content. Unlike presently where the meta description and keyword and author and copyright and so on are far from usefull or relevant. Many meta keywords or meta description attributes contain sitewide content rather than page relevant content.

==== Adoption ====
:Improved content displayed in search engine results. Screenreaders will have humanly readable content available. Less bandwidth use due to reduced redundancy. Does not clash with any existing standards.

[[Category:Feature Request|empty Template]]

Meta element href

2010-03-20T12:53:43Z

Rescator: /* My Solution */

:Extending the Meta element to support a href attribute that references a current document element id attribute.

== Use Case Description ==
A new href attribute would allow referencing element id's within the document, who's content would replace the meta content.

=== Current Limitations ===
The current meta element causes redundant content in many cases.
And it is hidden from view usually.''

=== Current Usage and Workarounds ===
No known current usage of meta href, no workarounds currently known/possible.

=== Benefits ===

Avoiding redundancy, meta content is no longer hidden from view, consistent and more varied meta content, humanly readable.

=== Requests for this Feature ===

* <cite>[http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-March/025575.html] WhatWG Mailing list</cite>

== Proposed Solutions ==

=== My Solution === (Roger Hågensen)
:Allow use of href attribute with meta element, enabling id referencing to replace content attribute with other document content.
<pre>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="description" href="#description" />
<meta name="author" href="#author" />
<meta name="date" href="#date" />
<meta name="copyright" href="#copyright" />
<meta name="keywords" href="#keywords" />
<title>href extending the meta tags.</title>
</head>
<body>
<article>
<header>Section header.</header>
<p id="#description">This is the first paragraph in the document or an aside or some other content perhaps.</p>
<p>More content here.</p>
<footer>Author: <a href="example.com/author/url/" id="#author">Roger Hågensen</a> on <time datetime="2010-03-18T08:00:00" id="#date">18th March 2010 at 8 o'clock.</time><br />
<span id="#copyright">© Roger Hågensen 2010</span><br />
Keywords: <span id="#keywords"><a href="http://example.com/tag/Example/">Example</a>, <a href="http://example.com/tag/Meta/">Meta</a>, <a href="http://example.com/tag/HTML5/">HTML5</a></span></footer>
</article>
</body>
</html>
</pre>

==== Processing Model ====
:Validators would check if there is a content or href attribute define for the meta element, content and href are mutually exclusive and validators should show a warning. Browsers and parsers should if both are present, ignore content attribute and use href instead. Parsers/browsers need to look through the all elements of the current document for an id attribute that matches the meta href, if found the content of the element need to have it's tags and any presentational markup stripped so the pure content is available. This stripped content will now be used as if it was the meta content attribute. If the id is not found, or if after stripping the tags of the referenced element the content is empty, then it should be treated the same as if the meta description was either missing or empty.

==== Limitations ====
:Only local to the document, url's is beyond this spec's current scope and should no be allowed.

==== Implementation ====
:Browsers, screenreaders and parsers will benefit from this, but more importantly indexers and search engines will benefit even more. As the meta href references an element in the document, which is humanly readable like say the first paragraph, it ensures that the meta description is actually proper page content. Unlike presently where the meta description and keyword and author and copyright and so on are far from usefull or relevant. Many meta keywords or meta description attributes contain sitewide content rather than page relevant content.

==== Adoption ====
:Improved content displayed in search engine results. Screenreaders will have humanly readable content available. Less bandwidth use due to reduced redundancy. Does not clash with any existing standards.

[[Category:Feature Request|empty Template]]

Meta element href

2010-03-20T12:52:37Z

Rescator: /* Current Usage and Workarounds */ forgot to write this up

:Extending the Meta element to support a href attribute that references a current document element id attribute.

== Use Case Description ==
A new href attribute would allow referencing element id's within the document, who's content would replace the meta content.

=== Current Limitations ===
The current meta element causes redundant content in many cases.
And it is hidden from view usually.''

=== Current Usage and Workarounds ===
No known current usage of meta href, no workarounds currently known/possible.

=== Benefits ===

Avoiding redundancy, meta content is no longer hidden from view, consistent and more varied meta content, humanly readable.

=== Requests for this Feature ===

* <cite>[http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-March/025575.html] WhatWG Mailing list</cite>

== Proposed Solutions ==

=== My Solution ===
:Allow use of href attribute with meta element, enabling id referencing to replace content attribute with other document content.
<pre>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="description" href="#description" />
<meta name="author" href="#author" />
<meta name="date" href="#date" />
<meta name="copyright" href="#copyright" />
<meta name="keywords" href="#keywords" />
<title>href extending the meta tags.</title>
</head>
<body>
<article>
<header>Section header.</header>
<p id="#description">This is the first paragraph in the document or an aside or some other content perhaps.</p>
<p>More content here.</p>
<footer>Author: <a href="example.com/author/url/" id="#author">Roger Hågensen</a> on <time datetime="2010-03-18T08:00:00" id="#date">18th March 2010 at 8 o'clock.</time><br />
<span id="#copyright">© Roger Hågensen 2010</span><br />
Keywords: <span id="#keywords"><a href="http://example.com/tag/Example/">Example</a>, <a href="http://example.com/tag/Meta/">Meta</a>, <a href="http://example.com/tag/HTML5/">HTML5</a></span></footer>
</article>
</body>
</html>
</pre>

==== Processing Model ====
:Validators would check if there is a content or href attribute define for the meta element, content and href are mutually exclusive and validators should show a warning. Browsers and parsers should if both are present, ignore content attribute and use href instead. Parsers/browsers need to look through the all elements of the current document for an id attribute that matches the meta href, if found the content of the element need to have it's tags and any presentational markup stripped so the pure content is available. This stripped content will now be used as if it was the meta content attribute. If the id is not found, or if after stripping the tags of the referenced element the content is empty, then it should be treated the same as if the meta description was either missing or empty.

==== Limitations ====
:Only local to the document, url's is beyond this spec's current scope and should no be allowed.

==== Implementation ====
:Browsers, screenreaders and parsers will benefit from this, but more importantly indexers and search engines will benefit even more. As the meta href references an element in the document, which is humanly readable like say the first paragraph, it ensures that the meta description is actually proper page content. Unlike presently where the meta description and keyword and author and copyright and so on are far from usefull or relevant. Many meta keywords or meta description attributes contain sitewide content rather than page relevant content.

==== Adoption ====
:Improved content displayed in search engine results. Screenreaders will have humanly readable content available. Less bandwidth use due to reduced redundancy. Does not clash with any existing standards.

[[Category:Feature Request|empty Template]]

Meta element href

2010-03-20T12:41:41Z

Rescator: Created page with ':Extending the Meta element to support a href attribute that references a current document element id attribute. == Use Case Description == A new href attribute would allow refe...'

:Extending the Meta element to support a href attribute that references a current document element id attribute.

== Use Case Description ==
A new href attribute would allow referencing element id's within the document, who's content would replace the meta content.

=== Current Limitations ===
The current meta element causes redundant content in many cases.
And it is hidden from view usually.''

=== Current Usage and Workarounds ===
Some evidence that this feature is desperately needed on the web. You may provide a separate examples page for listing these.

=== Benefits ===

Avoiding redundancy, meta content is no longer hidden from view, consistent and more varied meta content, humanly readable.

=== Requests for this Feature ===

* <cite>[http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-March/025575.html] WhatWG Mailing list</cite>

== Proposed Solutions ==

=== My Solution ===
:Allow use of href attribute with meta element, enabling id referencing to replace content attribute with other document content.
<pre>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="description" href="#description" />
<meta name="author" href="#author" />
<meta name="date" href="#date" />
<meta name="copyright" href="#copyright" />
<meta name="keywords" href="#keywords" />
<title>href extending the meta tags.</title>
</head>
<body>
<article>
<header>Section header.</header>
<p id="#description">This is the first paragraph in the document or an aside or some other content perhaps.</p>
<p>More content here.</p>
<footer>Author: <a href="example.com/author/url/" id="#author">Roger Hågensen</a> on <time datetime="2010-03-18T08:00:00" id="#date">18th March 2010 at 8 o'clock.</time><br />
<span id="#copyright">© Roger Hågensen 2010</span><br />
Keywords: <span id="#keywords"><a href="http://example.com/tag/Example/">Example</a>, <a href="http://example.com/tag/Meta/">Meta</a>, <a href="http://example.com/tag/HTML5/">HTML5</a></span></footer>
</article>
</body>
</html>
</pre>

==== Processing Model ====
:Validators would check if there is a content or href attribute define for the meta element, content and href are mutually exclusive and validators should show a warning. Browsers and parsers should if both are present, ignore content attribute and use href instead. Parsers/browsers need to look through the all elements of the current document for an id attribute that matches the meta href, if found the content of the element need to have it's tags and any presentational markup stripped so the pure content is available. This stripped content will now be used as if it was the meta content attribute. If the id is not found, or if after stripping the tags of the referenced element the content is empty, then it should be treated the same as if the meta description was either missing or empty.

==== Limitations ====
:Only local to the document, url's is beyond this spec's current scope and should no be allowed.

==== Implementation ====
:Browsers, screenreaders and parsers will benefit from this, but more importantly indexers and search engines will benefit even more. As the meta href references an element in the document, which is humanly readable like say the first paragraph, it ensures that the meta description is actually proper page content. Unlike presently where the meta description and keyword and author and copyright and so on are far from usefull or relevant. Many meta keywords or meta description attributes contain sitewide content rather than page relevant content.

==== Adoption ====
:Improved content displayed in search engine results. Screenreaders will have humanly readable content available. Less bandwidth use due to reduced redundancy. Does not clash with any existing standards.

[[Category:Feature Request|empty Template]]