This specification defines an API extending the
HTMLMediaElement that enables controlling remote
playback of media from a web page.
This document builds on the group's experience on presenting web content on external presentation-type displays, and re-uses patterns and design considerations from the Presentation API specification whenever appropriate [[PRESENTATION-API]].
Although this document is still a work in progress and is subject to change, the Working Group believes that the API surface is stable. Most of the remaining issues listed on the issue tracker are considered minor at this stage except for Issue #41.
Issue #41 discusses the set of media playback features that remote playback devices are expected to support. The group will seek further developer feedback and implementation experience to identify any interoperability issues around these features when used during remote playback, and will further clarify the specification based on feedback received.
For other issues or concerns, it is possible to file a bug or send an email to the mailing list. For small editorial changes like typos, sending a pull request is appreciated.
The Working Group invites everyone to review this document, and will work with relevant groups at W3C to conduct horizontal reviews on accessibility, internationalization, privacy, security and technical architecture principles.
No feature has been identified as being at risk.
The Second Screen Working Group will develop a test suite for the Remote Playback API during the Candidate Recommendation period and prepare an implementation report. For this specification to advance to Proposed Recommendation, two independent, interoperable implementations of each feature must be demonstrated, as detailed in the Candidate Recommendation exit criteria section.
This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.
Implementations that use ECMAScript to expose the APIs defined in this specification MUST implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [[!WEBIDL]].
This specification aims to make remote playback devices such as connected TVs, projectors or audio-only speakers, available to the Web and takes into account playback devices that are attached using wired (HDMI, DVI, or similar) and wireless technologies (Miracast, Chromecast, DLNA, AirPlay, or similar).
Devices with limited screen size or quiet speakers lack the ability to playback media content to a larger audience, for example, a group of colleagues in a conference room, or friends and family at home. Playing media content on an external larger and/or louder remote playback device helps to improve the perceived quality and impact of the played media.
At its core, this specification enables a page that acts as the browsing context to initiate and control remote playback of a particular media element on a selected remote playback device. How the remoting is initiated and controlled is left to the UA in order to allow the use of remote playback devices that can be attached in a wide variety of ways. For example, when a remote playback device is attached using HDMI or Miracast, the same UA that acts as the browsing context renders the remote media. Instead of playing the media on the same device, however, it can use whatever means the operating system provides for using the external remote playback device. In such a case, both the browsing context and the media player run on the same UA and the operating system is used to route the player output to the remote playback device. This is commonly referred to as the media mirroring case. This specification imposes no requirements on the remote playback devices connected in such a manner.
If the remote playback device is able to play the media and communicate with the browsing context but is unable to fetch the media, the browsing context needs to fetch the media data and pass it on to the remote playback device for rendering. This is commonly referred to as media remoting case.
If the remote playback device is able to fetch and play the media and communicate with the browsing context, the browsing context does not need to fetch or render the remoted media. In this case, the UA acts as a proxy that requests the remote playback device to play the media itself by passing the necessary data like the media source. This is commonly referred to as the media flinging case. This way of attaching to displays could be enhanced in the future by defining a standard protocol for delivering these types of messages that remote playback devices could choose to implement.
The API defined here is intended to be used with UAs that attach to remote playback device devices through any of the above means.
The use cases and requirements of this specification are captured in a separate document available here.
The following concepts and interfaces are defined in [[!HTML]]:
track, and
srclang.
[CEReactions]
The term URL is defined in the WHATWG URL standard [[!URL]].
The term throw in this specification is used as defined in [[!WEBIDL]].
The interface PresentationRequest is defined in
[[PRESENTATION-API]].
The term potentially secure is defined in [[!MIXED-CONTENT]].
The following exception names are defined by [[!WEBIDL]] and used by this specification:
InvalidAccessError
InvalidStateError
NotAllowedError
NotFoundError
NotSupportedError
OperationError
This section shows code examples that highlight the usage of the main
features of the Remote Playback API. In these examples,
player.html implements the player page controlling the
remote playback and media.ext is the media file to be
played remotely. Both the page and the media are served from the domain
https://example.org. Please refer to the comments in the
code examples for further details.
<!-- player.html -->
<!-- The video element with custom controls that supports remote playback. -->
<video id="videoElement" src="https://example.org/media.ext" />
<button id="deviceBtn" style="display: none;">Pick device</button>
<script>
// The "Pick device" button is visible if at least one remote playback device is available.
var deviceBtn = document.getElementById("deviceBtn");
var videoElem = document.getElementById("videoElement");
function availabilityCallback(available) {
// Show or hide the device picker button depending on device availability.
deviceBtn.style.display = available ? "inline" : "none";
}
videoElem.remote.watchAvailability(availabilityCallback).catch(function() {
// Availability monitoring is not supported by the platform, so discovery of
// remote playback devices will happen only after remote.prompt() is called.
// Pretend the devices are available for simplicity; or, one could implement
// a third state for the button.
deviceBtn.style.display = "inline";
});
</script>
<!-- player.html -->
<script>
devicesBtn.onclick = function() {
// Request the user to select a remote playback device.
videoElem.remote.prompt()
// Update the UI and monitor the connected state.
.then(updateRemotePlaybackState);
// Otherwise, the user cancelled the selection UI or no screens were found.
};
<script>
<!-- player.html -->
<script>
// The remote playback may be initiated by the user agent,
// so check the initial state to sync the UI with it.
if (videoElem.remote.state == "disconnected")
switchToLocalUI();
else
switchToRemoteUI();
videoElem.remote.onconnecting = switchToRemoteUI;
videoElem.remote.onconnect = swithToRemoteUI;
videoElem.remote.ondisconnect = switchToLocalUI;
// Handles both 'connecting' and 'connected' state. Calling more than once
// is a no-op.
function switchToRemoteUI() {
// Indicate that the state is 'connecting' or 'connected' to the user.
// For example, hide the video element as only controls are needed.
videoElem.style.display = "none";
// Stop monitoring the availability of remote playback devices.
videoElem.remote.cancelWatchAvailability();
};
function switchToLocalUI() {
// Show the video element.
videoElem.style.display = "inline";
// Start watching the device availability again.
videoElem.remote.watchAvailability(availabilityCallback);
};
<script>
A local playback device is the device the browsing context is running on along with the default video/audio outputs the device has.
A remote playback device is any other device but the local playback device that the browsing context can use to play media on.
A media element state is the set of all single media element properties observable by the page and/or the user via the user agent implementation. The new properties introduced by this spec are not considered part of the media element state for convenience.
paused attribute or the pause/resume
button reflecting that state on the default controls of the media
element would be a part of media element state.
A local playback state is the user agent implementation of media element state for the particular media element for playback on the local playback device.
A remote playback state is the user agent implementation of media element state for the particular media element for playback on the certain remote playback device.
For a good user experience it is important that the media element
state doesn't change unexpectedly when the
state changes. It
is also important that remote playback state is in sync with
the media element state so when the media is paused on the
remote playback device it looks paused to both the user and
the page.
RemotePlayback interface
[Exposed=Window]
interface RemotePlayback : EventTarget {
Promise<long> watchAvailability(RemotePlaybackAvailabilityCallback callback);
Promise<void> cancelWatchAvailability(optional long id);
readonly attribute RemotePlaybackState state;
attribute EventHandler onconnecting;
attribute EventHandler onconnect;
attribute EventHandler ondisconnect;
Promise<void> prompt();
};
enum RemotePlaybackState {
"connecting",
"connected",
"disconnected"
};
callback RemotePlaybackAvailabilityCallback = void(boolean available);
A RemotePlayback object is an interface allowing the page to detect availability of, connect to and control playback on remote playback devices.
A RemotePlaybackState enumeration represents the state of connection to some remote playback device.
A RemotePlaybackAvailabilityCallback represents the remote playback device availability information.
A RemotePlaybackAvailabilityCallback is the way
for the page to obtain the remote playback device
availability for the corresponding media element. If
the user agent can monitor the list of available remote playback
devices in the background (without a pending request to
prompt()), the
RemotePlaybackAvailabilityCallback behavior
defined below MUST be implemented by the user agent. Otherwise, the
promise returned by watchAvailability() MUST be rejected
with NotSupportedError.
The user agent MUST keep track of the set of
availability callbacks registered with each media
element through the watchAvailability() method. The
set of availability callbacks for each
RemotePlayback object is represented as a set of tuples
(callbackId, callback), initially
empty, where:
watchAvailability()
in a given browsing context;
Since there's one and only one RemotePlayback object per each media element, set of availability callbacks of a media element is the same set as the set of availability callbacks of the RemotePlayback object referred to by the element's remote property.
The combined set of all sets of availability callbacks of all RemotePlayback objects known to the browsing context is referred to as global set of availability callbacks.
The user agent MUST keep a list of available remote playback devices. This list contains remote playback devices and is populated based on an implementation specific discovery mechanism. It is set to the most recent result of the algorithm to monitor the list of available remote playback devices or an empty list if the algorithm hasn't been run yet.
The user agent MAY not support running the algorithm to
monitor the list of available remote playback devices
continuously, for example, because of platform or power
consumption restrictions. In this case the promise returned by
watchAvailability() MUST be rejected
with NotSupportedError, the global set of availability
callbacks will be empty and the algorithm to monitor the
list of available remote playback devices will only run as
part of the initiate remote playback algorithm.
When the global set of availability callbacks is not empty, the user agent MUST monitor the list of available remote playback devices continuously, so that pages can keep track of the last value received via the registered callbacks to offer remote playback only when there are available devices.
The user agent is expected not to monitor the list of available remote playback devices when possible, to satisfy the power saving non-functional requirement. For example, the user agent might choose not to run the monitoring algorithm when the global set of availability callbacks is empty, when every page that has media elements with non-empty set of availability callbacks is in the background.
Some remote playback devices may only be able to play a subset of media resources because of functional, security or hardware limitations. Examples are set-top boxes, smart TVs or networked speakers capable of rendering only certain formats of video and/or audio. We say that such a device is a compatible remote playback device for a media resource if the user agent can reasonably guarantee that the remote playback of the media specified by the resource will succeed on that device.
The user agent can use the srclang
attribute of the track element as a hint of
the language of the text track data to help identify a
compatible remote playback device.
The media resources of a media element, that were considered by the user agent to find a compatible remote playback device, are called the availability sources set.
The media resource of a media element, that is used to initiate remote playback on the selected remote playback device is called remote playback source. Remote playback source MUST belong to availability sources set.
The mechanism of picking the availability sources set and the remote playback source is implementation-specific. For example, the user agent MUST either use the currentSrc of the media element for both availability monitoring and remote playback or use all the media resources associated with the media element as the availability sources set and pick one of the resources as the remote playback source after user selects the remote playback device.
Remote playback is said to be unavailable for the
media element if the list of available remote playback
devices is empty or none of them is compatible with any
source from availability sources set for the media
element. The remote playback is said to be
available otherwise. A boolean set to
false if the remote playback is unavailable
for the media element or true if it is
available is called availability for the
media element.
If the user agent stops
monitoring the list of available remote playback devices
(for example by a user control or for power saving), it SHOULD
invoke all callbacks in the global set of availability
callbacks with false so that pages can update
their user experience appropriately. It SHOULD also set
the availability value for all
media elements to false so that availability
information can be propagated correctly if the user agent later
resumes
monitoring the list of available remote playback devices.
When the watchAvailability() method is
called, the user agent MUST run the following steps:
false as its argument.
A simple algorithm for assigning callbackId values is to keep a counter for each browsing context and incrementing it in step 6.
If the set of availability callbacks is non-empty, or there is a pending request to initiate remote playback, the user agent MUST monitor the list of available remote playback devices by running the following steps:
When a cancelWatchAvailability() method is
called, the user agent MUST run the following steps:
undefined,
clear the set of availability callbacks.
When the prompt() method is called, the
user agent MUST run the following steps:
prompt()
would not be able to show any UI.
state or
the list of available remote playback devices),
reject promise with a NotSupportedError and abort
all remaining steps.
By picking a remote playback device the user grants permission to use the device.
state
attribute
The state
attribute represents the RemotePlayback connection's current
state. It can take one of the values of RemotePlaybackState
depending on the connection state:
promise returned by prompt() is fulfilled. The local
playback of the media element continues in this state and media
commands still take effect on the local playback state.
prompt().
When the user agent is to establish a connection with the remote playback device, it MUST run the following steps:
The user agent SHOULD pause local audio and video output of the media element while the remote playback state is connected.
If the user agent is exposing a user interface to the user for the media element (i.e., using default controls), the user agent SHOULD convey the fact that the remote playback state is connected through an icon or other means.
Accept-Language header it sends to fetch media
resources.
A user agent MAY support connecting to a remote playback device from the browser, e.g. by including appropriate controls to the user interface that is exposed to the user. This feature is known as browser initiated remote playback. A user agent that supports browser initiated remote playback SHOULD initiate the remote playback only when the user has expressed an intention to do so via a user gesture, for example by clicking a button in the browser.
If the user agent supports browser initiated remote playback, it MUST support the state attribute and the corresponding events by following the algorithms to establish a connection with the remote playback device and disconnect from a remote playback device.
The HTMLMediaElement interface interacts with the remotely played media as soon as the connection with the remote playback device is established.
In particular, as soon as the state of a RemotePlayback object has changed to connected, the user agent MUST send all the media commands issued on the HTMLMediaElement object with which the RemotePlayback object is associated to the remote playback device in order to change the remote playback state vs the local playback state.
Similarly, the user agent MUST reflect all updates of the remote playback state received from the remote playback device on the media element state.
If sending any command fails, the user agent MAY disconnect from a remote playback device.
The remote playback device can implement a subset of the capabilities of the playback engine of the user agent, and some HTMLMediaElement APIs do not always make sense to use during remote playback. In this case, the local playback state is expected to reflect as closely as possible the actual remote playback state after a media command that is not supported during remote playback.
For example, after calling fastSeek() while
connected to a remote playback device that does not
support it, the seeking attribute of the
HTMLMediaElement is expected to remain false
and no seeking event is to be fired.
When the user agent is to disconnect from a remote playback device, it MUST do the following:
state
of remote is disconnected, abort all
remaining steps.
state to
disconnected.
If the remote playback device is abruptly disconnected during playback (for example, by power loss or a network disconnection), the user agent SHOULD run the steps to monitor the list of available remote playback devices before the steps to disconnect from a remote playback device. This allows callbacks in the set of availability callbacks to be invoked before the disconnect event is fired, so the page can update itself to show resumption of playback is not possible.
disconnected state.
The following are the event handlers (and their corresponding event handler event types) that must be supported, as event handler IDL attributes, by objects implementing the RemotePlayback interface:
| Event handler | Event handler event type |
|---|---|
onconnecting
|
connecting
|
onconnect
|
connect
|
ondisconnect
|
disconnect
|
HTMLMediaElement
partial interface HTMLMediaElement {
[SameObject] readonly attribute RemotePlayback remote;
[CEReactions] attribute boolean disableRemotePlayback;
};
The remote attribute MUST return the RemotePlayback instance associated with the media element.
disableRemotePlayback
attribute
Some pages may wish to disable remote playback of a media element;
for example, they may prefer to use a PresentationRequest to
present a complete document on a presentation screen. To support
this use case, a new disableRemotePlayback attribute is
added to the list of content attributes for audio
and video elements.
A corresponding
disableRemotePlayback
IDL attribute which reflects the value of each element’s
disableRemotePlayback content attribute is added to
the HTMLMediaElement interface.
The disableRemotePlayback
IDL attribute MUST reflect the content attribute of the same
name.
If the disableRemotePlayback attribute is present on the media element, the user agent MUST NOT play the media element remotely or present any UI to do so.
When the disableRemotePlayback attribute is added to the media element, the user agent MUST run the steps to disable remote playback:
disconnected, run the disconnect from a remote
playback device algorithm for the remote playback device
the media element is connected or connecting to.
Firing the callback provided via the
watchAvailability() method reveals one
bit of information about the presence (or non-presence) of a
remote playback device typically discovered through the local
area network. This could be used in conjunction with other
information for fingerprinting the user. However, this information is
also dependent on the user's local network context, so the risk is
minimized. Also, by design, the human readable name of a remote
playback device is not revealed to the page.
The API enables monitoring the list of available remote playback devices. How the user agent determines the compatibility and availability of a remote playback device with a media element's resource is an implementation detail. If a user agent matches a media resource to a particular type of device to determine its availability, this feature can be used to probe information about which remote playback device the user has without user consent.
The user agent SHOULD NOT monitor the list of available remote playback devices if the user disables background monitoring through a browser setting.
When the user is asked permission to use a remote playback device during the steps to change remote playback state, the user agent should make it clear what origin the request is coming from.
Display of the origin requesting remote playback will help the user understand what content is making the request, especially when the request is initiated from a nested browsing context. For example, embedded content may try to convince the user to click to trigger a request to start an unwanted remote playback.
Showing the origin that will be presented will help the user know
if that content is from an potentially secure (e.g.,
https:) origin, and corresponds to a known or
expected site.
The Remote Playback API abstracts away what "local" means for displays, meaning that it exposes network-accessible displays as though they were local displays. The Remote Playback API requires user permission for a page to access any display to mitigate issues that could arise, such as showing unwanted content on a display viewable by others.
This spec will not mandate communication protocols between the local playback device and the remote playback device, but it should set some guarantees of message confidentiality and authenticity between them.
For this specification to be advanced to Proposed Recommendation, there must be at least two independent, interoperable implementations of each feature. Each feature may be implemented by a different set of products, there is no requirement that all features be implemented by a single product. Additionally, implementations must demonstrate support for the media remoting and media flinging cases, either within the same product or within different products.
For the purposes of these criteria, we define the following terms: