Depth map

A depth map is an abstract representation of a frame of a depth stream track. A depth map is an image that contains information relating to the distance of the surfaces of scene objects from a viewpoint.

A depth map has an associated near value which is a double. It represents the minimum range in meters.

A depth map has an associated far value which is a double. It represents the maximum range in meters.

A depth map has an associated focal length which is a double. It represents the focal length of the camera in millimeters.

A depth map has an associated horizontal field of view which is a double. It represents the horizontal angle of view in degrees.

A depth map has an associated vertical field of view which is a double. It represents the vertical angle of view in degrees.

The data type of a depth map is 16-bit unsigned integer. The algorithm to convert the depth map value to grayscale, given a depth map value d, is as follows:

1. Let near be the the near value.
2. Let far be the the far value.
3. Apply the rules to convert using range linear to d to obtain quantized value d_16bit.
4. Return d_16bit.

The rules to convert using range linear are as given in the following formula:

$d_n=\frac{d-near}{far-near}$

$d_{16bit}=\lfloor d_n\cdot 65535\rfloor$

MediaStreamConstraints dictionary

          partial dictionary MediaStreamConstraints {
              (boolean or MediaTrackConstraints) depth = false;
          };
        

If the depth dictionary member has the value true, the MediaStream returned by the getUserMedia() method MUST contain a depth stream track. If the depth dictionary member is set to false, is not provided, or is set to null, the MediaStream MUST NOT contain a depth stream track. If the depth dictionary member is set to a valid Constraints dictionary, the MediaStream returned by the getUserMedia() method MUST contain a depth stream track that fulfills the specified mandatory constraints.

The permission associated with a depth camera source is "camera",

MediaStream interface

          partial interface MediaStream {
              sequence<MediaStreamTrack> getDepthTracks();
          };
        

The getDepthTracks() method, when invoked, MUST return a sequence of depth stream tracks in this stream.

The getDepthTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set whose kind is equal to "depth". The conversion from the track set to the sequence is user agent defined and the order does not have to be stable between calls.

The MediaStream consumer for the depth-only stream and depth+video stream is the video element [[!HTML]].

MediaStreamTrack interface

The kind attribute MUST, on getting, return the string "depth" if the object represents a depth stream track.

If a MediaStreamTrack of kind "depth" is muted or disabled, it MUST render black frames, or a zero-information-content equivalent.

MediaDeviceInfo interface

The string "depthinput" is the MediaDeviceKind value for the depth camera input device.

Media provider object

A media provider object can represent a depth-only stream (and specifically, not a depth+video stream). The user agent MUST support a HTMLMediaElement with an assigned media provider object that is a depth-only stream, and in particular, the srcObject IDL attribute that allows the HTMLMediaElement to be assigned a media provider object MUST, on setting and getting, behave as specified in [[!HTML]].

The video element

When a video element is potentially playing and its assigned media provider object is a depth-only stream, the user agent MUST, for each pixel of the media data that is represented by a depth map, given a depth map value d, convert the depth map value to grayscale and render the returned value to the screen.

For a video element whose assigned media provider object is a depth+video stream, the user agent MUST act as if all the MediaStreamTracks of kind "depth" were removed prior to when the video element is potentially playing.

ImageData interface

When an ImageData object's pixel values represent a depth map (that is, the image source for the 2D rendering context CanvasImageSource is a HTMLVideoElement whose media data represents a depth map), its Uint8ClampedArray source assigned to the data attribute represents the 16-bit depth value by assigning the low 8-bit of the 16-bit depth value d_low8bit on its red component, and high 8-bit of the 16-bit depth value d_high8bit on its green component.

If the values are read from the default Uint8ClampedArray view, they are represented as Canvas Pixel ArrayBuffer data as follows:

red_8bit = d_low8bit

green_8bit = d_high8bit

blue_8bit = 0

alpha_8bit = 0

          var depthVideo = document.querySelector('video');
          var canvas = document.querySelector('canvas');
          var context = canvas.getContext('2d');
          var w = depthVideo.videoWidth, h = depthVideo.videoHeight;

          context.drawImage(depthVideo, 0, 0, w, h);
          var imageData = context.getImageData(0, 0, w, h);

          // Create a new DataView dv on to an ArrayBuffer buffer
          // that exposes it as an array of unsigned 16-bit integers.
          var dv = new DataView(imageData.data.buffer);

          // Read every fourth unsigned 16-bit value in little endian representation.
          for (var i = 0; i < imageData.data.length; i += 4) {
            console.log(dv.getUint16(i, true));
          }

          // Alternatively, reconstruct unsigned 16-bit value. The result
          // is the same as if DataView and getUint16() was used.
          for (var i = 0; i < imageData.data.length; i += 4) {
            console.log(imageData.data[i] + (imageData.data[i+1] << 8));
          }
        

MediaTrackSettings dictionary

When the getSettings() method is invoked on a depth stream track, the user agent MUST return the following dictionary that extends the MediaTrackSettings dictionary:

          partial dictionary MediaTrackSettings {
              double              near;
              double              far;
              double              focalLength;
              double              horizontalFieldOfView;
              double              verticalFieldOfView;
          };
        

Constrainable properties

The near, far, focalLength, horizontalFieldOfView, and verticalFieldOfView constrainable properties are defined to apply only to depth stream tracks.

The near and far constrainable properties, when set, allow the implementation to pick the best depth camera mode optimized for the range [near, far] and help minimize the error introduced by the lossy conversion from the depth value d to a quantized d_8bit and back to an approximation of the depth value d.

If the far property's value is less than the near property's value, the depth stream track is overconstrained.

If the near value, far value, focal length, horizontal field of view, or vertical field of view is fixed due to a hardware or software limitation, the corresponding constrainable property's value MUST be set to the value reported by the underlying implementation. (For example, the focal length of the lens may be fixed, or the underlying platform may not expose the focal length information.)

          partial dictionary MediaTrackConstraintSet {
            ConstrainDouble near;
            ConstrainDouble far;
            ConstrainDouble focalLength;
            ConstrainDouble horizontalFieldOfView;
            ConstrainDouble verticalFieldOfView;
          };

          partial dictionary MediaTrackSupportedConstraints {
            boolean near = true;
            boolean far = true;
            boolean focalLength = true;
            boolean horizontalFieldOfView = true;
            boolean verticalFieldOfView = true;
          };

          partial dictionary MediaTrackCapabilities {
            (double or DoubleRange) near;
            (double or DoubleRange) far;
            (double or DoubleRange) focalLength;
            (double or DoubleRange) horizontalFieldOfView;
            (double or DoubleRange) verticalFieldOfView;
          };
        

WebGLRenderingContext interface