HTML5 Video Playback UI

by Kristofer Baxter

In the past we’ve written about HTML5 Video (HTML5 Video in IE11 on Windows 8.1, and HTML5 Video at Netflix) but we haven't spoken much about how we built the player UI. The UI Engineering team here at Netflix has been supporting HTML5 based playback for a little over a year, and now seems like the right time to discuss some of the strategies and techniques we are using to support video playback without a plugin.

One of our main objectives is to keep Netflix familiar to our members. That means we’re keeping the design of the HTML5 player consistent with our Silverlight experience. Features should be rolled out simultaneously for the two platforms. However, HTML5 users will enter playback faster, can enjoy 1080p content when GPU accelerated, and keep all the functionality they know and love.

Silverlight UI	HTML5 UI

In order to achieve a similar look and feel, we needed to recreate a few key elements of the Silverlight UI:

1. Scale interface to users resolution
2. Minimize Startup time via minimal dependency on data
3. Ensure High Performance on low end hardware

Scaling interface to users resolution

No matter what resolution the browser window used for playback is, our current playback UI ensures all of the controls maintain the same percentage size on screen. This lets users choose their own dimensions for playing content without the UI getting in the way.

Normally, a modern web application could implement this using CSS vw and vh units. However, we found this approach to be inadequate for our needs. Our player can be displayed in two fashions -- taking over the entire initial containing block of a viewport, or a smaller portion. To solve for this, we implemented a sizing scheme based entirely on font-relative lengths.

In this small example, you can see the scaling implementation in a direct form.

<style>
    .netflix-player-wrapper {
        font-size: 16px;
    }
    #netflix-player {
        position: absolute;
        width: 90%; height: 90%;
        left: 5%; top: 5%;
        overflow: hidden;
        background: #ccc;
        font-size: 1em;
    }
    #player-sizing {
        position: absolute;
        width: 1em; height: 1em;
        visibility: hidden;
        font-size: 1em;
    }
    #ten-percent-height {
        position: absolute;
        width: 80%; height: 10em;
        left: 10%; bottom: 8em;
        background: #000;
        display: flex;
    }    
    #ten-percent-height > p {
        display: block;
        margin: 1em;
        font-size: 2em;
        color: #fff;
    }
</style>
<div class="netflix-player-wrapper">
    <div id="netflix-player">
        <div id="player-sizing"></div>
        <div id="ten-percent-height"><p>Text</p></div>
    </div>
</div>
<script>
(function () {
    var sizingEl = document.getElementById("player-sizing"),
        controlWrapperEl = document.getElementById('netflix-player'),
        currentEmSize = 1.0;
                   
    function resize() {
        var wrapperHeight = controlWrapperEl.getBoundingClientRect().height,
            sizingHeight = sizingEl.getBoundingClientRect().height,
            wrapperOnePercentHeight = wrapperHeight / 100,
            offsetSize;

        if (sizingHeight > wrapperOnePercentHeight) {
            offsetSize = sizingHeight / wrapperOnePercentHeight;
            currentEmSize = currentEmSize / offsetSize;
        } else if (wrapperOnePercentHeight > sizingHeight) {
            offsetSize = wrapperOnePercentHeight / sizingHeight;
            currentEmSize = currentEmSize * offsetSize;
        }
        controlWrapperEl.style.fontSize = currentEmSize + "em";
    }
                
    window.addEventListener("resize", resize, false);
    resize();
})();
</script>

We implement this resizing functionality on a debounced interval in the player UI. Triggering it on every window resize would be wasteful.

By making an em unit represent 1% height of the "netflix-player" container, we can size all of our onscreen elements in a scaling manner - no matter how or where the netflix-player container is placed in the document.

Minimize Startup time via minimal dependency on data

Browser plugins like Flash and Silverlight can take several seconds to initialize, especially on a freshly booted machine. Now that we no longer need to initialize a plugin to play content, we can begin playback faster. However, we learned a lot about quick video startup in Silverlight, and can borrow techniques we developed to make our HTML5 UI launch content even faster.

When possible, allow playback to begin without title metadata.

If we already know which title the customer has selected to play (like a specific episode or movie), we can just start playback of that title immediately. Once the user has begun to buffer content, the UI can request display metadata. Metadata for the player can be a large payload since it includes episode data (title, synopsis, predicted rating), and is personalized to the user. By delaying the retrieval of metadata, users begin streaming 500 to 1200ms sooner in real-world usage.

For other conditions, such when a customer clicks play on a TV show and we want to start playback at the last episode that they were watching, we retrieve the specific episode the user wants before starting the playback process.

Populate controls which depend on rich data as that data becomes available.

Since we can begin playback before the player UI knows anything except which title to play, the player UI needs to be resilient against missing metadata. We display a minimal number of controls while this data is being requested. These controls include play/pause, exit playback, and full-screen toggling.

We use an eventing framework to let individual components know when data state has changed, so each component can stay decoupled. Here’s an example showing how we handle an event telling us the metadata is now loaded for the title.

function populateStatus() {
    if (Metadata.videoIsKnown(ObjectPool.videoId())) {
        // Update Status to reflect current playing item.
    } else {
        // Hide or remove current status
    }
}

Metadata.addEventListener(Metadata.knownEvents.METADATA_LOADED, populateStatus);

Ensure High Performance on all hardware

Not everyone has the latest and greatest hardware at their disposal, but this shouldn't prevent all sorts of devices from playing content on Netflix. To this end, we develop using a wide variety of hardware and test using a wide range of representative devices.

We’ve found the issues preventing great performance on low end hardware can mostly be avoided by adhering to the following best practices:

Avoid repaints and reflows whenever possible.

Reflows and repaints while playing content is quite costly to overall performance and battery life. As a result, we batch reads and writes to the DOM wherever possible. This helps us avoid accidental reflows.

Take advantage of getBoundingClientRect to determine the size of object.

This is a very fast way to get the dimensions of an object. However, it isn’t a free operation and results should be cached whenever possible.

Caching the size of objects when dragging, instead of recalculating them every time they are needed, is one such way to reduce the number of calls in quick succession.

function setupPointerData(e) {
    pointerEventData.dimensions = {
        handleEl:  handleEl.getBoundingClientRect(),
        wrapperEl: wrapperEl.getBoundingClientRect()
    };
    pointerEventData.drag = {
        start: { value: currentValue, max: currentMax },
        pointer: { x: e.pageX, y: e.pageY }
    };
}

function pointerDownHandler(e) {
    if (handleEl.contains(e.target)) {
        if (!dragging) {
            setupPointerData(e);
            dragging = true;
        }
    }
}

function pointerMoveHandler(e) {
    if (dragging && isValidEventLocation(e)) {
        if (!pointerEventData || !pointerEventData.dimensions) {
            setupPointerData(e);
        }
        // Use the handleEl dimensions, wrapperEl dimensions, 
        // and the event values to change the DOM.
    }
}

We have a lot of work planned

We’re working on exciting new features and constantly improving our HTML5 Video UI, and we’re looking for help. Our growing team is looking for experts to join us. If you’d like to apply, take a look here.

Building the Netflix UI for Wii U

Hello, my name is Joubert Nel and I’m a UI engineer on the TV UI team here at Netflix. Our team builds the Netflix experiences for hundreds of TV devices, like the PlayStation 3, Wii, Apple TV, and Google TV.

We recently launched on Nintendo’s new Wii U game console. Like other Netflix UIs, we present TV shows and movies we think you’ll enjoy in a clear and fast user interface. While this UI introduces the first Netflix 1080p browse UI for game consoles, it also expands on ideas pioneered elsewhere like second screen control.

Virtual WebKit Frame

Like many of our other device UIs, our Wii U experience is built for WebKit in HTML5. Since the Wii U has two screens, we created a Virtual WebKit Frame, which partitions the UI into one area that is output to TV and one area that is output to the GamePad.

This gives us the flexibility to vary what is rendered on each screen as the design dictates, while sharing application state and logic in a single JavaScript VM. We also have a safe zone between the TV and GamePad areas so we can animate elements off the edge of the TV without appearing on the GamePad.

We started off with common Netflix TV UI engineering performance practices such as view pooling and accelerated compositing. View pooling reuses DOM elements to minimize DOM churn, and Accelerated Compositing (AC) allows us to designate certain DOM elements to be cached as a bitmap and rendered by the Wii U’s GPU.

In WebKit, each DOM node that produces visual output has a corresponding RenderObject, stored in the Render Tree. In turn, each RenderObject is associated with a RenderLayer. Some RenderLayers get backing surfaces when hardware acceleration is enabled . These layers are called compositing layers and they paint into their backing surfaces instead of the common bitmap that represents the entire page. Subsequently, the backing surfaces are composited onto the destination bitmap. The compositor applies transformations specified by the layer’s CSS -webkit-transform to the layer’s surface before compositing it. When a layer is invalidated, only its own content needs to be repainted and re-composited. If you’re interested to learn more, I suggest reading GPU Accelerated Compositing in Chrome.


Performance

After modifying the UI to take advantage of accelerated compositing, we found that the frame rate on device was still poor during vertical navigation, even though it rendered at 60fps in desktop browsers.

When the user browses up or down in the gallery, we animate 4 rows of poster art on TV and mirror those 4 rows on the GamePad. Preparing, positioning, and animating only 4 rows allows us to reduce (expensive) structural changes to the DOM while being able to display many logical rows and support wrapping. Each row maintains up to 14 posters, requiring us to move and scale a total of 112 images during each up or down navigation. Our UI’s posters are 284 x 405 pixels and eat up 460,080 bytes of texture memory each, regardless of file size. (You need 4 bytes to represent each pixel’s RGBA value when the image is decompressed in memory.)

Layout of poster art in the gallery

To improve performance, we tried a number of animation strategies, but none yielded sufficient gains. We knew that when we kicked off an animation, there was an expensive style recalculation. But the WebKit Layout & Rendering timeline didn’t help us figure out which DOM elements were responsible.

WebKit Layout & Rendering Timeline

We worked with our platform team to help us profile WebKit, and we were now able to see how DOM elements relate to the Recalculate Style operations.

Our instrumentation helps us visualize the Recalculate Style call stack over time:

Instrumented Call Stack over Time

Through experimentation, we discovered that for our UI, there is a material performance gain when setting inline styles instead of modifying classes on elements that participate in vertical navigation.

We also found that some CSS selector patterns cause deep, expensive Recalculate Style operations. It turns out that the mere presence of the following pattern in CSS triggers a deep Recalculate Style:

.list-showing #browse { … }

Moreover, a -webkit-transition with duration greater than 0 causes the Recalculate Style operations to be repeated several times during the lifetime of the animation.
After removing all CSS selectors of this pattern, the resulting Recalculate Style shape is shallower and consumes less time.


Delivering great experiences

Our team builds innovative UIs, experiments with new concepts using A/B testing, and continually delivers new features. We also have to make sure our UIs perform fast on a wide range of hardware, from inexpensive consumer electronics devices all the way up to more powerful devices like the Wii U and PS3.

If this kind of innovation excites you as much as it does me, join our team!