Abstract

Video production is a collaborative process in which stakeholders regularly review drafts of the edited video to indicate problems and offer suggestions for improvement. Although practitioners prefer in-person feedback, most reviews are conducted asynchronously via email due to scheduling and location constraints. The use of this impoverished medium is challenging for both providers and consumers of feedback. We introduce VidCrit, a system for providing asynchronous feedback on drafts of edited video that incorporates favorable qualities of an in-person review. This system consists of two separate interfaces: (1) A feedback recording interface captures reviewers' spoken comments, mouse interactions, hand gestures and other physical reactions. (2) A feedback viewing interface transcribes and segments the recorded review into topical comments so that the video author can browse the review by either text or timelines. Our system features novel methods to automatically segment a long review session into topical text comments, and to label such comments with additional contextual information. We interviewed practitioners to inform a set of design guidelines for giving and receiving feedback, and based our system's design on these guidelines. Video reviewers using our system preferred our feedback recording interface over email for providing feedback due to the reduction in time and effort. In a fixed amount of time, reviewers provided 10.9 (σ=5.09) more local comments than when using text. All video authors rated our feedback viewing interface preferable to receiving feedback via e-mail.

The VidCrit interface consists of a direct navigation pane for navigating the feedback session using the webcam and source video timelines, and a segmented comments pane for reviewing transcribed and segmented critiques. The direct navigation pane features the source and webcam videos along with a title (A), a feedback session timeline (B), a source video timeline (C) and the source video transcript (D). The segmented comments pane features sorting, filtering and search options (E), along with a list of segmented comments (F).

Research Paper

PDF (10M)

Video

Abstract

We present QuickCut, an interactive video editing tool designed to help authors efficiently edit narrated video. QuickCut takes an audio recording of the narration voiceover and a collection of raw video footage as input. Users then review the raw footage and provide spoken annotations describing the relevant actions and objects in the scene. QuickCut time-aligns a transcript of the annotations with the raw footage and a transcript of the narration to the voiceover. These aligned transcripts enable authors to quickly match story events in the narration with semantically relevant video segments and form alignment constraints between them. Given a set of such constraints, QuickCut applies dynamic programming optimization to choose frame-level cut points between the video segments while maintaining alignments with the narration and adhering to low-level film editing guidelines. We demonstrate QuickCut's effectiveness by using it to generate a variety of short (less than 2 minutes) narrated videos. Each result required between 14 and 52 minutes of user time to edit (i.e. between 8 and 31 minutes for each minute of output video), which is far less than typical authoring times with existing video editing workflows.

Research Paper

PDF

Video

Abstract

Searching for scenes in movies is a time-consuming but crucial task for film studies scholars, film professionals, and new media artists. Our formative interviews reveal that such users search for a wide variety of entities — actions, props, dialogue phrases, character performances, locations — and they return to particular scenes they have seen in the past. Today, these users find relevant clips by watching the entire movie, scrubbing the video timeline, or navigating with opaque DVD chapter menus. We introduce SceneSkim, a tool for searching and browsing movies using synchronized captions, scripts and plot summaries. Our interface integrates information from different documents to allow expressive search at several levels of granularity: Captions provide access to accurate dialogue, scripts describe shot-by-shot actions and settings, and plot summaries contain high-level event descriptions. We propose new algorithms for finding word-level caption to script alignments, parsing text scripts, and aligning plot summaries to scripts. Film studies graduate students evaluating SceneSkim expressed enthusiasm about the usability of the proposed system for their research and teaching.

The SceneSkim interface consists of a search pane for finding clips matching a query and a movie pane for browsing within movies using synchronized documents. The search pane features a keyword search bar (A), search filters (B) and a search results view (C). The movie pane includes the synchronized summary, script, captions, and movie.

Research Paper

PDF (4.6M)

Video

Abstract

Well-performed audio narrations are a hallmark of captivating podcasts, explainer videos, radio stories, and movie trailers. To record these narrations, professional voiceover actors follow guidelines that describe how to use low-level vocal components---volume, pitch, timbre, and tempo---to deliver performances that emphasize important words while maintaining variety, flow and diction. Yet, these techniques are not well-known outside the professional voiceover community, especially among hobbyist producers looking to create their own narrations. We present Narration Coach, an interface that assists novice users in recording scripted narrations. As a user records her narration, our system synchronizes the takes to her script, provides text feedback about how well she is meeting the expert voiceover guidelines, and resynthesizes her recordings to help her hear how she can speak better.

Caption text goes here.

Research Paper

PDF (6.9M)

Results

Narrations recorded using our tool in a pilot study

Video

MP4 (63M)

Abstract

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer’s experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

CrowdCrit allows designers to submit preliminary designs to be critiqued by crowds and clients. The system then aggregates and visualizes the critiques for designers.

Research Paper

PDF (10.4M)

Abstract

Designers often create physical works-like prototypes early in the product development cycle to explore possible mechanical architectures for a design. Yet, creating functional prototypes requires time and expertise, which discourages rapid design iterations. Designers must carefully specify part and joint parameters to ensure that parts move and fit and together in the intended manner. We present an interactive system that streamlines the process by allowing users to annotate rough 3D models with high-level functional relationships (e.g., part A fits inside part B). Based on these relationships, our system optimizes the model geometry to produce a working design. We demonstrate the versatility of our system by using it to design a variety of works-like prototypes.

Creating works-like prototypes. Users start by creating a rough 3D model of a design and then specifying the desired functional relationships between parts (a). Our system optimizes part and joint parameters to generate a working model (b) that can be fabricated as a physical prototype (c).

Research Paper

PDF (10.3M)

Video

MOV (104M)

Abstract

We present a method for automatically identifying and validating predictive relationships between the visual appearance of a city and its non-visual attributes (e.g. crime statistics, housing prices, population density etc.). Given a set of street-level images and (location, city-attribute-value) pairs of measurements, we first identify visual elements in the images that are discriminative of the attribute. We then train a predictor by learning a set of weights over these elements using non-linear Support Vector Regression. To perform these operations efficiently, we implement a scalable distributed processing framework that speeds up the main computational bottleneck (extracting visual elements) by an order of magnitude. This speedup allows us to investigate a variety of city attributes across 6 different American cities. We find that indeed there is a predictive relationship between visual elements and a number of city attributes including violent crime rates, theft rates, housing prices, population density, tree presence, graffiti presence, and the perception of danger. We also test human performance for predicting theft based on street-level images and show that our predictor outperforms this baseline with 33% higher accuracy on average. Finally, we present three prototype applications that use our system to (1) define the visual boundary of city neighborhoods, (2) generate walking directions that avoid or seek out exposure to city attributes, and (3) validate user-specified visual elements for prediction.

The violent crime rate in San Francisco is an example of a non-visual city attribute that is likely to have a strong relationship to visual appearance. Our method automatically computes a predictor that models this relationship, allowing us to predict violent crime rates from streetlevel images of the city. Across the city our predictor achieves 73% accuracy compared to ground truth. (columns 1 and 2, heatmaps run from red indicating a high violent crime rate to blue indicating a low violent crime rate). Specifically, our predictor models the relationship between visual elements (column 3), including fire escapes on fronts of buildings, high-density apartment windows, dilapidated convenience store signs, and unique roof style, relate to increased violent crime rates. Our predictor also identifies street-level images from San Francisco that have an unsafe visual appearance (column 4). Detections of visual elements are outlined in color.

Research Paper

PDF (12M)

Supplemental Material

PDF (27M)

Abstract

The D3 JavaScript library has become a ubiquitous tool for developing visualizations on the Web. Yet, once a D3 visualization is published online its visual style is difficult to change. We present a pair of tools for deconstructing and restyling existing D3 visualizations. Our deconstruction tool analyzes a D3 visualization to extract the data, the marks and the mappings between them. Our restyling tool lets users modify the visual attributes of the marks as well as the mappings from the data to these attributes. Together our tools allow users to easily modify D3 visualizations without examining the underlying code and we show how they can be used to deconstruct and restyle a variety of D3 visualizations.

The mappings from a bar chart built using D3 before (left column) and after (right column) deconstruction and restyling with our tools. Deconstructed mappings are shown to the right of the original visualization, and the mappings after restyling are shown to the right of the restyled result along with any unmapped attributes that were changed in the restyling. We use the arrow notation to indicate linear and categorical mappings. Mappings removed and added during restyling are highlighted in red and and green respectively. Changed mappings and attributes are highlighted in blue.

The original bar chart shows the 20 countries with the highest unemployment rates sorted by unemployment rate along the y-axis. We restyle the chart into a dot plot. Original visualization by Leon du Toit (http://bl.ocks.org/leondutoit/6436923/).

Research Paper

PDF (8.6M)

Additional Materials

D3 Deconstructor Chrome extension with source code

Preview Video

MP4 (4.3M)

Abstract

Increasingly, authors are publishing long informational talks, lectures, and distance-learning videos online. However, it is difficult to browse and skim the content of such videos using current timeline-based video players. Video digests are a new format for informational videos that afford browsing and skimming by segmenting videos into a chapter/section structure and providing short text summaries and thumbnails for each section. Viewers can navigate by reading the summaries and clicking on sections to access the corresponding point in the video. We present a set of tools to help authors create such digests using transcript-based interactions. With our tools, authors can manually create a video digest from scratch, or they can automatically generate a digest by applying a combination of algorithmic and crowdsourcing techniques and then manually refine it as needed. Feedback from first-time users suggests that our transcript-based authoring tools and automated techniques greatly facilitate video digest creation. In an evaluative crowdsourced study we find that given a short viewing time, video digests support browsing and skimming better than timeline-based or transcript-based video players.

A video digest affords browsing and skimming through a chapter/section organization of the video content. The chapters are topically coherent segments of the video that contain major themes in the presentation. Each chapter is further subdivided into a set of sections, that each provide a brief text summary of the corresponding video segment as well as a representative keyframe image. Clicking within a section plays the video starting at the beginning of the corresponding video segment.

Research Paper

PDF (49M) | PDF (11M)

Results

Video digests generated with our system

Abstract

Highly-produced audio stories often include musical scores that reflect the emotions of the speech. Yet, creating effective musical scores requires deep expertise in sound production and is time-consuming even for experts. We present a system and algorithm for re-sequencing music tracks to generate emotionally relevant music scores for audio stories. The user provides a speech track and music tracks and our system gathers emotion labels on the speech through hand-labeling, crowdsourcing, and automatic methods. We develop a constraint-based dynamic programming algorithm that uses these emotion labels to generate emotionally relevant musical scores. We demonstrate the effectiveness of our algorithm by generating 20 musical scores for audio stories and showing that crowd workers rank their overall quality significantly higher than stories without music.

Our algorithm re-sequences the beats (circles) of the input music (bottom row) to match the emotions of the speech (top row). Our algorithm inserts pauses in speech and music, and makes musical transitions that were not in the original music in order to meet these constraints.

Research Paper

PDF (3.1M)

Results

Musical scores generated by our system

Video

MP4 (31.9M)

Abstract

We present an interactive approach for decompositing bitmap drawings and studio photographs into opaque and semi-transparent vector layers. Semi-transparent layers are especially challenging to extract, since they require the inversion of the non-linear compositing equation. We make this problem tractable by exploiting the parametric nature of vector gradients, jointly separating and vectorising semi-transparent regions. Specifically, we constrain the foreground colours to vary according to linear or radial parametric gradients, restricting the number of unknowns and allowing our system to efficiently solve for an editable semi-transparent foreground. We propose a progressive workflow, where the user successively selects a semi-transparent or opaque region in the bitmap, which our algorithm separates as a foreground vector gradient and a background bitmap layer. The user can choose to decompose the background further or vectorise it as an opaque layer. The resulting layered vector representation allows a variety of edits, such as modifying the shape of highlights, adding texture to an object or changing its diffuse colour.

Our interactive vectorisation technique lets users vectorise an input bitmap (a) into a stack of opaque and semi-transparent vector layers composed of linear or radial colour gradients (b). Users can manipulate the resulting layers using standard tools to quickly produce new looks (c). We outline semi-transparent layers for visualisation; these edges are not part of our result. We rasterised figures to avoid problems with transparency in some PDF viewers. See supplemental material for vector graphics.

Research Paper

PDF (6.6M)

Video

MP4 (69.4M)

Abstract

We present a user-assisted video stabilization algorithm that is able to stabilize challenging videos when state-of-the-art automatic algorithms fail to generate a satisfactory result. Current methods do not give the user any control over the look of the final result. Users either have to accept the stabilized result as is, or discard it should the stabilization fail to generate a smooth output. Our system introduces two new modes of interaction that allow the user to improve the unsatisfactory stabilized video. First, we cluster tracks and visualize them on the warped video. The user ensures that appropriate tracks are selected by clicking on track clusters to include or exclude them. Second, the user can directly specify how regions in the output video should look by drawing quadrilaterals to select and deform parts of the frame. These user-provided deformations reduce undesirable distortions in the video. Our algorithm then computes a stabilized video using the user-selected tracks, while respecting the user-modified regions. The process of interactively removing user-identified artifacts can sometimes introduce new ones, though in most cases there is a net improvement. We demonstrate the effectiveness of our system with a variety of challenging hand held videos.

Automatic video stabilization using the state-of-the-art is unsatisfactory as shown in a) as the background and subjects are heavily skewed. We visualize clusters of tracks used for stabilization c) and the user removes tracks on dynamic objects d) using mouse clicks. Tracks that are not used for the final rewarp are drawn in grey. The green outline in e) and f) shows the original frame boundaries. The distortion of the frame in e) is removed by having the user draw a quadrilateral (white lines) and its desired transformation shown in f). The new track selection and user-drawn transformations are used to re-stabilize the video to obtain the final result as shown in b). Notice that the background is rectified and that the subjects are no longer distorted.

Research Paper

PDF (47.7M)

Video

MP4 (83.1M)

Abstract

News articles, reports, blog posts and academic papers often include graphical charts that serve to visually reinforce arguments presented in the text. To help readers better understand the relation between the text and the chart, we present a crowdsourcing pipeline to extract the references between them. Specifically, we give crowd workers paragraph-chart pairs and ask them to select text phrases as well as the corresponding visual marks in the chart. We then apply automated clustering and merging techniques to unify the references generated by multiple workers into a single set. Comparing the crowdsourced references to a set of gold standard references using a distance measure based on the F1 score, we find that the average distance between the raw set of references produced by a single worker and the gold standard is 0.54 (out of a max of 1.0). When we apply clustering and merging techniques the average distance between the unified set of references and the gold standard reduces to 0.39; an improvement of 27%. We conclude with an interactive document viewing application that uses the extracted references; readers can select phrases in the text and the system highlights the related marks in the chart.

Examples of references extracted by our system, as shown in our interactive document viewing application. The user can select text (yellow background) and the application highlights the corresponding visual marks in the chart (fully saturated bars). The application also places red underlines beneath related phrases.

Research Paper

PDF (16.6M) | Interactive Document Viewer

Video

MP4 (4.0M)

Abstract

We present MotionMontage, a system for recording multiple motion takes of a rigid virtual object and compositing them together into a montage. Our system incorporates a Kinect-based performance capture setup that allows animators to create 3D animations by tracking the motion of a rigid physical object and mapping it in realtime onto a virtual object. The animator then temporally annotates the best parts of each take. MotionMontage merges the annotated motions into a single composite montage using a combination of dynamic time warping and optimization of a Semi-Markov Conditional Random Field. Our system also supports the creation of layered animations in which multiple objects are moving at the same time. To aid the animator in coordinating the motions of the objects we provide spatial markers which indicate the positions of previously recorded objects at user-specified points in time. We perform a user study to evaluate the perceived quality of the montages created with our system and find that viewers (including both the original animators and new viewers) generally prefer the animation montage to any individual take.

A user working with the MotionMontage system to record multiple takes of a 3D animation.

Research Paper

PDF (5.3M)

Video

MP4 (83.4M)

Other Videos

Teaser Video MP4 (4.2M)

Motion Graphs Comparison MP4 (8.3M)

Abstract

3D modelers often wish to showcase their models for sharing or review purposes. This may consist of generating static viewpoints of the model or authoring animated fly-throughs. Manually creating such views is often tedious and few automatic methods are designed to interactively assist the modelers with the view authoring process. We present a view authoring assistance system that supports the creation of informative view points, view paths, and view surfaces, allowing modelers to author the interactive navigation experience of a model. The key concept of our implementation is to analyze the model's workflow history, to infer important regions of the model and representative viewpoints of those areas. An evaluation indicated that the viewpoints generated by our algorithm are comparable to those manually selected by the modeler. In addition, participants of a user study found our system easy to use and effective for authoring viewpoint summaries.

Our view authoring environment integrated in a 3D modeling software: (A) the main modeling window, (B) the authoring panel visualizing authored views in the spatial context, and (C) the navigation panel showing the authored views in the temporal sequence.

Research Paper

PDF (2.3M)

Video

MOV (47.9M)