Test Audiences Can Make or Break New TV Series

Pressure to find TV hits is intensifying in an industry racked by fragmented viewership and competition from digital media; ‘absolutely gut-wrenching’

ENLARGE

NBC’s ‘The Good Place,’ about a mean woman who tries to change her ways, stars Kristen Bell and Ted Danson. Photo: Justin Lubin/NBC

Joe Flint

The Wall Street Journal

CANCEL

Biography Joe Flint
@JBFlint
[email protected]

May 15, 2016 5:30 a.m. ET

49 COMMENTS

Gerald Cuesta stared at the chart in disbelief, as the lines kept going lower and lower.

The television writer was observing the reaction of a test audience to “Babylon Fields,” a show he was trying to get on NBC’s schedule for the 2014-2015 season.

NBC to Unveil Its Lineup for New Television Season

The quirky drama about the dead coming back to life got off to a decent start in the focus group. But a scene featuring cannibalism sent the score “down to grandma’s basement,” Mr. Cuesta said, adding, “I knew right then and there we’re not getting on the air.”

He was right. Despite having a fan of “Babylon Fields” in NBC Entertainment President Jennifer Salke, Mr. Cuesta’s dead wouldn't rise after the test.

Mr. Cuesta holds no ill will toward his mysterious judges. “It’s simply part of the process. If you’re going to create TV, the chances that you will go through this are great.”

More in Media and Advertising

Get the latest news and analysis on media and advertising via a daily newsletter, at WSJ.com/CMO.
Listen to the WSJ Media Mix podcast, featuring discussions on need-to-know topics with the people who matter in media and marketing, available through your podcast app, or at WSJ.com/Podcasts

Over the past month, lots of writers and producers have come to know Mr. Cuesta’s pain. This is the time of year when the broadcast TV networks are busy figuring out their prime-time lineups for the next season. They’ll show off their schedules to advertisers in New York this week at glitzy presentations and parties.

Among the shows that have already made the cut are a television version of the hit movie “Lethal Weapon” on Fox, the comedy “The Good Place” on NBC starring Kristen Bell and Ted Danson—about a mean woman who tries to change her ways—and the drama “Designated Survivor” on ABC starring Keifer Sutherland as a cabinet member who gets thrust into the presidency after an attack on Washington.

The testing of new shows, known as pilots, is a crucial part of the process of deciding which ones to “pick up” and which to toss. The pressure to find new hits is intensifying in an industry racked by fragmented viewership and competition from digital media.

The firm that has been playing judge and jury to much of the television and film industry for more than 50 years is Screen Engine/ASI. The company tests some 250 pilots a year; programs that got their thumbs up after being tested there include NBC’s “ER,” ABC’s “Modern Family,” and FX’s “American Horror Story.”

The most traditional pilot test is to gather roughly 50 people—half men and half women—in a screening room. Viewers are recruited based on age and sex and race, as well as a network’s core demographic. “It’s like putting on a wedding each time you do one,” said Kevin Goetz, chief executive of Screen Engine/ASI.

ENLARGE

Test audiences use hand-held devices to grade shows. ‘Seinfeld’ went through the process and famously tested poorly. Photo: Castle Rock Entertainment/Everett Collection

The test audience watches a show and grades it with a hand-held device that has a knob the viewer moves to the right or left based on whether they like or dislike what they are seeing.

Meanwhile, some new players in media are trying to reinvent audience testing to improve on perceived drawbacks of the traditional model.

Amazon.com AMZN 0.10 % posts pilots of new shows for its streaming service on its website, where a large number of users can rate them before the company makes its decisions. New media specialists BuzzFeed and Vox, which both received investments from Comcast Corp. CMCSA 0.26 % , see the potential to use online videos as a testing ground to see what might develop into TV shows. (Streaming giant Netflix Inc. NFLX 1.41 % says it doesn’t test shows.)

For its part, ASI is now testing whether shows are “bingeable” in an era when watching multiple episodes of serialized shows in a single sitting is becoming more common. In one test for a cable show, sample audiences watched 10 episodes of a show over four days, with intense focus groups along the way to figure out “at what point in the season are you in or out,” Mr. Goetz said.

Many in the industry loathe the idea of random people playing tastemaker and telling them a show that they’ve spent millions of dollars developing stinks.

ENLARGE

Among the shows that have already made the cut is a television version of the hit movie ‘Lethal Weapon’ on Fox. Photo: Richard Foreman/FOX

Last month, the Fox comedy “The Grinder” about a TV lawyer played by Rob Lowe who quits showbiz to join his family’s law practice, poked fun at pilot testing. In an episode titled “A System on Trial,” Mr. Lowe’s character, Mitch Grinder, takes a colleague with him to a shopping mall to find people for a focus group. He says he wants people with “no distractions” and “no thoughts” who are “blank slates” and then “we blindly trust those people and allow them to steer us.”

Former NBC Entertainment President Warren Littlefield who is now an executive producer of the FX cult hit “Fargo,” said testing is “absolutely gut-wrenching.”

Most shows, he added, don’t test well, and that is especially true for anything a little different. Perhaps the most famous show to test poorly, he recalled, was the NBC hit “Seinfeld,” whose characters weren’t seen as likable enough.

Mr. Littlefield believes testing for mass-market appeal is less relevant because there are so many shows on cable TV that offer edgy plotlines and draw small but loyal audiences.

“The research was to sand down the edges that possibly offended or pushed away a larger audience. Today we seek those edges and embrace them,” he said.

ENLARGE

’Homeland’ also went through the test audience process. Photo: Showtime Networks Inc./Everett Collection

HBO eschews testing altogether. “Our bet is on the creative vision, and you can never measure that by just looking at a pilot without any context,” said Mike Lombardo, president of programming for HBO.

Still, testing has plenty of backers. “I don’t believe we’re that smart and that we know what the rest of the country wants to see,” said Preston Beckman, a former top programming executive at Fox and NBC.

Showtime Chief Executive David Nevins said he uses testing to see how specific stories and characters are resonating with viewers. Testing can tell you “if they’re confused, into it, or bored,” which helps with the editing process, he said.

For writers watching audiences dissect their shows, emotions can get out of control.

What’s News

Facebook to Sell Video Ads on Behalf of Other Firms May 16, 2016
Fox Turns to Familiar Brands for New TV Lineup May 16, 2016
NBC to Unveil 2016-17 Television Schedule May 15, 2016
J. Walter Thompson CEO Forms Diversity Committee May 13, 2016
Playboy Unveils ‘Safe For Work’ Web Video Slate May 13, 2016
Vine Stars Are Leaving for Facebook and Other Platforms May 13, 2016

“I’m part therapist in there,” said Twentieth Century Fox Television President Jonnie Davis, describing what it is like to watch a writer observe a test. “Have I ever had to jump in front of the glass to stop a [show] creator from ‘going Frankenstein’? Yes.”

Write to Joe Flint at [email protected]

Save Article

Sign In to Save

Subscribe to WSJ
Text Size

Regular

Medium

Large
Google+

Print

49 comments

Newest | Oldest

GARY LUDWICK subscriber

As a Creative Director at an advertising agency, we use focus groups more to test new product concepts as opposed to the actual tv or radio commercial ( although we do that from time to time)

This line: "Have I ever had to jump in front of the glass to stop a [show] creator from ‘going Frankenstein" really struck a chord with me, though.

Had a client with a new idea for solar water heating literally leave the viewing booth with one way glass and run into the presentation room because the focus group, "they just don't get it!".

He just had to explain it all himself (in put-you-to-sleep boring detail).

The product failed.

But I agree that when it comes to creative endeavors like Seinfeld or Penny Dreadful, it's a lousy way to measure that intangible, excellence.

MARK NAHER subscriber

It has been my experience that the average American is not qualified to critique something artistic or creative. The uninitiated will turn anything 'edgy' into stale and boring. "Edgy" scares the average person too much. They need something safe.

Shaheen Bagheri

Test audiences are the worst possible solution for pushing the envelope in creative T.V. HBO's Mike Lombardo said it best. As Steve Jobs once said, “It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them.” This can be debated on the product level, but on the artistic level there is no debating. Any time art is altered based on average opinions, the integrity will always take a turn for the worse. This is why HBO shows are so successful, because they are "embracing the edges." The average shows on cable are unbearable to watch because they undermine the intelligence of the viewers, but there are enough people with average taste (no offense) who will flock like moths towards the T.V. at any modern Baywatch-esque show. Leave it in the hands of the test audience and you will suck the creativity out of the T.V.

JOHN CASSIDY subscriber

Hope this one makes it past the censors.

Best Seinfeld lines:

Please Jerry, I need that chicken. -Kramer

I’m not a lesbian. I hate men, but I’m not a lesbian.-Elaine

There was shrinkage!.-George

Let's watch them slice this fat ba*ta*d up. –Jerry

No soup for you!

Phillip Hymel subscriber

@JOHN CASSIDY

Yada, yada, yada...

Justin Murray subscriber

I deal with statistics on a regular basis and this kind of testing is a bit troubling. The same size is pretty small, the selection criteria is problematic and we're dealing with differing conditions from normal behavior.

First, the sampling issue. Samples need to be stratified and randomly selected in sufficient numbers. There usually need to be around 1,000 or so people in a sample to avoid most errors. Additionally, there needs to be stratification. Saying "Half women, half men" and then tossing in different age groups fails this on numerous levels. There is no representation at any level here since they'll end up sampling maybe 2 or 3 people in a strata.

Second, people behave differently in groups than alone. Television is not watched in groups of 50. Audience mechanics have a strong impact on how we react. Something we may find humorous on our own will not be if the rest of the room doesn't laugh (like Seinfeld above). The opposite is also true. This gives a bad result.

Brian Charles subscriber

@Justin Murray Whoa Justin. This is called QUALITATIVE research and it is done because they need a conversation with the consumer to understand WHY they don't like something. If they don't like something but it can be easily fixed, no problem.

In my experience this kind of research can be very effective when people trust the process. Generally, if you do it over time, you build up a good sense of what works and what doesn't. It's only when people talk themselves into thinking they "have something new and breakthrough that defies convention" that they get into trouble.

@Brian Charles @Justin Murray

If it worked out so well, then 80% of shows wouldn't fail in their first season. They've clearly haven't figured out what works. They're attempting to determine whether or not to greenlight a show on what is essentially the opinion of three people off the street.

@Justin Murray @Brian Charles Justin you have to realize what happens when ego gets involved or the numerous ways a show can go wrong. Sometimes these people are on a power trip about deciding what people watch and they greenlight something based on opinion.

Sometimes they do it because they want to keep a particular producer happy because while this show may be a dud they want their next hit.

Sometimes a show gets mismarketed. Journeyman was a terrific show but it wasn't pure Sci-Fi and it wasn't a traditional drama and the studio couldn't communicate to the masses what it was to drive sufficient trial.

Sometimes the time slot is wrong, sometimes a show starts off well and then goes astray.

Art is messy, marketing is messy. Good research ups the odds but no system is perfect.

Andrew Ettinger subscriber

The first season renewal rate for all network TV shows is far less than 20%, a pretty dismal number. That makes me wonder why any TV executive thinks that this type of testing is at all effective. Imagine if we used a 20% success rate with your financial investments. Oh wait, that's most mutual funds too I suppose.

Fred Jolowski subscriber

People watch the Networks?!

Milton Scritsmier subscriber

Nothing but retreads. The Kristen Bell and Ted Danson comedy was done by Christina Applegate in "Samantha Who?" a few years ago. Even Danson starred as an unrepentant mean character in "Becker".

Keifer Sutherland as a cabinet member who becomes president when everybody else is killed in a surprise attack? How about Mary McDonnell, the education minister in "Battlestar Galactica" who becomes president after a surprise attack kills all the other cabinet members?

@Milton Scritsmier

"How about Mary McDonnell, the education minister in "Battlestar Galactica" who becomes president after a surprise attack kills all the other cabinet members?"

Or King Ralph, where John Goodman becomes King of England after faulty wiring kills all the other royals?

Jack Hoff subscriber

Lethal Weapon - wow - what an original concept. A buddy buddy cop premise and it might be with two cops who are different colors and personalities. That hasn't been done before... more than 500 times.

JOHN HAWKINS subscriber

@Jack Hoff

Yes, but this time one of the partners will no doubt tackle his transgender identity in an intolerant but changing world...reaching out to the community for compassion while still bringing the vengeance of the law to bear against universally hated drug dealers, wall street bankers, and conservatives.

I just always assumed they picked shows based on how much left leaning politics they could insert into the content.

another theory was selection based on how easily they could portray the middle of the country in a backward or negative fashion according to their vision of them....so they could then criticize the image they themselves had created.

Josh Shapiro subscriber

It sounds to me that the issue is not testing, but using an untargeted audience for "general appeal". Why not identify a niche target of people who the show is being tailored to, and then use them to get the reaction?

Gary Kennedy subscriber

"HBO eschews testing altogether." And IMHO they have the best record of all. "The Wire", "The Sopranos", "Rome", "Treme", "Six Feet Under", "Oz", etc. A few others are in that league, like FX's "Fargo", but not many.

Nikhil Khanna subscriber

@Gary Kennedy Don't forget GoT!

Alonzo Quijana subscriber

"...“The Good Place” on NBC starring Kristen Bell and Ted Danson—about a mean woman who tries to change her ways...."

Yeah. Two old retreads. And a crusty -- mean -- character trying to cope with some fish-out-of-water situation, I presume, while spewing sarcastic one-liners. Probably has to run a kindergarten, or some life-saving charity. You know it is going to be awful.

Stephen Phillips subscriber

If Ted Danson is they best they have to offer, no wonder TV is struggling...

Ed Dyjak subscriber

@Alonzo Quijana

I'd rather watch reruns of 'My Name Is Earl'.

I like different, quirky, provocative shows that probably do not test well in focus groups or on meters. No wonder broadcast TV does not interest me. It's programmed to some boring bland, homogenous middle.

Bryan Smith subscriber

That's why NBC was so smart to drop "Star Trek" because it was not getting the audience that was needed, for the advertising numbers it wanted.

Or why FOX network made the same mistake with "FireFly"

Yep, no audiences there either...

Thank the gods for cable, satellite and now the internet...

Kirth Gersen subscriber

"focus group" in any product development, whether it's cars or TV shows, virtually guarantees lowest common denominator, or superfluous features.

That's largely why you'll never see a "Game of Thrones" or "Breaking Bad" on the 4 national "mainstream" networks.

Or why cars will have 20 cup holders, "Siri integration", but are virtually unrepairable after a few years.

Frank Westmoreland subscriber

The problem is the series model, with the same actors and story line year after year. This forces even the most talented writers to insert a lot of what I call "soap opera filler" to keep the series going. Thus there is a lot of episodes where the characters just stand around and talk to one another, not advancing the story.

The model should be like the HBO series "True Detective," where each year you have about 10 episodes, like an elongated movie. Then each new year, you have a new cast with a different story. I watched two seasons of "True Detective," and there appeared to be far less soap opera filler than a multi-year series with the same cast and story line.

This model would also enable a lot more actors---especially white females and racial minorities---to be in a tv series. So many white women and racial minorities are upset that they don't get a chance at stardom. It would also mean less soap opera filler, which I hope would mean more interesting entertainment for the viewers.

Richard Lawson subscriber

Focus groups are killing creativity. To avoid being killed by the ratings, writers and producers keep producing the same boring stories with a slight twist. I can't watch any show without wondering if I have already seen it before. The great shows and producers of the past would not have a chance of success today.

The TV networks are dinosaurs. A better system is for producers to find their own advertisers or investors and post the show online for viewers to choose. Oh wait, is that YouTube?

Carrie Gorringe subscriber

Notice the continued dependence upon "proven" franchises and names? Frankly, I don't need another show featuring The Same Old Thing..

Howard Diamond subscriber

Every television show is derivative. Just variations of the three main sources of American culture: Shakespeare, the Bible, and the Three Stooges.

Leonard Rothbart subscriber

@Howard Diamond Combine Shakespeare, the Bible, & the Three Stooges: now *there's* a concept! :)

Right, but some show runners have the ability to use the motifs in ways that are unique; most don't.

@Howard Diamond Yes, its pretty close to that. Another way I've heard it is that there are only "X" number of plots, and Shakespeare created all of them. Pick you own "X", here is an argument for "7":

https://wizzley.com/there-are-only-seven-stories-in-the-world/

But not everybody has to be an innovator. A while back, a NY Times poll voted J.S. Bach the greatest composer of all time. Yet, "Bach was not a great innovator, and he did not strive to be one. ... Bach thought nothing of borrowing from himself or other composers ... He is now frequently spoken of as the high point not only of the Baroque style period, but of several centuries of the Western musical tradition ... "

https://www.quora.com/What-is-it-about-Bachs-music-that-makes-it-transcendent

Dan Morris subscriber

99% of TV is boring. I cut the cable over 11 years ago and have never missed it for a minute. I probably watch 2 hours a year of network tv and stream anything that I have an interest in, which isn't much in these days where government workers seem to end up being the heroes.

Kevin Snyder subscriber

"Perhaps the most famous show to test poorly, he recalled, was the NBC hit “Seinfeld,” whose characters weren’t seen as likable enough."

That means that the testing was correct. Seinfeld was about two things: Nothing, and bad behavior. "You are a bad man, Jerry!"

Ben Milano subscriber

I bet if testing was as prominent in the 50' and 60'see as it is today, a large majority of great and classic shows, would have never seen the light of day.

I agree with HBO & NETFLIX. Let the writers, directors, actors, etc. do what they do, put in on the air and let the viewers make the decision.

Andy Kushner subscriber

@Ben Milano

In principle I agree, but since broadcast networks prime time viewership isn't what it used to be, the pockets aren't so deep to support the production costs. I don't know the financial arrangements, but if a production company thinks their show is great, perhaps they could charge the network a reduced fee during a trial period in return for proper advertising support from the network.

JAMES VUGTEVEEN subscriber

I have given up on live TV. Same actors recycled into different story lines that usually have poor writing or poor acting. An hour with 40 minutes of story and 20 minutes of commercials that have become distasteful. Hallmark Channel is a prime example. HGTV has become very predictable with it's outcomes. How many times can you listen to "phony excitement" at the end. Spare me!!

I now watch Netflix and Amazon, with programs that I choose. Most have stood the test of time; and I guess I do my own testing. Usually commercial free; the British do it best, with Doc Martin, Miss Fisher, Agatha Christie, and great movies.

@JAMES VUGTEVEEN I agree. Ted Danson is already in a new "boring" show? He just finished the cancelled CSI. How many shows has he been on, anyway? I get confused about who he is supposed to be now.

David Ledet

If all you did was watch common television, you'd believe that 1/2 the population were gay.

Wm Ort subscriber

@David Ledet

... and abortions don't actually happen ...

Liberal Media Types seem deathly afraid to show any view of pregnancy abortion ... I wonder why this????

@David Ledet Mr. Ledet, Especially HGTV.

I actually a same race couple on a commercial the other day.....

"you'd believe that 1/2 the population were gay"

some day someone will ask how a group that the CDC says makes up less that 2.4% of the population has achieved so much political power...

holmen lloydk subscriber

...and every brain surgeon or top judge was Afro-American

@Roy Berger @Dave Lind

"Your reaction is typical of someone trying to suppress subconscious urges"

Bigot, Racist, Homophone, Xenophobe.....the 21st century equivalent of McCarthy calling someone a communist in the 1950s except that now those terms are used frequently and with great zeal by the left.

Once uttered, those terms are supposed to make all conservatives immediately abandon their positions and yield to the "morals" of the more "enlightened" point of view presented.....

Yep....people are finally catching on.