Roll Call: Latest News on Capitol Hill, Congress, Politics and Elections
August 30, 2015

Buyer Beware: There Are Polls, and Then There Are Polls

Given the “success” (note sarcasm) of some polls in the Virginia gubernatorial race Tuesday, it might be worthwhile to note the very divergent surveys in the Texas gubernatorial race.

Public Policy Polling, which showed Terry McAuliffe with a 7-point lead among likely voters right before the election, found Republican Attorney General Greg Abbott leading Democratic state Sen. Wendy Davis by 15 points in its November 1-4 survey of 500 “Texas voters,” 50 percent to 35 percent. PPP, a Democratic firm based in North Carolina, used automated telephone interviews in conducting the survey.

But a University of Texas/Texas Tribune poll of 1,200 registered voters, conducted October 18-27, found the margin a much more narrow 6 points, 40 percent to 34 percent. The University of Texas/Texas Tribune poll is conducted by YouGov, which employs a controversial methodology. The survey, conducted over the Internet, relies on “a proprietary opt-in survey panel” that is not selected randomly.

Widely respected Langer Research Associates issued a “briefing paper” on opt-in online surveys in August of last year noting that the American Association for Public Opinion Research has been quite critical of such surveys — even arguing that reporting a margin of error for an opt-in sample is “misleading”— and that a number of major media organizations “have adopted policies ruling out the reporting of opt-in online surveys as news, given the unreliability of the method.”

Wisely, in my view, Roll Call does not report opt-in online surveys, but Roll Call’s parent company, The Economist, partners with YouGov to produce The Economist/YouGov Poll, which also uses the opt-in survey panel Internet methodology.

Comments (2)

  1. billychuck

    Nov. 8, 2013
    4:30 a.m.

    I’m not sure PPP is the best pollster to cite there as a comparison. (http://www.newrepublic.com/article/114682/ppp-polling-methodology-opaque-flawed). Not seen many complaints from Cohn, Silver et all about YouGov’s methodology, perhaps because it did fairly well compared to plenty of live phone polls (http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/), as did Ipsos, another online poll.

  2. Stephan Shakespeare

    Nov. 8, 2013
    4:52 p.m.

    do you actually understand what ‘margin of error’ means? the statistical formula is applied to randomly selected samples, but no polling methodology produces randomly selected samples (as even with your preferred method there is always opt-out, usually majority opt-out, so utterly non-random). The only true way to do it is to compare with past results. We’re pretty good. But that would require you doing some real math…

Sign In

Forgot password?

Or

Subscribe

Receive daily coverage of the people, politics and personality of Capitol Hill.

Subscription | Free Trial

Logging you in. One moment, please...