Tuesday, November 06, 2012

Poll Watching

In the aftermath of the 2000 election, I did a detailed analysis of the Florida recount that was in process and determined that there was an exceedingly high probability that George W. Bush would be declared the victor if Al Gore in fact was allowed a recount in every county he requested.
I sent my analysis to several newspapers and talk show hosts around the nation. It turns out that about a dozen, or so, other people did the same thing and the idea that we (George W. Bush supporters) needn't fear a transparent recount. In the end, my projection absolutely nailed the end result.
If it had been 4 years later, I probably would have started a blog and would be living a different life right now. Well, maybe.

Why do I mention this? I'm a bit of a stats guy with pretty extensive industrial experience, so I do pay attention to things like polling data, and the modeling method I used in 2000 was essentially the same as what Nate Silver uses at FiveThirtyEight . His expected value model methodology for the electoral college is rock solid.

But... and you know there was one... the devil is in the details. The thing I had in 2000, was that there was quite a bit of data showing that manual recount corrections, under/overvotes, "hanging chads," etc. all followed the original distribution, so it was very straightforward to calculate the probability that Al Gore would pick up enough votes to overtake the certified result with Bush in the lead.
As of this writing, the Real Clear Politics national average has the president up by 0.7% and the average of the states would give President Obama 303 electoral votes (with no toss-ups). Nate Silver has the president at 313 electoral votes with >90% chance of winning (this is approaching statistical certainty).

Now with any Monte Carlo-based model, the key is in how representative the data are to the underlying populations. If the state polls are correct, then Mitt Romney has very little chance of winning. If they are not correct... Nate Silver may go the way of John Zogby.

There is and have been a lot of concerns discussed freely about the accuracy of various state polls, most of which are centered on the turn-out assumption by party that I won't rehash here. I do want to touch on a few things that give me pause in the accuracy of the state polls.
Caution 1. Regarding the sample adjustments that many polls make. It is a common technique to apply corrective factors to the samples in establishing a likely voter turn-out model (here is an example of an analysis done this way regarding Minnesota's Marriage Protection Amendment back in September). The problem with such an approach is that any adjustments applied based on the poll internals, no longer carry the confidence interval of the top line poll question; the error is increased and is dependent upon the sample sizes of the internal categories used for the adjustments. The accuracy is improved, but at the expense of precision, and I have never seen this explicated consistently in published results, particularly at the state level.

Caution 2. There has been a growing problem in polling regarding response rate of telephone polls since the advent of caller ID. I consider this a potential hidden wild card to create the next Literary Digest event,  i.e., where many of these polls in swing states would in effect be unscientific.

Caution 3. While probabilistic estimates like FiveThirtyEight's for an Obama victory have risen from the mid 70% range to over 90%, the Intrade odds have only risen from the mid 60% range to just under 70%. This 1:3 movement indicates a damper of doubt for the recent polls.

Caution 4. As a corrollary to the voter turn-out rates (which is still Caution 0), my old blog buddy, David Hartline, has noted in the state of Ohio that while the focus has been on the populations in 2008 and 2010, the polling has effectively ignored the demographic shifts that have occurred as a result of the recession and weak Obama recovery.
So what does this mean? I think any Obama victory will be narrower than what's being projected by RCP and FiveThirtyEight, but we'll probably know pretty early (New Hampshire, Virginia, Pennsylvania) whether the state polls are flawed seriously and there was good reason for all that optimism from the new media right the last couple days. It also means that things will be close and all those extra people doing that other "poll watching" will come into play, ready to delegitimize the result regardless who wins. And it also means that there will be thuggery and shenanigans, which, being in a shenanigan-friendly state with the control of both the state house and senate in play, concerns me greatly. It doesn't matter if they cheat if it's not close, but it's going to be close.

Additional Information

On Polling Models, Skewed & Unskewed
Is Nate Silver's Value at Risk?


The thuggery has begun.

Update 2

Well, it's pretty clear the state polls were correct.

No comments:

Post a Comment


Related Posts Plugin for WordPress, Blogger...

Because Life is Life
and not just on election day