New NBC/SM poll has issues, like Clinton up by 5%

I've been looking at the trending of polling over the last two years using RealClearPolitics average of surveys among various pollsters as my base of polls to include.  I've been screening out Registered Voter polls in favor of likely voter polls, as likely voters are more likely to vote, and therefore their opinion is more likely to matter come voting day.  Likely voter models do contain a risk in that they can fine tune results too finely and skew results.  But for 30 years they have generally proven more predictive than Registered Voter polls.

In looking at those polls there is a clear trend towards Trump in September.  But my most recent Excel refresh suddenly shifted back towards Clinton - even my weighted average model, which sums voters across polls so polls with larger populations have a higher impact on my results than smaller polls. Granted, different pollsters could be calling the same voter and I could be double counting, but the risk of that is miniscule and even if it were the case, any attempt at averaging the polls would be impacted by the issue. So I'm willing to overlook the problem.

What I was not able to overlook was a reverse shift.  So I dug into the polls and the most recent NBC/SurveyMonkey poll caught my attention immediately.  It polls 13,320 likely voters and has a margin of error of a mere 1.2%.  The poll, stands athwart all other recent polls has Hillary Clinton ahead by 5%. Wow! What could account for such a dramatic shift?  A major Trump gaffe?  His son's Skittles comment?  People coming to their senses?

Well, first we have to take a look at the poll's particulars - crosstabs, methodology, etc.  My first concern is that they have scrubbed out very few Registered voters, claiming their Registered Voters polling is pretty close to likely voters already;

The NBC News|SurveyMonkey Weekly Election Tracking Poll screens for adults who say they are registered to vote, and we weight the data to reflect the demographic composition of registered voters (by age, race, sex, education and region) using the Census Bureau's Current Population Survey. Demographically, our results closely mimic the population of registered voters.

In addition, we also believe that the method used by SurveyMonkey to recruit respondents into the weekly tracking poll selects the most likely voters from among the population of people taking SurveyMonkey surveys. Our respondents are selected from the nearly 3 million people who take surveys on the SurveyMonkey platform each day. To do so, for a random sample of those taking a survey, SurveyMonkey displays a map of past election results colored in gradations of red, blue and purple and they ask those individuals to "help us predict the 2016 elections." Because individuals choose whether or not they want to help predict the election, those individuals choosing to participate are arguably more likely to be politically interested and likely to vote than respondents who see the same invitation and decline to participate.

That's a pretty strong assumption. People take polls for a number of reasons - financial gain, the opportunity to influence public opinion or to pretend to be an adult all are possible. Yes, there is some validity in their point but it is not necessarily an overarching correlation to likely voter-hood. Which brings me to another point.

After reviewing the previous research and available evidence in our own data, NBC News and SurveyMonkey have concluded that the best approach for our tracking survey data is a "likely voter model" that makes only modest adjustments to our self-reported "registered voter" results.

The survey is an online survey. Identifying who is completing the survey is impossible, it's blind. In addition there is no indication of whether all self-reported volunteers were included or how many or whom was excluded. The crosstabs were not provided in a robust way. I'm not convinced that this survey would pass scientific rigor despite it's large sample size. There have been 5 recent Likely Voter polls including this one. 3 have Trump ahead. 1 is a tie. This poll is clearly the outlier among the polls, not only showing Clinton leading but by a wide margin that does not approach any of the other polls.

In fact since June 1st, only 14 polls have had Hillary at 50% or more support and 4 of them have been by NBC/SM. If we look at only like voter models the total of Clinton >50% drops to 5 surveys; PPP (a Democratic pollster) in July, Bloomberg and Quinnipiac in August when Clinton did seem to be surging. In September this NBC/SM poll and a Washington Post/ABC poll from September 8th. If more polls come out in the next few days, I'd be more inclined to believe this one. But for now, not so much.