Marketing Research Blog

It’s sad! Political pollsters are giving marketing researchers a bad name. Umpteen pollsters are asking potential voters the same question but getting very different results. Who is right and who is wrong?

If these folks are so good at doing what they do, why is it that the results are so inconsistent. Were 4 different marketing research companies hired by General Motors to ask consumers the same questions but get very different results, General Motors would fire them all and hire a competent researcher and conduct their own surveys.

Getting consist results is not difficult. One only needs to select a sample of consumers or potential voters that constitutes a representative sample of the population of interest (whether it be likely automobile buyers or likely voters) and then to simply ask them unbiased questions.

In the humble opinion of this marketing researcher, it is appalling that we’re witnessing such diverse results–a diversity that is due largely to incompetence, greed, or political bias. It does not have to be that way.

As we are getting so close to the presidential election, it is time to mention margin-of-error once again. Just yesterday, I heard one of the radio personalities on Chicago 890 AM—WLS comment that many people didn’t understand the concept of margin-of-error. And that’s true.

Then he proceeded to explain what it meant. In doing so, though, he got it just half right. The radio personality gave a hypothetical example that 50% of voters were likely to vote for one of the candidates–with a margin-or-error of plus or minus 5 percentage points. He described that his statement meant that the “real percentage” was in the range of 45-55%–which is correct. But he failed to indicate how likely it was that is the “true score” fell in the 45% to 55% range. If nothing else was stated, the confidence limit (the degree of certainty the “true value” was in the range was 95 %.).  Furthermore, because of the size of the range + or - 5%, it tells us that the sample size was considerably less than 1100.  It is much more frequent that we see margins-of-error of + or - 3%, as the radio personality’s colleague had indicated. When the statement is made that 50% preferred one candidate–with a margin or error of 3% (+ or - ) 3%, this will always mean that one can be confident that 95% of the time when such a survey is undertaken the actual score will be within + or - 3% of the observed score (which in this case would have been 50%). This would also imply that the research had used a sample of approximately 1100 individuals–hopefully individual that were representative of the population under study–such as “likely voters.”

Suppose that, in a Fox News Poll, it is reported that likely voters prefer Obama over McCain by 2% but that this difference in preference is within the margin of error. This would mean two things:

1. the difference in preference is not significant at the at the .05 level of statistical significance; and
2. That a difference in preference of the reported magnitude (2%) would be expected to occur by chance 1 time out of 20 (that is, 5% of the time that such a survey with whatever sample size they used was conducted).

Had I conducted that survey, I would have reported the results to Fox News as:

• In a poll of likely voters, 48% preferred Obama and 46% preferred McCain a difference that was not statistically significant (P < .05).

So the way that I portrayed Fox News as having reported the results would have been true and accurate.If Fox had happen to mention that the margin of error was +/- 3%, that statement would have meant that the sample size (the number of people surveyed) was approximately 1100. The magnitude of the margin of error is always a reflection of sample size. A sample size smaller than 1100 would have produced a margin of error larger than 3%. Likewise, a larger sample size than 1100 would have resulted in a margin of error less than 3%.