It’s sad! Political pollsters are giving marketing researchers a bad name. Umpteen pollsters are asking potential voters the same question but getting very different results. Who is right and who is wrong?
If these folks are so good at doing what they do, why is it that the results are so inconsistent. Were 4 different marketing research companies hired by General Motors to ask consumers the same questions but get very different results, General Motors would fire them all and hire a competent researcher and conduct their own surveys.
Getting consist results is not difficult. One only needs to select a sample of consumers or potential voters that constitutes a representative sample of the population of interest (whether it be likely automobile buyers or likely voters) and then to simply ask them unbiased questions.
In the humble opinion of this marketing researcher, it is appalling that we’re witnessing such diverse results–a diversity that is due largely to incompetence, greed, or political bias. It does not have to be that way.
As we are getting so close to the presidential election, it is time to mention margin-of-error once again. Just yesterday, I heard one of the radio personalities on Chicago 890 AM—WLS comment that many people didn’t understand the concept of margin-of-error. And that’s true.
Then he proceeded to explain what it meant. In doing so, though, he got it just half right. The radio personality gave a hypothetical example that 50% of voters were likely to vote for one of the candidates–with a margin-or-error of plus or minus 5 percentage points. He described that his statement meant that the “real percentage” was in the range of 45-55%–which is correct. But he failed to indicate how likely it was that is the “true score” fell in the 45% to 55% range. If nothing else was stated, the confidence limit (the degree of certainty the “true value” was in the range was 95 %.). Furthermore, because of the size of the range + or - 5%, it tells us that the sample size was considerably less than 1100. It is much more frequent that we see margins-of-error of + or - 3%, as the radio personality’s colleague had indicated. When the statement is made that 50% preferred one candidate–with a margin or error of 3% (+ or - ) 3%, this will always mean that one can be confident that 95% of the time when such a survey is undertaken the actual score will be within + or - 3% of the observed score (which in this case would have been 50%). This would also imply that the research had used a sample of approximately 1100 individuals–hopefully individual that were representative of the population under study–such as “likely voters.”
George Stungis died a couple months ago from complications of diabetes. Every time I see his name on my cell phone, I miss him.
George, a Ph.D. physicist, hired me years ago at Brown & Williamson Tobacco Company as a senior staff advisor. That opportunity George provided constituted my formal introduction to marketing research. George was a brilliant man. While at Brown and Williamson, I had many adventures with him solving unsolvable problems.
I recall when we were getting ready to launch Barclay: Stungis said “If it gets one share point, your career is guaranteed for the rest of your professional life.” And, shortly after was launched, it had achieved 1.1 share points—the most successful new product launch in the industry (to my knowledge) since Philip Morris launched Merit.
A couple months later after George and I had returned from corporate headquarters (BAT) in London, I was fired by the new senior marketing vice president. I don’t mind being fired (the V.P. didn’t say fired; he just said that they needed a quantitative type—not me). The fact is, though, that I am a quantitative type. But that’s neither here nor there. George was out of town when I was “fired.” When he returned, I fussed at him saying “George, if I was to be fired, you should have been the one to fire me.” But I didn’t hold a grudge. I still had to make a living to support my family. So I hired my old boss, George Stungis, to work with me doing competitive analyses for major corporations.
Subsequently, using a decision analytic approach similar to what we’d done with these competitive analyses, George and I created a mathematical model that we published in the Journal of Homeland Security entitled “A Terrorist Target Selection and Prioritization Model.” This model:http://www.homelandsecurity.org/newjournal/articles/stungis.html , which received some acclaim, allowed us to predict just what the bad guys were likely to do. Putting together a team of specialists, we utilized our model to assess a seemly obscure county in Florida—the results of which were also published in the Journal of Homeland Security–in an article entitled “A Prescription for Safeguarding Against Terrorist Attacks”. Here is that article http://www.homelandsecurity.org/journal/Default.aspx?oid=146&ocat=1.
George was also my Christian friend. He believed that old physicists become cosmologists—the study of everything. My wife thought I was crazy when I was reading the numerous books on cosmology that Stungis had recommended—books written by Ph.D. physicists, some of whom were Nobel Prize winners. He had planned for the two of us to write a scientific article to mathematically prove the existence of God. Neither he nor I needed any proof, but we thought it would be great fun to write it.
But we won’t be able to do so because George is already with the Lord. So the next time that I will see him face-to-face will be in the New Jerusalem—heaven on earth.
Let me tell you the rudiments of what Stungis postulated about the spirit—the human spirit that departs each of us when we die. To have any notion of what he was talking about, I had to learn a little bit about general relativity, quantum mechanics, string theory, and worm holes.
String theory proposes dimensions beyond the 4 we can easily conceptualize—height, width, depth, and time. Stungis envisioned that our spirit, when we die, passes through a worm hole to another dimension (heaven). For that to be possible, he postulated a very simple solution, namely that the spirit was an electromagnetic particle with no spin, no energy, and no mass and, thus, could pass through the worm hole to heaven.
I’m certain that for a lot of folks, it would be hard to believe that organizations hire researchers to conduct pure research, that is, research conducted to expand knowledge-not to address a specific issue. But I was hired by the tobacco industry to conduct pure research not applied research.
As a freshly minted Ph.D., I was truly in “hog heaven” getting to conduct many, many experimental studies. Experimental studies, unlike correlational ones, permit the researcher to identify causal relationships between variables.Since I had specialized in human factors psychology, I was especially interested in the effect of smoking condition (smoker, non-smoker, and smoker-deprived) on complex human performance. Though I invariable found no significant differences (p > .05) as a function of smoking condition, I still was able to publish articles in the scientific literature in spite of the generally accepted wisdom that “null” results aren’t publishable.
Everyone that has ever taken statistics 101 has heard the expression “Correlation does not imply causation.” Consider a matrix with two rows (smoker vs. non-smoker) and two columns (cancer vs. no cancer). If proportionally more smokers have cancer than non-smokers, this would mean that smoker and cancer are correlated. But it does not mean that cancer causes smoking. Being such an emotional charged issue, though, many would have a hard time believing that such evidence doesn’t support an explanation that smoking causes cancer. The only way that one could prove that smoking “caused” cancer would be via experimentation. However, an experimental evaluation would be unethical and immoral. I say all this but I certainly don’t promote smoking. Even during my tobacco industry days, I was prone to telling folks that it wasn’t smart to intake anything into their lungs except air.
Here is an example of another non-causal relationship for which it should be readily understandable that it doesn’t define a causal relationship. Consider a matrix with two rows (households with TVs vs. households without TVs) and two columns (cancer vs. non-cancer). One would find higher cancer rates among household that didn’t have TVs. But I suspect that few folks would believe that providing televisions to non-TV households would decrease the cancer rates. Experimental studies demonstrate cause and effect; correlational studies don’t!
At conferences over the years, I’ve encountered lots of in-house marketing research folks who were disappointed that few of their recommendations were put into action. I told them that my experience was quite the opposite–that most recommendations that I had made were acted upon. When they asked why, I simply told them that “I was so convinced that what I was recommending was the right thing to do that I spent the necessary time selling them on the correctness of the conclusions and recommendations.”
In retrospect, I believe that there were two main reasons that most of the research that I conducted, as an in-house research, was acted upon:
I saw myself as a team member, not simply a supplier to in-house clients–as some folks do. So it was always “us” not “we” versus them. In my opinion, the in-house supplier-client conception that many folks seem to promote is far off the mark; and
I was proactive, not reactive. For the vast majority of the research that I conducted as an in-house research type, I saw the need and implemented the research with out being asked.
It probably boils down to the fact that there are 3 types of people (and organizations):
In the OJ Simpson Trial, the DNA evidence was so incredibly strong that some pundits suggested that the qustion “Is OJ guilty?” constituted an intelligence test. Though not politically correct, the “yes” or “no” clearly indicated one’s reasoning ability since the likelihood that the DNA came from anyone other than OJ was less than 1 in a billion.
I’ve long used an equally simple way to gauge the IQ of folks who consider themselves to be marketing researchers. Let me explain!
Suppose you have created two versions of an ad designed to increase your brand’s unaided awareness. In a marketing research study, unaided awareness for Ad “A” is found to be 21% and 18% for Ad “B.” But the difference is not statistically significant (p > .05). Assuming that there are no other differences between the two ads and that you must make the decision right now, the question that you should ask the researcher is “Which ad would you recommend be run in the media?”If the so-called “marketing researcher” responds with something like “It doesn’t matter or flip a coin,” I’d humbly suggest that you fire him or her and that you run Ad “A.” That is, go with the higher number. The best information that you have is that Ad “A” performed better on unaided awareness than did Ad “B.” The fact that there was no significance difference only means that the observed 3-point difference could have occurred more than 5 times out of 100–not that no difference was present.
I’ve posed this problem to numerous folks who consider themselves “marketing researchers” including some Ph.D. types. Sadly many of these supposedly learned folks said, “Flip a coin; it makes no difference.” Unfortunately, after statistics 101, they never got their brains out of low gear; clearly they flunked this simple intelligence test.
Suppose that, in a Fox News Poll, it is reported that likely voters prefer Obama over McCain by 2% but that this difference in preference is within the margin of error. This would mean two things:
the difference in preference is not significant at the at the .05 level of statistical significance; and
That a difference in preference of the reported magnitude (2%) would be expected to occur by chance 1 time out of 20 (that is, 5% of the time that such a survey with whatever sample size they used was conducted).
Had I conducted that survey, I would have reported the results to Fox News as:
In a poll of likely voters, 48% preferred Obama and 46% preferred McCain a difference that was not statistically significant (P < .05).
So the way that I portrayed Fox News as having reported the results would have been true and accurate.If Fox had happen to mention that the margin of error was +/- 3%, that statement would have meant that the sample size (the number of people surveyed) was approximately 1100. The magnitude of the margin of error is always a reflection of sample size. A sample size smaller than 1100 would have produced a margin of error larger than 3%. Likewise, a larger sample size than 1100 would have resulted in a margin of error less than 3%.
Welcome world! I’m regularly going to be blogging about marketing research:
How to use it;
How not to abuse it;
How to get the most out of it;
I will make many of the technical aspects of marketing research understandable; and
I will encourage you to ask questions
The first topic that I plan to blog about will be “margin of error,” a term that we often hear on news programs and that we will be hearing more and more as the Presidential Election draws nearer.
It’s sad! Political pollsters are giving marketing researchers a bad name. Umpteen pollsters are asking potential voters the same question but getting very different results. Who is right and who is wrong?
