Millennium Marketing Research®

Tom Schori DBA Millennium Marketing Research®, 808 Ironwood, Normal IL 61761, 309-532-8466

Starting Page    Millennium Marketing Research?    Types of Research    Principals    Articles   Quote

When it doubt, go with the bigger number.

By Thomas R. Schori, Ph.D., and Michael L. Garee, Principals,  Millennium Marketing Research, 808 E. Ironwood, Normal, IL 61761-5239. 

Over the years, we’ve seen reasonably intelligent businesspeople regularly make some rather ill-advised marketing decisions, as a result of their misunderstanding of simple statistics. Unfortunately, it’s not just those whose only exposure to statistics was in a college Economics 101 class or an industry seminar on beginning marketing research who make such decisions. Otherwise intelligent people, with varying degrees of education and expertise, also make these same, faulty decisions. Let us illustrate.

Suppose a marketer has developed two different print advertisements for a product. Since only one ad will ultimately be run, the marketer wisely decides to test the two ads among consumers, and then pick the ad which "performs" best in the test.

While there are a number of ways an ad test can be conducted, our marketer decides to do a "mall-intercept" test. Consumers, representative of the target market, are shown a collection of ads for numerous products within the same product category, including of course the marketer’s product. One-half of the consumers are shown ad "A," and the other one-half, ad "B." After being exposed to the ads, consumers are then asked a series of questions designed to determine which ads they best recall.

In the best case scenario, the results may leave no room for ambiguity. For example, if ad "A" registered a recall score of, say, 80%, and ad "B" a recall score of, say, 40%, then obviously, the choice of which of the two ads to run would be crystal clear. Problems arise, however, when the results are quite so neat.

Let’s say, for example, that ad "A" had a recall score of 82%, while ad "B" had a recall score of 78%, and the difference between the two scores is shown to be not statistically significant. What to do? Oftentimes, otherwise bright professionals will conclude things such as the following:

• one ad is just as good as the other;
• the observed differences are meaningless;
• it doesn’t make any difference which ad is run;
• flip a coin to decide which ad to run.

While such conclusions may seem reasonable, actually, they aren’t appropriate here. That the difference was not statistically significant really only means that differences between the two recall scores could have occurred by chance more than 5% of the time, i.e., more than 5 times in 100 such tests (assuming the criterion for statistical significance was set at the .05 level).

So, what would be a more practical conclusion to draw in this situation? Actually, it’s quite simple. Determine if there are any other criteria that should be considered before deciding upon which ad to run. For example. . .

• Would it cost more to run one ad than the other?
• Does one ad have some special appeal over the other?
• Would it be more costly to develop one ad as opposed to the other?
• Does one ad appeal more to your boss (or his or her boss!) than the other?

If the answers to each of these questions is "no," then the most logical decision to make is simply to "go with the bigger number!" That is, run ad "A" which had an 82% recall score, rather than Ad "B" which had a 78%. It really isn’t any more difficult than that!