Millennium Marketing Research®
Tom Schori DBA Millennium Marketing Research®, 808 Ironwood, Normal IL 61761, 309-532-8466

Be on the lookout for 'Tom Swift and his factor analytic machine.'

By Thomas R. Schori, Ph.D., and Michael L. Garee, Principals,  Millennium Marketing Research, 808 E. Ironwood, Normal, IL 61761-5239. 

It has been our experience that researchers, even very respected ones, often fall prey to applying statistical procedures that they really aren’t qualified to use. As a result, they may use statistical procedures that aren’t appropriate to the problem at hand, interpret the results incorrectly, or both.

Just a few months ago, we attended a conference hosted by a very well-known research organization. In one of the sessions a senior researcher presented factor analytic results pertaining to a study the organization had just completed. He identified four factors that accounted for 50%, 25%, 15%, and 10% of the information associated with 100 original variables. He then misinterpreted this observation as meaning that the first factor represented the most important driver of consumer behavior. Cringing at his blunder, it immediately brought to mind an article we wish that we had authored.

Years ago, an article appeared in the scientific literature entitled, "Tom Swift and his factor analytic machine." The author decried the fact that, as a result of the "ready availability "of computer programs with which researchers could perform factor analyses, many investigators were using factor analysis in senseless ways. In essence, they were just churning out one factor analysis after another, none of which made any sense whatsoever.

For those of you who are mathematically gifted, factor analysis is simply a procedure that reduces the dimensionality of a matrix of less than full rank¾ the well known Eigen vector/Eigen value problem. Put in more understandable terms, factor analysis is used to identify the underlying dimensions of some characteristic, such as intelligence, product satisfaction, etc. For instance, we might have a test battery of, say, 200 items, all of which are thought to be related to what we term intelligence. Intuitively, we know that all 200 items don’t represent independent aspects of intelligence. Consequently, we might have a representative sample of 1000 people take the test. Then, we might decide to perform a factor analysis to determine what independent factors the 200 items represent. From that analysis, we might identify four broad, general, underlying factors, which we could term quantitative reasoning, verbal reasoning, creative thinking, and visualization~four factors which subsume the vast majority of the information associated with all 200 items. From each individual’s responses on the 200 items, we could then calculate scores for each individual and record them under the four factors. But significant to note, we still wouldn’t be able to conclude which of the four factors was the most important factor of intelligence. And, that of course, was what the senior research mentioned above was attempting to do when he analyzed the data from his research.

Just how right the "Tom Swift" author’s observations had been became abundantly clear shortly after having read the article, when one of the senior consulting partners happened across a scientific article in a well-respected professional journal. In the article, an investigator reported having performed a factor analysis on 16-PF scores, i.e., on 16 personality factor scores. As it turned out, though, the personality factors used on the 16-PF, had originally been factor analytically determined. Suffice it to say that factor analyzing factor analytically determined scores make no sense at all. None whatsoever! A fact which apparently escaped the investigator and, sadly enough, those individuals who, presumably, reviewed the article for its scientific merit prior to accepting it for publication. The end result of the scientific article was that all the conclusions drawn were utter nonsense.

Of course, it’s not just factor analyses that can easily be performed and "wrongly interpreted." It’s also canonical correlation, multiple regression, discriminant analysis, cluster analysis, path analysis, multidimensional scaling, multivariate analysis of variance, univariate analysis of variance, analysis of covariance, and many, many more complex analyses, plus the more simple statistical procedures like the t-test and Chi square. Each of these statistical procedures has its place of course, but when they are misused and misinterpreted they become worthless.

The question becomes, how does one avoid being misled by amateurs in the guise of seemingly credible researchers, i.e., paying attention to researchers who misuse statistical procedures and/or misinterpret statistical findings? Here are some ways to avoid it:

  • If the researcher uses complex statistical analytic tools of the sort mentioned above but is not a recognized statistician, then have a recognized statistician review the appropriateness of the analyses which were employed as well as the interpretation of the conclusions.
  • Make sure the researcher tells you the name of the statistical procedure used to do the analysis, not just the name of the software program used. It currently is in vogue for statistically unqualified researchers to say these data were analyzed using Joreskog and Sorbom’s LISREL8, rather than saying that these data were analyzed using a structural equation model. Watch out! Which program the researcher uses should be of no consequence whatsoever. Telling what program one uses is generally a hint that the researcher knows nothing more about the analysis than that which can be found in the one paragraph or so blurb about it in the statistical manual accompanying the program. What is important, of course, is what type of analysis was employed and what conclusions can be drawn from the results it produced.
  • Be on guard for researchers using more than one criteria for statistical significance in a given research report, e.g., p < .05, p < .01, and p < .001. Using three levels of significance for different statistical tests suggests that some results are "more significant" than others. Nonsense. The statistically competent research will use one criterion for significance and only one.

Watch out for Tom Swift and his factor analytic machine and those who act like Tom Swift!