orgtheory.net

Previous orgtheory posts on  controversial statistical topics: interaction terms, Bayesian statistics, p-values and survey response rates.

I’ve come to the conclusion that a common statistical practice – the reporting of an R-sqaured for a logit or other categorical data model – is lame and we should immediately stop. Why? Let’s count the ways:

• Categorical data models estimate the effects of unobserved latent variables. You can’t compare the estimate with this purely theoretical object. All R2 are weird concoctions.
• There’s literally seven different R-squared statistics (at least) and they don’t always correlate well. Most people don’t know the difference between them. Even statisticians disagree on which one you should use.
• Pseudo-R2 measures usually do not measure variance, but they measure changes in likelihood and related quantities. For example, McFadden’s R2 measures changes in likelihood functions, which have no obvious interpretation. Unless the pseudo-R2 is either 0 or 1, the statistic is uninterpretable in relation to your data.
• The pseudo-R2 is not an analog to the OLS R2 and it’s misleading to say so. But folks believe it is.
• The pseudo-R2 for some models, like the Tobit, actually create *neagtive* values and are thus confusing for many readers.
• Reporting pseudo-R2 leads to the reader to think that you are directly measuring goodness of fit. Wrong.
• There is no decent way to decide what is good or bad fit. A psuedo-R2 measure of .02 tells you nothing about the match between predicted and observed quantities.

For the full run down, check out this 2007 article by Illinois b-school’s Glenn Hoetker in the Strategic Management Journal. Thanks to my colleague Scott Long, for bringing this article to my attention.