lemma give you some advice…KISS
Omar
In the latest issue of Econ Journal Watch (the site appears to be currently [4/16/08 10:30am EST]) down for some reason), there is a very interesting piece (link here) by Philip Coelho and James McClure, amusingly entitled “The Market for Lemmas: Evidence that omplex Models Rarely Operate in Our World.” The purpose of the paper is to investigate the “scientific returns” to complexity in math-heavy argumentation in economics. Their argument, drawing on classic Marshall, is that there is that long, complex chains of mathematical reasoning are likely to be less that optimal because (1) long and complex chains of reasoning are more likely to be wrong (”truth content declines as an exponential function of the number of links in the reasoning chain”), and (2) less likely to have empirical implications (”less likely to contain operational statements” as they put it in a rather arcane Vienna-Logi-Po phrasing).
Their analysis looks at a bunch of articles published in the main Econ journals (AER, JPE, etc.) throughout the century, with a particular focus on the last three decades. They find two things: first, the number of articles that contain the word “lemma” has increased dramatically over-time. Their JSTOR search yields about 22 articles who contain the word lemma in the 1960s, as compared to 353 in the 1990s. From this they surmise that the math-complexity of the average Econ mathematical-theoretical article has been increasing over time. Second, they find that mathematically complex articles (their “operational criterion”, is [hence the title] simply counting the number of “lemmas” that the article includes; they define “high mathematical complexity” as containing five or more lemmas) are less likely to either contain empirical statements or to attempt to provide empirical assessments of the derived propositions. Finally, they show that mathematically complex articles in economic theory as defined above are less likely to be cited in comparison to lemma-heavy econometric articles. The conclude that a lot of economic theory, in its removal from empirical reality an entanglement in a skein of lemmas, runs the risk of being “not even wrong” (a fate worse that just being plain wrong in their view).
This article is interesting and got me to think about an organizational theory that I like very much, but that has become increasingly lemmalized in the last few years (Fabio has a nice assessment of the most recent book here). While Hannan, Polos, Carroll et al, see the increasing formalization of the theory as a good thing, and they rely on more flexible styles of logical formalization (non-monotonic, fuzzy, etc) that are much less constraining that the usual first-order logic of most economic theory, if Coelho and McClure are correct, we should find that the impact of the new and improved lemma-heavy version of org ecology should decline with the addition of each lemma. However, one good thing that Hannan et al have going for them is that they are trained sociologists (Polos is a logician of course) and they don’t mind including “operational statements” and (god forbid) writing a grant to get some data to test the empirical implications of the theory.
they don’t mind including “operational statements” and (god forbid) writing a grant to get some data to test the empirical implications of the theory.
Yes, I look forward to lazy-s shaped graphs for years and years to come.
Peter
March 16, 2008 at 3:21 pm
I agree w/Omar, but I have a different emphasis. The problem with much mathematization is not the chain of reasoning. Just because a theorem has a long proof doesn’t mean it’s wrong because ideally mathematical proofs don’t depend on probabilistic steps. Each link has to be logically correct. The issue then is human error in indentifying errors. For example, the Appel-HAken proof of the four color mapping theorem was very long, but each step, beyond the sketch of the proof, was computer generated. My beef with formalization is that using fancy math permits us to avoid hard science.
fabiorojas
March 16, 2008 at 6:06 pm
because ideally mathematical proofs don’t depend on probabilistic steps. Each link has to be logically correct.
Proofs in cases like this have to be valid, but they don’t have to be sound. Given that formal economic theory often requires rather strong — indeed, empirically false — axioms and assumptions to get off the ground, maybe the point is that longer and more complex arguments based on them are more likely to end up in bad or pointless places. Saying ”truth content declines as an exponential function of the number of links in the reasoning chain” is a sloppy sort way to put it. Ironically enough, it uses technical language purely for effect, and in a way that is mistaken if taken literally.
Kieran
March 16, 2008 at 9:49 pm
Sociology is the discipline that is usually accused of having too much complexity in our theorizing. Formalization, at least of the type that Hannan et al. are doing, is meant to reduce some of that complex theorizing into simplified statements that should lead to testable hypotheses. This was the reasoning behind introducing formal logic into sociological theory. It turns out that most sociological theories are so complex (i.e. messy) that they can’t be subject to the rules of formal logic.
brayden
March 16, 2008 at 11:23 pm
This is ignorant; but, might the accumulation of lemmas suggest systematic progress in a given field? If lemmas represent underlying accumulated assumptions, statements perhaps proven by (or accepted b/c of) previous work, and then these accumulated lemmas are used as a scaffold to move arguments/theories forward — seems to work, or?
[I guess, if you pile on enough of these general (and thus partly false) statements onto each other, then, perhaps after a while you get so far from reality where even generality is lost --- I guess thats the point made in the post.]
tf
March 17, 2008 at 1:31 am
Brayden sez: “It turns out that most sociological theories are so complex (i.e. messy) that they can’t be subject to the rules of formal logic.”
So the logical rules that work for “easy” fields like brain surgery and nuclear physics somehow fail to work for organizational behavior? C’mon!
Fabio Rojas
March 17, 2008 at 4:05 am
Come on Fabio, brain surgery ain’t rocket science.
Kieran
March 17, 2008 at 4:44 am
Teppo,
Coelho and McLure’s argument against lemmas comes from the empirical observation that lemma-heavy arguments in economics are less likely to connect to the empirical world (and therefore less likely to impact ongoing research). Clearly this is not a logical (!) necessity. There’s no a fortiori reason why having lots of intermediate linkages in an argument dictates the absence of empirical applications. Instead, it seems that what’s happening is that epistemic culture of economics (which shouldn’t be surprising to anybody with even a cursory knowledge of the discipline) appears to be fostering theorizing that does not have any clear empirical payoff (otherwise reviewers would be asking for those empirical implications to be made clear in those papers).
So, I guess the answer to the query as to whether an accumulation of lemmas signals progress or not is: it depends! Are the various “research programs” that are formalized in economics (in Lakatos’ sense) “degenerative” or “progressive”? I think it would require someone with more local knowledge of economics to answer that question, but if a lot of the formalization is really not leading anywhere empirically, that could be taken as preliminary evidence of a degenerative research program, covering itself up with a protective belt of logical implications with no connection to empirical reality.
Omar
March 17, 2008 at 12:44 pm
My point was just that the first attack of formal logic in org. ecology was intended to simplify or bring order to the theory rather than make it more complex. Whether or not the use of logic contributes to the empirical study of organizations will depend on how generative the project has been. My guess is that ecologists’ will continue to have plenty to offer empirically. It began as an empirical enterprise and my guess is that empirical ecological studies will not die off quickly.
You have to give credit to Hannan et al. for producing students who are well grounded in both theory and empirical capability. Last year at AOM I saw a really fascinating panel that came out of a Hannan doctoral seminar at Stanford in which each student (sometimes paired with a faculty member) investigated some aspect of the grades-of-difference question (see Hannan, Polos, and Carroll). If this is any sign, it appears that the formal logic project has been extremely generative.
brayden
March 17, 2008 at 1:17 pm
Brayden: Impressive, sounds like a nice programmatic, cumulative effort.
tf
March 17, 2008 at 2:15 pm
Bryaden: My sentiment is something like
(a) in most cases, alternative formal logic adds nothing that can’t be said in regular English and standard math
(b) the use of alternative forms of logic covers up conceptual problems (e.g., instead of figuring out why X and Y are both true, you introduce loose rules to allow you to say “X and Y” and then move on without offering any fundamental solution)
For example, let’s stick to the grades of difference issue. I didn’t go to this conference, but my guess is that you could probably describe the entire problem and solution in standard english and plain old high school algebra. I’ve read HP&L and that’s what I thought: nice application of partial membership. Why do we need non-monotonic logic?
At the deepest level, here’s the issue: throughout the history of science, we’ve heard periodic calls for bizarre new math. And what’s the pattern? Some excitement, then it’s a dud. Why? Usually, progress in science doesn’t come from new forms of reasoning. Progress comes from discovering new ideas using old fashioned reason.
Even pop ecology fits that pattern. What are the biggest achievements of the field? (a) resource competition, (b) age effects, and now (c) establishment of identities in markets. Did any of these ideas come from fancy math? None. They all came from thinking hard about social processes. None are the outcomes of mathematical reasoning. Math is just the dressing that makes it acceptable to people.
fabiorojas
March 17, 2008 at 3:26 pm
…except that when formal logic was first used (see my above link), it uncovered some theoretical inconsistencies that Hannan et al. needed to deal with before the theory could hold together. Remember, the org. ecologists didn’t bring formal logic into the ecology to make it more acceptable; they were forced to use it in order to combat claims that their theory had internal inconsistencies.
My guess is that many of our theories, usually laid out with plain old English, aren’t entirely consistent or coherent. But we’re usually okay with that because we’re more concerned about the empirical contribution (the kinds you mention in your last comment Fabio) than we are with the logical fluidity of our theory.
brayden
March 17, 2008 at 4:18 pm
Brayden, I’d argue that the inconsistencies come from the assumptions of PE, not the use of first order logic. Basically, if through regular old reasoning, I came up with conflicting claims, my first guess is that the theory is wrong, not first order logic! This whole research paradigm just jumps to the extreme conclusion that you need entirely new logics rather than consider that maybe PE is wrong on some level, which is a simpler hypothesis.
fabiorojas
March 17, 2008 at 7:24 pm
Omar,
Coelho and I provide evidence consistent with Don Gordon’s hypothesis that mathematical complexity and the generation of operational propositions are negatively related in economics. From Gordon:
“[T]he relationship between x and y may be stable long enough for a shift along that function but not stable long enough for a shift along that function plus a subsequent shift along another [z].”
Above, think of y = f(x) and z = g(y), so that z = g[f(x)]. Consider the impact of a change in x upon z:
dz/dx = (dz/dy)×(dy/dx).
Intuitively, long chains are used in economics to try to explain relationships involving a multitude of interrelationships. Mathematics is timeless, but the interrelationships unfold in historic time. The point is that the ceteris paribus implicit in mathematics is contrary to economic reality.
For example, in economics, in general equilibrium theory, one might formalize the impact of a change in peanut butter prices in Pittsburg upon the price of tea in China. But the long chain of mathematics (timeless) disregards the passage of time it takes for the impact to unfold in the real world. The longer the chain the more likely it is that the passage of time in the real world will confound the predictions of a chain that assumes ceteris paribus for each link.
J. McClure
March 21, 2008 at 10:26 pm