is evidence-based management still alive?
A commenter asks a good question: Is evidence-based management still alive?
From my perspective, here are a few questions and concerns related to evidence-based management:
- Whose evidence? Even in the academic literature there is no consensus on such basic things as, say, the proper role of incentives in organizations — and thus one can invariably find “evidence” and arguments to support almost any management practice.
- Who has time for evidence? So, managers deal with highly uncertain situations and amassing evidence for, say, a particular strategy can be hard if not impossible, certainly time consuming. In practice, managers may not be able to spend time pulling together information, or information simply may not be available, and thus managers just have to act. There’s a tendency by academics to gleefully point to various stupid mistakes that managers and organizations make — undoubtedly they do. But sometimes I wonder, particularly when I’m cynical: what, if anything, would get done if academics ran organizations?
- What does evidence and data tell us? In other words, there’s an important epistemological question related to evidence as well: does data tell us the truth? Evidence tends to give primacy to what we see, observe, perceive — usually in the form of data. But, while we can readily see that the sun revolves around the earth, we now of course know otherwise. Thus, rather than give sole primacy to empirics and data — which evidence-based management seems to emphasize — I think theory and logic also deserve careful consideration.
orgtheory.net 
Everytime someone learns and applies things, they’re using evidence-based management; just not the academic, formal kind.
This issue of The Economist also has a good article on Cisco’s management, which seems evidence-based to an extent.
LikeLike
Thorfinn
September 3, 2009 at 12:23 am
There is an EBMgt collaborative within the Academy of Management, and this group held a few sessions related to the topic at AOM this year.
The listserv for this group is EBMGT@AOMLISTS.PACE.EDU. There is not an extremely high level of activity on the listserv, but there is certainly some activity.
I am doing some work related to points #2 and #3 above, so I will be very interested in any additional comments stimulated by Teppo’s post.
LikeLike
TJ
September 3, 2009 at 12:28 am
You pose interesting questions, but when I read this post, I was actually thinking something different. That is, is there an analogy of evidence-based medicine which differs from evidence-based management?
My rudimentary understanding of evidence-based medicine was that it came about by medical practitioners wanted to look at the circumstance and situation and then sharing practices, R&D findings, etc. to work toward treatment and approaches as opposed to the legacy method of working from drug towards what things do the drugs treat.
I think in the business world, R&D exists. I also think that b-schools and the like work from cases back towards applying different tools and trying to understand the effects. I also think that individual authors try to amass their anecdotal findings.
But on first blush, I’m not sure that the management world has the same kind of emerging processes for aggregating these findings and practices into a usable knowledge and simulation bases like the medical world has been doing. Or at least the knowledge bases seem very dispersed throughout various functional areas and industries …
LikeLike
Steve Shu
September 3, 2009 at 3:38 am
Thorfinn: I think your use of “evidence-based” muddles things. Interestingly, so do Pfeffer and Sutton.
The only useful distinction to make is between basing decisions/practices on evidence or basing them on experience. There is no doubt (in my mind) that *theory* should be based on evidence. The question is whether practice should also be evidence-based.
And only “academic” evidence should be considered in distinguishing EBMgt from its alternatives. So, an argument for EBMgt is an argument for basing management decision on scientific research about management.
Pfeffer and Sutton, like I say, confuse things a bit. There is an example of this in their YouTube video (about learning from experience) and in their discussion of forced rankings at a “renowned but declining high-technology firm” (HBR 2006, p. 68). FRs were implemented on the basis of McKinsey’s The War for Talent *not the experience of top managaers* and *based on the evidence* provided in that report. Now, P&S argue that the report, and thereby the evidence, was flawed.
But it seems obvious that IF you are going base decisions on evidence, it should be GOOD evidence. As Teppo rightly points out, the fact that evidence is often flawed and biased is a general argument *against* EBMgt, and the problem described by P&S is simply a case in point. In their example, EBMmgt *caused* the problem. Better evidence may have supported a better decision, but imperfect evidence is very common.
The alternative is for managers to base their decisions on knowledge they personally master, understand, and are in a position to update on a continuous basis. I.e., personal experience: the opposite of EBMgt.
(I haven’t made up my mind about whether EBMgt is generally more advisable than experience-based alternatives. I’m just trying to pin down the distinction.)
LikeLike
Thomas
September 3, 2009 at 9:14 am
Jerry’s earlier post puts a whole new spin on the value of evidence-based medicine. If the research itself is tainted by corporate powers, then one should be skeptical about blindly adopting practices backed up by evidence.
LikeLike
brayden
September 3, 2009 at 1:52 pm
I agree. While I think scientific studies, with very public access to the raw data, should be done on medications and treatments to see how well they work, I think experience-based medicine has the advantage that the prescribing doctor gains familiarity with side-effects and symptoms of sub-optimal doses.
The same goes for management decisions. While an evidence-based decision may be largely “correct”, the implementing manager may lack the requisite “feel” for the “proven” process to tweak it. She might be better off implemting a less optimal practice with which she is familiar.
LikeLike
Thomas
September 3, 2009 at 2:06 pm
I like the point by Thomas that “The question is whether practice should also be evidence-based”. I guess my main point was that managers are mostly left to theory and personal experience as opposed to gaining additional perspectives via the more crowdsourced nature of EBMgt, however limited or inaccurate.
Theories may be inaccurate as well. Consider the simple biases that once something is published it uninteresting or hard to publish something that refutes a prior publication (e.g., such of the case where it is well-known by others in the field that the results of a particular experiment underlying a theory could not be reproduced).
LikeLike
Steve Shu
September 3, 2009 at 2:16 pm
As an organizations scholar whose wife is a doctor, I’ve thought a bit about the difference between the management and medicine domains with regards to being evidence based. All the epistemological concerns aside about whether scientific research provides useful truths, there is a parallel sociological problem that is more pressing. In medicine, it is valid and even encouraged for people to blur the lines between scholarship and practice, where in management there is a chasm between practitioners and academics. Management Ph.D. students interact with MBA students minimally if at all, and the cross-readership between HBR and ASQ is small at best. In medicine, the top doctors are the top researchers, and everyone reads JAMA and NEJM, both of which include a range of articles from doctors’ reflections on practice and policy to the most rigorous quantitative research. These journals both provide the evidence and the discussion of how to do evidence-based medicine in practice.
To me the question of evidence based management is moot unless we’re talking about these structural differences. To me the root cause of these problems always end up coming back to the status consciousness of management scholars, and the desire to validate ourselves among our academic colleagues by passing as sociologists, psychologists, economists, and political scientists. That pull for academic status will always pull our research attention, social networks, and professional goals (e.g. publishing vs. consulting) away from the dialogue with practitioners that is essential to evidence-based anything.
Medicine, as a much more established, prestigious, and thus less status conscious field, doesn’t have to play these games as much. There are some researchers gunning for publications in Science, Nature, and the basic biology journals whose medical practice may be seen as less prestigious, but there is a huge sphere of “valid” identity and accomplishment at the nexus between evidence generation and application.
Andrew Van de Ven’s work on Engaged Scholarship is I think a strong remedy for some of this, by articulating what the practice would look like. But really making that vision real would require some more radical transformations of how we see ourselves and what we are striving for.
LikeLike
Jason Jay
September 3, 2009 at 6:26 pm
“the top doctors are the top researchers”
This gives me pause for thought. It seems intuitively right, and marks an interesting difference between doctors and managers.
But how true is it? I’m thinking about cancer and diabetes research. Does it apply there too?
LikeLike
Thomas Basbøll
September 3, 2009 at 6:36 pm
Quick note on the question “who has time for evidence?” Having worked in an executive position for a large pharma company, my experience is that in practice there is quite a bit of evidence-gathering, although it tends to suffer from the problems you mention: the evidence serves an agenda, it’s of dubious quality, it’s cherry-picked from whatever is easiest to get. In that sense, it’s more like law than medicine–the evidence is circumstantial at best, and perhaps not even evidence at all.
To the point that “managers just have to act”: a great many management decisions are made based on experience and intuition, and little more. Still, there are different kinds of decision, and different criteria in reaching them. Not all management decisions are made from the gut. Many rely on various inputs, careful consideration, and logic. That said, what almost never goes into executive-level decisions is theory. Most executives don’t read ASQ.
LikeLike
Joseph Logan
September 4, 2009 at 2:01 pm
“evidence based management” is one of those jargons that made me cringe by the time I was into my 2nd year of business school. What other kind of management is there ? guesswork-based-management ? unsubstantiated-theory-based-management?
LikeLike
sd
September 4, 2009 at 3:22 pm
Well, sd, as in medicine, there is a meaningful distinction to be made between basing decisions (in medicine: treatments) on scientific evidence and basing them on practical experience. It becomes a jargon only when we don’t use it to mark that distinction (see my comments above).
LikeLike
Thomas Basbøll
September 4, 2009 at 3:43 pm
Your points pose an even stronger case for research doctors to dispense with clinical trials than they do for managers to dispense with evidence-based policy. They are more over-worked, they have a more complicated system, and they receive inconsistent information from significantly more diverse sources and entrenched interests. But try to make a case against clinical trials.
And point three hurts me. Of course we don’t have direct access to the state of the systems we intervene in, and are working with a sampled biased towards the observable, but anything else is, as above, guesswork-based-management and unsubstantiated-theory-based-management.
And while these other two approaches may often perform as well as an evidence-based approach, only the latter has an established methodology for building consensus as to how to improve: by closing the feedback loop with observation in a way that controls for human limits.
Whether the methodology will always work to build a consensus is a different story and subject to many of the same human limits as above. The only thing going for it is that it is better than anything else.
LikeLike
Seth
September 4, 2009 at 3:55 pm
[…] be certified. This certification issue seems fraught with some of the same problems as “evidence-based management” — passing muster depends on whose evidence we are using and who is certifying. […]
LikeLike
can organizational strategies be certified? would wal-mart’s strategy be certified? « orgtheory.net
December 17, 2010 at 9:57 pm