economists and survey skepticism

Over at Evil Twin, Nicolai Foss gently chides Bloom and Van Reenen for publishing a paper in the AER proceedings called “New Approaches to Surveying Organizations.” The issue is the validity of survey data versus other types of data:

As a rule register data are not available that can be used to address numerous interesting issues in organizational economics, labor economics, productivity research and so on. Scholars working on these issues have to resort to those softy surveys and interviews that have been the workhorses of business school faculty for decades. This is a new recognition in economics. Case in point: A recent paper by Nicholas Bloom and John Van Reenen, “New approaches to surveying organizations.”  There is absolutely nothing, I submit, in this short, well-written paper that would surprise virtually any empirically oriented business school professor (i.e., virtually all bschool professors) to whom this would not be anything “new” at all, but rather old hat.

This is not a critique of Profs. Bloom and Van Reenen at all (on the contrary, it is excellent that they educate their economist colleagues in this way). It is just striking and a little bit amusing, however, that we have had to wait until 2010 until empirical approaches that have been mainstream in management research for decades reach the pages of the American Economic Review.

I agree. In the comments, Bloom argues that he didn’t find any papers addressing these issues. This is odd, a lot of the suggestions for surveys make sense and many are well discussed in the literature on surveying individuals. For example, did they consult Dillman’s works? There are also handbooks discussing surveying organizations. There’s a huge industry of people who study survey bias.

A few additional comments: I have heard multiple economists express survey skepticism. The correct response is that reliability of survey responses varies and some questions are better than others. For example, people seem to be pretty good at reporting health, while they outright lie about attending church. Surveys by themselves aren’t good or bad, but individual questions can be high quality or low quality. Also, a lot of our most important data is from self-reports – like the Census, CPS, HRS, etc. I don’t see people ditching the Census.

Second, the real problem in survey research in organizations isn’t bias. It’s response rate. There’s all kinds of tricks to boost response rates for people, but getting people to respond at work (or about work) is really, really hard. And it’s miserable for longitudinal work. If Bloom and Van Reenen can produce a solution to low response rates from orgs, I’ll be really impressed.

Adverts: From Black Power/Grad Skool Rulz

Written by fabiorojas

March 20, 2012 at 12:01 am

3 Responses

Subscribe to comments with RSS.

  1. I think both you and Foss are overstating economists’ skepticism towards survey data. Almost all applied economists use survey data and willingly admit it. Being skeptical of the sources is different than not believing they are useful or informative.

    There are a couple major cases where I would describe economists as very suspicious of surveys:

    1. Poor sample design or statistical work. There are numerous cases of this among all disciplines and I would say economists can definitely be elitist snobs about this when they see other fields do shoddy statistical work.

    2. Reporting unsalient information. You can imagine some questions are just tough to answer because people don’t really know the answer to it either for lack of information or lack of relevance. Questions like ‘how many bananas did you buy last year?’ or ‘how much would I have to pay you to chop off your thumb?’ fail on this dimension in a way that ‘how old were you when you first got pregnant?’ does not. Now sometimes those responses are the best we may have but revealed preference techniques almost always strictly dominate survey responses in these cases.



    March 20, 2012 at 12:17 am

  2. You have to remember that Economists are quite spoiled when it comes to data because they rely on the government to collect most of it for them.



    March 20, 2012 at 1:23 am

  3. The AER Papers & Proceedings is not, to my knowledge, peer-reviewed; nor does it possess the same prestige in economics as the AER. Thus I wouldn’t infer from the presence of Bloom and Van Reenen’s paper an endorsement of those methods by the economics mainstream.


    Taylor Spears

    March 20, 2012 at 5:59 pm

Comments are closed.

%d bloggers like this: