economists and survey skepticism

Over at Evil Twin, Nicolai Foss gently chides Bloom and Van Reenen for publishing a paper in the AER proceedings called “New Approaches to Surveying Organizations.” The issue is the validity of survey data versus other types of data:

As a rule register data are not available that can be used to address numerous interesting issues in organizational economics, labor economics, productivity research and so on. Scholars working on these issues have to resort to those softy surveys and interviews that have been the workhorses of business school faculty for decades. This is a new recognition in economics. Case in point: A recent paper by Nicholas Bloom and John Van Reenen, “New approaches to surveying organizations.”  There is absolutely nothing, I submit, in this short, well-written paper that would surprise virtually any empirically oriented business school professor (i.e., virtually all bschool professors) to whom this would not be anything “new” at all, but rather old hat.

This is not a critique of Profs. Bloom and Van Reenen at all (on the contrary, it is excellent that they educate their economist colleagues in this way). It is just striking and a little bit amusing, however, that we have had to wait until 2010 until empirical approaches that have been mainstream in management research for decades reach the pages of the American Economic Review.

I agree. In the comments, Bloom argues that he didn’t find any papers addressing these issues. This is odd, a lot of the suggestions for surveys make sense and many are well discussed in the literature on surveying individuals. For example, did they consult Dillman’s works? There are also handbooks discussing surveying organizations. There’s a huge industry of people who study survey bias.

A few additional comments: I have heard multiple economists express survey skepticism. The correct response is that reliability of survey responses varies and some questions are better than others. For example, people seem to be pretty good at reporting health, while they outright lie about attending church. Surveys by themselves aren’t good or bad, but individual questions can be high quality or low quality. Also, a lot of our most important data is from self-reports – like the Census, CPS, HRS, etc. I don’t see people ditching the Census.

Second, the real problem in survey research in organizations isn’t bias. It’s response rate. There’s all kinds of tricks to boost response rates for people, but getting people to respond at work (or about work) is really, really hard. And it’s miserable for longitudinal work. If Bloom and Van Reenen can produce a solution to low response rates from orgs, I’ll be really impressed.

Adverts: From Black Power/Grad Skool Rulz