the rhetorics of numbers

As an extension of my work on rankings, I’ve been thinking lately about how different actors understand or interpret the same numbers differently.  This is still a little vague (ok, I’m being modest here—it’s a lot vague), but I thought this might be a good place to get some early feedback, especially in terms of some organizational applications.

A central and consistent finding of work on public evaluation is that once measures are released, they are very much unleashed: they are interpreted in unpredictable ways, they are used for a wide variety of political purposes, and they have the potential to generate as many unintended as intended consequences.  Most of the attention given to this disjuncture between how numbers are designed and how they are used has focused on the back end of this process—how numbers, once created, are used and misused. We know far less, however, about how those who make numbers, especially those who do so professionally, think and talk about these numbers as they produce them.  A better understanding of the front end of the process—this “culture of number production”—would, I think, provide some fresh insights into why some measures stray so far from the purposes for which they were designed  (although I’m sure it won’t prevent them from straying in the future).

It would then be interesting (at least to me) to compare the views of producers to how various users of these numbers think and talk about them. This might be best done with a very public evaluation such as the SAT, comparing how the people at ETS talk about the meaning of these numbers to the views of student test-takers (possibly both before they take the test and when they are applying to colleges or deciding not to), the parents of these test-takers, admissions officers at colleges, etc.  An important part of this would be trying to see if and how it mattered that some people talked about these numbers in terms of standard deviations and confidence intervals, others in terms of “smart” or “dumb,” and still others in terms of “college material.”  But such an approach could also be an informative way to look at any third party evaluation of organizations: What was the measure intended to relate and how was it designed to do so? How do those who are evaluated interpret the results? How does these interpretations differ, if at all, from those of interested constituents?

Finally, and a bit tangentially, it would be interesting to study how the presentation of numbers and statistics affects how people react to them.  For example, one of the ways in which the medical field is combating the spread of deadly but largely preventable infections within hospitals is by framing the statistics in a different way so that doctors and nurses take them more seriously.  Knowing that 10 out of 1000 patients contract staph infections may not stir medical personnel to change their behavior, but discussing the case histories of the 15 people who have died in your hospital, along with the specific accidents or acts of negligence that led to their death, might be more likely to motivate people to wash their hands more or be more careful with those darned catheters.  It seems possible, then, that the disjuncture between the creation and use of evaluative measurements could be improved through a better understanding of how people respond to variations in how these measurements are “packaged.”