getting what we measure
Mikaila Mariel Lemonik Arthur is an Associate Professor of Sociology at Rhode Island College and is the author of Student Activism and Curricular Change in Higher Education. Her current research explores network effects on curricular change in higher education. Her primary teaching responsibilities include social research methods and law and society courses, and this spring she is teaching a new interdisciplinary upper-level general education course on higher education.
One of the hallmarks of modernity is the focus on rationality and efficiency in organizational function: organizations of all types, from hospitals to Fortune 500 corporations, from universities to small not-for-profits, seek to improve their performance in terms of measureable outcomes. But, as the aphorism goes, “What gets measured gets done, what gets measured and fed back gets done well, what gets rewarded gets repeated” (variously attributed to any number of management scholars). For example, pharmaceutical companies’ focus on stock prices, sales figures, and the next blockbuster drug has led to a focus on treatments for common, chronic conditions, such as the umpteenth heartburn medication, and less focus on the development of new antibiotics, a trend that may soon prove to have devastating effects on our attempts to control infections disease.
In higher education, a similar dynamic is occurring. In the past, colleges and universities were primarily measured (and funded) based on enrollments. This meant that encouraging more students to enroll, and keeping them enrolled in classes until after the third week (or whenever official enrollment statistics are due), was often the highest priority, and whether students ever graduated did not matter nearly as much. You get what you measure: students in seats.
More recently, the emphasis has shifted to retention and graduation as measureable outcomes. This change encouraged administrators to consider what was necessary to keep students in school and to improve time-to-degree, but it came with its own perverse incentives. For example, administrators turned to student evaluations as a way to increase student satisfaction; some colleges and universities discourage faculty from failing students because failures decrease graduation rates and increase dropout rates. This leads to colleges in which students can graduate with a 2.0, never having written a paper (a phenomenon discussed in recent books like Arum and Roksa’s Academically Adrift and Armstrong and Hamilton’s Paying for the Party). It also contributes to rampant grade inflation, including elite institutions where over half of all grades awarded are As (happy students=repeat customers). You get what you measure: grads with high grades.
A variety of colleges and universities have thus sought ways to curb grade inflation, such as providing average class grades on transcripts and setting strict grading curves. By encouraging tougher grading standards, these methods may indeed reduce the average GPA of enrolled students, but tougher grading standards do not necessarily translate into better educated graduates—and in any case, most colleges and universities have not chosen to enact these sorts of reforms. Indeed, the ease by which average grades can be manipulated highlights the fact that grades themselves may not be even an adequate proxy measure of student learning, and thus the assessment movement was born.
Today, accrediting agencies require colleges and universities to demonstrate that students meet measurable learning outcomes, and projects like the Lumina Foundation’s Degree Qualifications Profile encourage institutions and departments to clearly state the intended outcomes of their programs in measureable language. Some colleges and universities have gone further, developing competency-based degrees in which students supposedly demonstrate their skills rather than their seat time to graduate. At first blush, many critics argued that these programs are just another kind of teaching to the test. But teaching to the test is only a problem if the test is not actually able to test the desired learning outcomes—you get what you measure: results on the test.
It has already become clear to advocates of competency-based learning that competency is a pretty low floor, and instead they have begun to use the term “proficiency.” One goal of proficiency-based degree plans has been to shorten the time and cost of a degree, particularly by reducing Baumol’s cost disease by disrupting the relationship between seat time, faculty workload, and degree production. So far, competency- and proficiency-based programs are rare and likely appeal only to a particular self-selected group—but as Chambliss and Takacs point out in their forthcoming book How College Works, college only works if it works for all students, including the lazy, the unmotivated, and the perhaps not-so-smart.
So if we get what we measure and what gets rewarded gets repeated—and we measure proficiencies and reward completion—what do we get? Degrees as checklists? Students who cannot earn a college degree because, while they are excellent writers and have superb disciplinary knowledge, they cannot (in Lumina’s language) construct and define “a cultural, political, or technological alternative vision of either the natural or human world,” a key bachelor’s-level competency? An even more extreme bifurcation of the higher education field in which some colleges and universities develop rigorous proficiency measures and provide students with the supports necessary to excel while others assess writing, critical thinking, and speaking with machine- or peer-grading?
Or is it possible to build a system that measures proficiencies in a real, valuable way and which rewards completion without reducing the rigor of these proficiencies? In other words, can find a way to measure what we want to get instead of getting what we happen to have measured?