orgtheory.net

getting what we measure

Mikaila Mariel Lemonik Arthur is an Associate Professor of Sociology at Rhode Island College and is the author of Student Activism and Curricular Change in Higher Education. Her current research explores network effects on curricular change in higher education. Her primary teaching responsibilities include social research methods and law and society courses, and this spring she is teaching a new interdisciplinary upper-level general education course on higher education.

One of the hallmarks of modernity is the focus on rationality and efficiency in organizational function: organizations of all types, from hospitals to Fortune 500 corporations, from universities to small not-for-profits, seek to improve their performance in terms of measureable outcomes. But, as the aphorism goes, “What gets measured gets done, what gets measured and fed back gets done well, what gets rewarded gets repeated” (variously attributed to any number of management scholars). For example, pharmaceutical companies’ focus on stock prices, sales figures, and the next blockbuster drug has led to a focus on treatments for common, chronic conditions, such as the umpteenth heartburn medication, and less focus on the development of new antibiotics, a trend that may soon prove to have devastating effects on our attempts to control infections disease.

In higher education, a similar dynamic is occurring. In the past, colleges and universities were primarily measured (and funded) based on enrollments. This meant that encouraging more students to enroll, and keeping them enrolled in classes until after the third week (or whenever official enrollment statistics are due), was often the highest priority, and whether students ever graduated did not matter nearly as much. You get what you measure: students in seats.

More recently, the emphasis has shifted to retention and graduation as measureable outcomes. This change encouraged administrators to consider what was necessary to keep students in school and to improve time-to-degree, but it came with its own perverse incentives. For example, administrators turned to student evaluations as a way to increase student satisfaction; some colleges and universities discourage faculty from failing students because failures decrease graduation rates and increase dropout rates. This leads to colleges in which students can graduate with a 2.0, never having written a paper (a phenomenon discussed in recent books like Arum and Roksa’s Academically Adrift and Armstrong and Hamilton’s Paying for the Party). It also contributes to rampant grade inflation, including elite institutions where over half of all grades awarded are As (happy students=repeat customers). You get what you measure: grads with high grades.

A variety of colleges and universities have thus sought ways to curb grade inflation, such as providing average class grades on transcripts and setting strict grading curves. By encouraging tougher grading standards, these methods may indeed reduce the average GPA of enrolled students, but tougher grading standards do not necessarily translate into better educated graduates—and in any case, most colleges and universities have not chosen to enact these sorts of reforms. Indeed, the ease by which average grades can be manipulated highlights the fact that grades themselves may not be even an adequate proxy measure of student learning, and thus the assessment movement was born.

Today, accrediting agencies require colleges and universities to demonstrate that students meet measurable learning outcomes, and projects like the Lumina Foundation’s Degree Qualifications Profile encourage institutions and departments to clearly state the intended outcomes of their programs in measureable language. Some colleges and universities have gone further, developing competency-based degrees in which students supposedly demonstrate their skills rather than their seat time to graduate. At first blush, many critics argued that these programs are just another kind of teaching to the test. But teaching to the test is only a problem if the test is not actually able to test the desired learning outcomes—you get what you measure: results on the test.

It has already become clear to advocates of competency-based learning that competency is a pretty low floor, and instead they have begun to use the term “proficiency.” One goal of proficiency-based degree plans has been to shorten the time and cost of a degree, particularly by reducing Baumol’s cost disease by disrupting the relationship between seat time, faculty workload, and degree production. So far, competency- and proficiency-based programs are rare and likely appeal only to a particular self-selected group—but as Chambliss and Takacs point out in their forthcoming book How College Works, college only works if it works for all students, including the lazy, the unmotivated, and the perhaps not-so-smart.

So if we get what we measure and what gets rewarded gets repeated—and we measure proficiencies and reward completion—what do we get? Degrees as checklists? Students who cannot earn a college degree because, while they are excellent writers and have superb disciplinary knowledge, they cannot (in Lumina’s language) construct and define “a cultural, political, or technological alternative vision of either the natural or human world,” a key bachelor’s-level competency? An even more extreme bifurcation of the higher education field in which some colleges and universities develop rigorous proficiency measures and provide students with the supports necessary to excel while others assess writing, critical thinking, and speaking with machine- or peer-grading?

Or is it possible to build a system that measures proficiencies in a real, valuable way and which rewards completion without reducing the rigor of these proficiencies? In other words, can find a way to measure what we want to get instead of getting what we happen to have measured?

50+ chapters of grad skool advice goodness: From Black Power/Grad Skool Rulz

Written by fabiorojas

January 29, 2014 at 12:01 am

One Response

Subscribe to comments with RSS.

  1. First, thought it was an interesting post. Don’t know if I took my comment is off topic, but your post got me thinking.

    I am a trained sociologist who currently works in the program evaluation field, and I think your post touches upon some of the key issues that the field of program evaluation has been grappling with for a bit now. Here I am thinking of some of the work of Michael Scriven on the role of values in program evaluation (for more from Scriven see an overview of his model of program evaluation here: http://michaelscriven.info/images/KEC_7.25.2013.pdf). In your post, you touched upon a few things which made me of how we evaluate what happens within a given field.

    Within higher education, a program evaluator might ask: what standards (or values) are used to determine the merit, worth, and significance (Scriven’s words) of various actors/organizations/programs operating within the field of higher education.

    For me, I think it’s probably impossible for us to seek a set of standards that apply to all actors/organizations/programs operating within a complex field/system like higher education. Programs and organizations within higher education tend to have sets of values that are intrinsic to their operations and vary because of the nature of the program. Here I think of the differences between colleges focused on distance education or those that work with nontraditional students (maybe through retraining, vocational education, or clinical training like nursing) and an R1 university where more of a priority is given to basic research and training more researchers through doctorate programs. Internally, these organizations are motivated by different missions (and they should be because they operate to meet different societal needs/wants).

    However, both R1 universities and distance education schools operate within the same field and what they do and how they operate is also extrinsically motivated. Organizations and institutions external to them help determine their merit, worth, significance within the field. This is part of the reason why you can talk about something like an “assessment movement.” There are extrinsic expectations/requirements operating within the field of higher education, which have altered the behavior of a number of programs/colleges/universities operating within the field. These extrinsic requirements are often brought on by both organizations/actors operating within the field of higher education (government regulators, accrediting bodies, members of various disciplines, funding agencies, etc.) and organizations/actors operating in other fields (the market, employers, etc). They are important because they help influence what ultimately becomes valued within the field. Not to mention the fact that the field itself is operating within a societal and cultural context which shapes what is considered a “good” college education in one part of the world versus another part of the world.

    In general, I think an issue arises when we recognize that resources are finite and scarce within higher education (and any given field really). Thus, as you rightly hint at, an important task becomes determining how much weight should be attached to the different missions and types of organizations operating within the field of higher education so that we can maximize our resources and investments. Or as you put it, so “we can find a way to measure what we want to get instead of getting what we happen to have measured?”

    My only caution would be that we recognize that we want (and need) various outcomes within higher education. Thus, what we end up valuing will be multidimensional. This can allow room for all types of organizations including those focused on writing and critical thinking as you note, those focused on building good relationships between professors/students in small liberal/arts environment as Takacs and Chambliss seem to be proposing, and those which use testing and assessments as credit in lieu of courses such as the distance education/nontraditional models in order to get people trained for job vacancies or to fill gap between workers’ skills and the current jobs available.

    Like

    Scott Dolan

    January 29, 2014 at 3:35 pm


Comments are closed.

%d bloggers like this: