Archive for the ‘education’ Category
Thad Domina, Andrew Penner and Emily Penner have a really interesting new paper out in Sociological Science, AKA the official journal of people on my Twitter feed.
So it turns out that some crazy Californians had the great idea to assign high school students color-coded IDs based on their standardized test results. Everyone got a white, gold, or platinum card based on their level of proficiency on the California state tests. The students had to display their ID card whenever they were on campus, and gold and platinum card holders got certain perks, including discounts on school events, but most visibly an “express” line in the cafeteria. What could possibly go wrong?
Well, this raised some eyebrows once word traveled beyond the schools themselves. The article doesn’t name the two schools, so I won’t either, but Google exists, and it sounds pretty clear that creating these visible new categories had real status effects. You can find quotes about how kids in honors classes with platinum cards told other kids in honors classes who only had a gold card that they shouldn’t even be there, and a principal apparently told some girls they should try to find platinum prom dates.
The program was in place for two years, then was shut down after quite a bit of unfavorable press. But not before our intrepid sociologists managed to collect some data.
The article looks at two main things: first, did the rewards affect test scores and other academic outcomes, and second, did creating status groups have effects.
The answer is yes on both accounts. Scores went up significantly on both math and English language arts (ELA) exams. As predicted, effects on other indicators of achievement (grades and scores on exit exams, a different test) were less consistently positive. Moreover, the effects were greater for students near the threshold (i.e. if you just barely missed the gold card one year, your scores increased more). Effects were larger for Asian students than white students, and larger for white students than Hispanic students.
So far, this is consistent (well, with the exception of the racial/ethnic variation) with behavior of the “adolescent econometricians” (p. 266, quoting Manski) assumed by the program. But here’s where it gets interesting.
The authors also use the card cutoffs to perform a regression discontinuity analysis. The students just above the gold card threshold basically look like those just below it. But it turns out they do much better. Or more accurately, kids who got the low-status white card do worse – to the tune of 0.35 SDs on the ELA test, and 0.10 SDs on the math test (both significant at p = 0.01 level).
Interestingly, the impact on grades is even greater: receiving a low-status white card reduces math grades an amount equivalent to moving from a C- down to a D, and ELA grades all the way from a C+ to a D. That’s a big drop. Domina, Penner & Penner speculate that this may be because the status categories are particularly salient for teachers, who may then treat or grade low-status students differently.
So as usual, the lesson here is that while yes, people respond to incentives, they do so in social contexts. You can’t just assume incentives are going to have similar effects on all groups of people, or ignore the effects of new status groups that are produced.
Two other thoughts here. The one interpretation I might quibble with is the authors’ attribution of the racial/ethnic differences to stereotype threat. While that’s one possibility, it’s also certainly plausible that the different response reflects cultural difference (e.g. in the importance attached to test results).
The other is about generalizability. Admirably, the authors don’t really try to generalize at all beyond saying that status categories have effects. But I can imagine people looking at an experiment like this and assuming that whatever the results were, they’d hold up across schools.
But status systems vary from place to place, of course. At the small rural high school I went to, wearing around a card advertising your high test scores would have been a fast ticket to social exclusion. Fryer and Torelli’s “Acting White” paper is hotly debated (and I haven’t followed the whole debate), but they argue that there are racial differences in whether academic achievement is associated with higher social status, and that this gap is greater in schools with more interracial contact. Clearly not every school mirrors the relatively wealthy, high-achieving schools that implemented this program, and not every kid associates high academic achievement with high social status.
That’s just a caution against assuming too much uniformity across social settings, though, not a criticism of the actual paper, which is a great use of a natural (?) experiment. And just think, they didn’t have to suffer through multiple R&Rs to publish it.
I just discovered an Economist article from last year showing that, once again, which college you go to is a lot less important than what you do at college. Using NCES data, PayScale estimated return on investment for students from selective and non-selective colleges. Then, they separated STEM majors from arts/humanities. Each dot represents a college major and its estimated rate of return:
Some obvious points:
- In the world of STEM, it really doesn’t matter where you go to school.
High prestige arts majors do worse, probably because they go into low paying careers, like being a professional painter (e.g., a Yale drama grad will actually try Broadway, while others may not get that far). [Ed. Fabio read the graph backwards when he wrote this.]
- A fair number of arts/humanities majors have *negative* rates of return.
- None of the STEM majors have a negative rate of return.
- The big message – college matters less than major.
There is also a big message for people who care about schools and inequality. If you want minorities and women to have equal pay, one of the very first things to do is to get more into STEM fields. All other policies are small in comparison.
I’m pretty sure that Atlantic-NYT-New Yorker-etc. readers have a near-bottomless appetite for articles on the elite college admissions process. I kind of do too, although I always feel a little gross afterwards, like I had one too many pieces of my kids’ Halloween candy. Not that I would ever eat their Halloween candy.
Anyway, the latest offering in this genre is a three-part series in the Atlantic (part one, part two, part three) by Alia Wong. Mostly it’s about how the process became so competitive, and what elite colleges are doing to try to make it less insane — in particular, how to stop incentivizing a certain kind of self-obsessed self-discipline among the the young demographic aiming at these schools.
The piece is well-researched, although a lot of this ground has already been covered exhaustively. But I did think the article overlooked a couple of key things.
First: Although the article is clear at the outset that this is a problem of “the 3%”, as Derek Thompson put it, it folds broader college-admissions developments in with elite college-admissions developments in ways that don’t always work. In particular, both parts point to enrollment management as a factor driving change in the elite admissions process.
I think enrollment management — the emergence of rationalized practices for selecting and aiding students to achieve some institutional objectives (financial, rankings, whatever) — is incredibly important in understanding changes in higher ed as a whole. But I suspect it doesn’t explain much of what’s changed in elite admissions in particular.
Enrollment management started as a financial tool for mid-level private colleges that were having trouble making ends meet (gated link). Elites have never embraced it quite so much, though they have adopted some of its practices. Where enrollment management matters is in understanding the rise of merit aid at less-elite colleges, for example, or in explaining the shift toward out-of-state students at public universities.
Second: So far, at least, the article overlooks the extent to which the elite-admissions-frenzy is driven by an increasingly winner-take-all society. If the returns to being in the 1% are increasing, then the returns to an Ivy degree are likely increasing as well. Conversely, as the middle class is hollowed out, to upper-middle-class parents, the costs of not getting one’s kids on the academic fast track look like dooming them to lifelong economic instability, as opposed to pointing them toward a nondescript but perfectly pleasant middle-class life.
Finally: There’s a simple solution to this problem: a lottery. Set some baseline GPA and SAT criteria, then admit people at random. Tweak the criteria to ensure racial and socioeconomic diversity, if you like. Even favor alums, if you absolutely must. But ditch the whole magical individualized selection process.
Of course, that’s pretty much a nonstarter. The whole point of Harvard admissions is, as Karabel suggests, to get to choose the next generation of elites. A lottery would undermine the very purpose.
[Edited to add link to part three, which doesn’t really affect my overall take. If anything, it further conflates general commercialization of higher ed with the elite admissions frenzy in an unhelpful way. Both important, but not closely related.
Also, ’cause I haven’t done nearly enough promoting — though still planning to — see Craig Tutterow and James Evans’ piece modeling how rankings might drive the development of enrollment management here in our new volume of RSO on “The University Under Pressure“. (Links gated, email me for copies.)]
A recent Atlantic article by Victoria Clayton makes the case that the GRE should be ditched based on some new research. The case for the GRE rests on the following:
- The GRE does actually, if modestly, predict early graduate school grades and you need to do well in courses to get the degree.
- Many other methods of evaluating graduate school applications are garbage. For example, nearly all research on letters of recommendation shows that they do not predict performance.
To reiterate: nobody says GRE scores are perfect predictor. I also believe that their predictive ability is lower for some groups. But the point is not perfection. Th point is that the GRE sorta, kinda works and the alternatives do not work
So what is the new evidence? Actually, the evidence is lame in some cases. For example, Clayton cites a 1997 Cornell study claiming that GRE’s don’t correlate with success. True, but if you actually read the research on GRE’s there have been meta-analyses that compile data from multiple studies and find that the GRE does actually predict performance. This study compiles data from over 1,700 samples and shows that, yes, GRE does predict performance. Sorry, it just does, test haters.
Clayton also cites a Nature column by Miller and Stassun that correctly laments the fact that standardized tests sometimes miss good students, especially minorities. As I pointed out above, no one claims the GRE makes perfect predictions. Only that the correlation is there and that is better than the alternatives that simply don’t predict performance. But at least Miller and Stassun offer a new alternative – in depth interviews. Miller and Stassun cite a study of 67 graduate students at Fisk and Vanderbilt selected via this method and note that their projected (not actual) completion rate is 80% – much better than the typical 50% of most grad programs.
Two comments: 1. I am intrigued. If the results can be replicated in other places, I would be thrilled. But so far, we have one (promising) study of a single program. Let’s see more. 2. I am still not about to ditch GRE’s because I am not persuaded that academia is ready to implement a very intensive in-depth interview admissions system as its primary selection mechanism. The Miller and Stassun column refers to a study of physics graduate students – small numbers. What is realistic for grad programs with many applicants is that you need to screen people for interviews and that screen will include, you guessed it, standardized tests.
Bottom line: The GRE is far from perfect but it is usable. There is no evidence to systematically undermine that claim. Some alternatives don’t work and the new proposed method, in depth interviews, will probably need to be coupled with GREs.
A quick post to announce the publication of “The University Under Pressure“, a new volume of Research in the Sociology of Organizations. Edited by Catherine Paradeise and myself, contributors include Mikaila Mariel Lemonik Arthur, Julien Barrier, Sondra Barringer, Manuelito Biag, Amy Binder, George Breslauer, Daniel Davis, Irwin Feller, Joseph Hermanowicz, James Evans, Ghislaine Filliatreau, Otto Hüther, Daniel Kleinman, George Krücken, Séverine Louvel, Christine Musselin, Robert Osley-Thomas, Richard Scott, Abby Stivers, Pedro Teixeira, and Craig Tutterow. A short description:
Universities are under pressure. All over the world, their resource environment is evolving, demands for accountability have increased, and competition has become more intense. At the same time, emerging countries have become more important in the global system, demographic shifts are changing educational needs, and new technologies threaten, or promise, to disrupt higher education. This volume includes cutting-edge research on the causes and consequences of such pressures on universities as organizations, particularly in the U.S. and Europe. It provides an empirical overview of pressures on universities in the Western world, and insight into what globalization means for universities and also looks at specific changes in the university environment and how organizations have responded. The volume examines changes internal to the university that have followed these pressures, from the evolving role of unions to new pathways followed by students and finally, asks about the future of the university as a public good in light of a transformation of student roles and university identities.
I’m sure I’ll write some more substantive posts about the volume and some of the papers in it — as well as why I think developing an organizational sociology of higher ed is important — in the days to come. But in the meanwhile, take a look. And drop me a line for access to pdfs.
On Twitter, Michigan higher ed prof Julie Posselt compares the quantitative GRE scores for various social science disciplines. Take home message: social sciences are comparable in terms of recruits, but economics has stronger math skills. Take home point #2: there is still a lot of overlap. The bottom third of econ overlaps with other social sciences. This probably reflects that the extraordinarily mathematical approach to econ is a phenomenon of the strong programs that attract those with degrees in physical science and they have pushed the more traditional economics student to the bottom of the distribution.
I’m really bad at keeping up with the media cycle.
So last Wednesday, Vox put up this cute piece with the catchy title, “How Amazon Could Destroy College as We Know It.” Written in the form of a letter from Jeff Bezos to shareholders in the year 2030, it tells the story of how Amazon came to supplant traditional higher education by developing, and selling at cost, badges that people could earn to demonstrate particular skill sets. As the value of badges became evident, companies became more and more interested in using them in hiring—to the detriment, presumably, of traditional indicators like college degrees.
It’s a clever article, and well-written. It also doesn’t quite make the claim the headline implies—that the rise of Amazon badges would destroy higher education. Nevertheless, although I think that the piece gets at something real that is going on, and that is eventually going to be an important source of change, this is not how I see it going down.
Anyway, Wednesday night I started writing a blog post using a similar Bezos-to-shareholders conceit, but from a 2030 that looked quite different. It just wasn’t quite working, I think because it’s hard to see Amazon pioneering the kind of change I can imagine. Pearson, maybe. But even I can’t name the CEO of Pearson. (Apparently it’s John Fallon.)
So the format wasn’t quite working, but the underlying point still nagged. While badges may become a thing, and perhaps Amazon may even pioneer them, they are not going to be “the” new form of educational currency. The world in which “as many as half of major US employers now consider Amazon badges to be one of their top five criteria when determining whom to hire” will remain a fantasy.