Archive for the ‘education’ Category

students evaluations are garbage and so are letters of recommendation – but NOT gre scores, haters!

with 3 comments

For a long time, I believed that student evaluations were valid measures of teaching effectiveness. My belief was based on the following issues.

  • First, there are a fair number of studies that claim a correlation between student evaluations and learning. The critics conveniently overlook this literature.
  • Second, I believed that students can spot a miserable teacher. You don’t need to be steeped in pedagogy theory to see if an instructor is disorganized, or is simply a horrid lecturer.
  • Third, most complaints about student evaluations seemed pretty self-interested. Who complains about evals the most? The professors!* Doesn’t mean they are wrong, but one should examine self-interested claims with some caution.
  • Fourth, critiques of student evaluations of faculty are often couched in bad logic. For example, if an instrument is biased against group X, it doesn’t mean automatically that the instrument is not consistent or valid. It might be the case that the instrument is less valid and consistent for group X, but still points in the right direction. You can only say that student evaluations are “worthless” if the correlation between evals and learning is zero and that is a stubbornly empirical point. Yet, critics in the popular media jump from bias to a lack of validity.

But over time, there have been a parade of better studies that explore the link between outcomes and evaluations and the answer is often null. So what should any seriously interested person do? Wrong answer: Cherry pick studies that confirm one’s belief. Better answer: look for a meta-study that combines data from new and old studies. This fall, Studies in Educational Evaluation published on such meta-study of student evaluations of teacher. Bob Uttl, Carmela White, and Daniela Wong Gonzalez performed such a meta-analysis can come to the following conclusions:

• Students do not learn more from professors with higher student evaluation of teaching (SET) ratings.

• Previus meta-analyses of SET/learning correlations in multisection studies are not interprettable.

• Re-analyses of previous meta-analyses of multisection studies indicate that SET ratings explain at most 1% of variability in measures of student learning.

• New meta-analyses of multisection studies show that SET ratings are unrelated to student learning.

There article is not perfect, but it is enough to make me seriously reconsider my long standing belief in student evaluations. I am very willing to consider that student evaluations are garbage.

However, I want to the reader to be consistent in their intellectual practice. If you believe that student evaluations are bunk, then similar evidence suggests that letters of recommendation are garbage as well. Here is what I wrote two years ago:

I slowly realized that there are researchers in psychology, education and management dedicated to studying employment practices. Surely, if we demanded all these letters and we tolerated all these poor LoR practices, then surely there must be research showing the system works.

Wrong. With a few exceptions, LoRs are poor instruments for measuring future performance. Details are here, but here’s the summary: As early as 1962, researchers realized LoRs don’t predict performance. Then, in 1993, Aamondt, Bryan and Whitcomb show that LoRs work – but only if they are written in specific ways. The more recent literature refines this – medical school letters don’t predict performance unless the writer mentions very specific things; letter writers aren’t even reliable – their evaluations are all over the place; and even in educational settings, letters seem to have a very small correlation with a *few* outcomes. Also, recent research suggests that LoRs seem to biased against women in that writers are less likely to use “standout language” for women.

The summary from one researcher in the field: “Put another way, if letters were a new psychological test they would not come close to meeting minimum professional criteria (i.e., Standards) for use in decision making (AERA, APA, & NCME, 1999).”


If you are the type of person who thinks student evaluations are lousy, then you should also think letters of recommendation are garbage as well. To believe otherwise is simply inconsistency, as the evidence is similar in both cases.

While I am at it, I also want to remind readers that similar analysis shows that standardized tests are actually not bad. When you read the literature on standardized tests, like the GRE, you find that standardized tests and grades are actually correlated – the intended purpose. And I haven’t seen many other meta-analyses that over turn the point.

To summarize: student evaluations and letters of recommendation are bunk, but standardized tests are not.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($4.44 – cheap!!!!)/Theory for the Working Sociologist (discount code: ROJAS – 30% off!!)/From Black Power/Party in the Street / Read Contexts Magazine– It’s Awesome!

*For the record, my evals range from slightly below average to very good. And I’ve actually won multiple teaching awards. So this is not a “sour grapes” issue for me.



Written by fabiorojas

January 5, 2018 at 4:53 am

the PROSPER Act, the price of college, and eroding public goodwill

The current Congress is decidedly cool toward colleges and the students attending them. The House version of the tax bill that just passed eliminates the deduction on student loan interest and taxes graduate student tuition waivers as income. Both House and Senate bills tax the largest college endowments.

Now we have the PROSPER Act, introduced on Friday. The 500-plus page bill does many things. It kills the Department of Education’s ability to keep aid from going to for-profit schools with very high debt-to-income ratios, or to forgive the loans of defrauded student borrowers . It loosens the rules that keep colleges from steering students into questionable loans in exchange for parties, perks, and other kickbacks.

And it changes the student loan program dramatically, ending subsidized direct loans and replacing them with a program (Federal ONE) that looks more like current unsubsidized loans. Borrowing limits go up for undergrads and down for some grads. The terms for income-based repayment get tougher, with higher monthly payments and no forgiveness after 20 years. Public Service Loan Forgiveness, particularly important to law schools, comes to an end. (See Robert Kelchen’s blog for some highlights and his tweetstorm for a blow-by-blow read of the bill.)

To be honest, this could be worse. Although I dislike many of the provisions, given the Republican higher ed agenda there’s nothing shocking or unexpectedly punitive, like the grad tuition tax was.

Still, between the tax bill and this one, Congress has taken some sharp jabs at nonprofit higher ed. This goes along with a dramatic downward turn in Republican opinion of colleges over the last two years.Capture

Obviously, some of this is a culture war. Noah Smith highlights student protests and the politicization of the humanities and social sciences as the reason opinion has deteriorated. I think there are aspects of this that are problems, but the flames have mostly been fanned by those with a preexisting agenda. There just aren’t that many Reed Colleges out there.

I suspect colleges are also losing support, though, for another reason—one that is much less partisan. That is the cost of college.

I think colleges have ignored just how much goodwill has been burned up by the rise in college costs. For the last couple of weeks, I’ve been buried in data about tuition rates, net prices, and student loans. Although intellectually I knew how much prices risen, it was still shocking to realize how different the world of higher ed was in 1980.

The entire cost of college was $7,000 a year. For everything. At a four-year school. At a time when the value of the maximum Pell Grant was over $5,000, and the median household income was not far off from today’s. Seriously, I can’t begin to imagine.

The change has been long and gradual—the metaphorical boiling of the frog. The big rise in private tuitions took place in the 90s, but it wasn’t until after 2000 that costs at publics (both sticker price and net price—the price paid after scholarships and grant aid) increased dramatically. Unsurprisingly, student borrowing increased dramatically along with it. The Obama administration reforms, which expanded Pell Grants and improved loan repayment terms, haven’t meant lower costs for students and their families.

Picture1What I’m positing is that the rising cost of college and the accompanying reliance on student loans have eroded goodwill toward colleges in difficult-to-measure ways. On the one hand, the big drop in public opinion clearly happened in last two years, and is clearly partisan. Democrats have slightly ticked up in their assessment of college at the same time.

But I suspect that even support among Democrats may be weaker than it appears, particularly when it comes to bread-and-butter issues, rather than culture-war issues. Only a small minority (22%) of people think college is affordable, and only 40% think it provides good value for the money. And this is the case despite the growing wage gap between college grads and high school grads. Sympathy for proposals that hit colleges financially—whether that means taxing endowments, taxing tuition waivers, or anything else that looks like it will force colleges to tighten their belts—is likely to be relatively high, even among those friendly to college as an institution.

This is likely worsened by the common pricing strategy that deemphasizes the importance of sticker price and focuses on net price. But the perception, as well as the reality, of affordability matters. Today, even community college tuition ($3500 a year, on average) feels like a burden.

The point isn’t whether college is “worth it” in terms of the long-run income payoff. In a purely economic sense there’s no question it is and will continue to be. But pushing the burden of cost onto individuals and families, rather than distributing it more broadly, makes it feel unbearable, and makes people think colleges are just in it for the money. (Which sometimes they are.) I’m always surprised that my SUNY students think the mission of the university is to make money off of them.

This perception means that students and their families and the larger public will be reluctant to support higher education, whether in the form of direct funding, more financial aid, or the preservation of weird but mission-critical perks, like not taxing tuition waivers.

The PROSPER Act, should it come to fruition, will provide another test for public institutions. Federal borrowing limits for undergraduates will rise by $2,000 a year, to $7,500 for freshmen, $8,500 for sophomores, and $9,500 for juniors and seniors. If public institutions immediately default to expecting students to take out the new maximum in federal loans each year, they will continue to erode goodwill even among those not invested in the culture wars.

The sad thing is, this is a self-reinforcing cycle. Colleges, especially public institutions, may feel like they have no choice but to allow tuition to climb, then try to make up the difference for the lowest-income students. But by adopting this strategy, they undermine their very claim to public support. Letting borrowing continue to climb may solve budget problems in the short run. In the long run, it’s shooting yourself in the foot.



Written by epopp

December 4, 2017 at 3:55 pm

the purpose of teaching sociology

I have been asked to write a short piece on the topic of teaching social theory. My answer will have three parts. On the blog, I will post initial thoughts:

  1. What is the point of teaching sociology at all?
  2. What are the different goals and reasons we teach social theory?
  3. Why did I move to the mechanism oriented approach that I advocate in Theory for the Working Sociologist.

Why teach sociology?

I begin this brief essay with a simple question – why teach sociology? I respond with a simple answer: there are many reasons to teach sociology. We might teach sociology because it is a body of scientific and humanistic knowledge about the social world. Another reason to teach sociology is that we want people to think carefully about their social world. We might also teach simply to broaden a students knowledge. If nothing else, teaching is a variety of public sociology, where we bring our discipline to the public via the classroom.

You can also answer this question in reference to the student. Most students have a tangential interesting in sociology, only a handful will want academic careers. Thus, we teach because we want the public to have basic familiarity with sociology.

If you buy these answers, then it guides you toward a general attitude toward teaching. Most of our time we should focus on the basic facts of the social world and the ideas we can use to analyze them. We should only occasionally used oddball ideas or examples. The instructor should boil down their discipline to core ideas that will be easy to remember and relentlessly focus on them in class. It also draws toward the idea of respect for students. We care what they think and we want them to have a positive view of the field.

What do you think about teaching sociology? Use the comments.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($4.44 – cheap!!!!)/Theory for the Working Sociologist (discount code: ROJAS – 30% off!!)/From Black Power/Party in the Street / Read Contexts Magazine– It’s Awesome!

Written by fabiorojas

December 1, 2017 at 5:01 am

conflict vs. consensus theory: another bogus teaching distinction

In a lot of old textbooks, and a few newer ones as well, you get this argument that social theories can be divided up into theories of conflict vs. theories of consensus. Marx is a conflict guy because his theory revolves around class divisions. Durkheim is consensus guy because he talks about social solidarity.

Sad! Social theory is not built in this way. Usually, consensus and conflict are dependent variables that are explained by other independent variables. Let’s take Marx. Did he *always* claim that there would be class conflict? Nope! Example: the theory of false consciousness describes the conditions under which people do not resist capitalist institutions. So, for Marx, the degree of conflict is a variable that is driven by other things. Same for Durkheim. The degree of social solidarity varies in Durkheim’s theory and is affected by things like urbanization and the division of labor.

The conflict/consensus distinction is not a horrible idea, but it is not one that supported by further examination of sociology’s major theories. It conflates dependent and independent variables.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($4.44 – cheap!!!!)/Theory for the Working Sociologist (discount code: ROJAS – 30% off!!)/From Black Power/Party in the Street

Written by fabiorojas

June 8, 2017 at 12:02 am

contemporary vs. classic social theory: another bogus teaching distinction

A while back, I was telling a friend on the phone about my book, Theory for the Working Sociologist. He asked me about it and I said, “it’s social theory but illustrated with modern research.” He then said, “oh, that could be a book for a contemporary theory theory course.” I mumbled, “sure,” but soon as we were done, I was like, “no, that’s not right.”

In this post, I want to explain why I don’t buy into the “classic v. contemporary” distinction in theory. Let’s start with a statement of what I do and do not argue:

  •  Claim: Breaking up social theory courses into “classic” and “contemporary” is not a great way to teach. It misleads people about the basic nature of sociology and it is not an optimal way to teach *average* undergrads and grad students about how sociology works,
  • Do not claim the following: Sociology/social theory has no historical phases. A historical treatment of theory has no value. The humanities (e.g., close readings of classic texts) has no place in sociology.  Older texts have no value. I reject these claims.

Let me lay out the argument in a number of steps:

  1. The purpose of a social theory course is to teach undergraduates and beginning graduate students “theory,” by which I mean some set of broad applicable ideas that relate to the empirical investigation of society.
  2. The history of social theory and social theory are different things. History of thought is about understanding specific ideas and texts in relation to the biographies of authors and their institutional and historical context. Social theory is a body of thought that motivates thinking throughout sociology. Overlapping? Sure. But theory and history are distinct. For example, a wrong idea can be important for history of thought, but now irrelevant for theory.
  3. Advanced students can learn social theory in any format (historical, mathematical, sign language, you name it!). *Average* students, at the B.A. and Ph.D. level, are confused by historical approaches. By teaching theory in a historical format, most students take away the lesson that “theory” is synonymous with “history.” Nice to know, but not relevant to research.
  4. Historical approaches to theory are sup-optimal for learning because older texts tend to be written in a highly verbose fashion and refer to a lot of things that even modern educated people may not know about. Example: In Weber’s description of bureaucracy, he alludes to Bakunin, the Russian anarchist, as a foil. Just explaining that single reference to undergrads took me about 20 minutes. Now imagine doing that for all of Weber’s references!
  5. Finally, historical approaches make it hard for typical students to transfer what they learned in theory class to another class, and thus undermine the entire purpose of theory class!

Also, I’d add that no other discipline, except philosophy, teaches its core theory in a historical classic/contemporary format. Economists teach it in terms of scope conditions – micro an macro-economic theory. In political science, it is also broken down by topic (“American politics”) – only the political philosophers (“theory” in poli sci) do it by time period, E.g., classic political theory (Greeks) vs. modern (1500 and beyond). Literary theory (“criticism”) gets its own course while historical groupings are used for specific subjects (“early American novels”). Theories of art courses are different than art history courses. The physical sciences pretty much separate all historical scholarship into a few highly specialized courses. You learn proof writing in math either in a proof writing course or in real analysis, which is the modern theory of calculus. The history of math is its own course. The same goes for physics – you learn physical theory (stripped of history) in classical mechanics (not time period – classic means stemming from Newton’s laws; classical mechanics is still a real area of physics) and quantum mechanics. If you really want mechanics the way Newton did it, you can take a course in that. But no one pretends it is teaching you how to do physics in general. You get the modern, better presentation in your basic physics course.

I think that the classical/contemporary approach to teaching theory comes from a desire to be an old style humanist. I suppose there is nothing wrong that. But for most students, this is an incredibly inefficient and misleading way to teach theory. Even if they do learn some of it in your class, I guarantee many will forget everything you said once grades are submitted. Instead, boil down sociology’s main arguments, illustrate them with modern research and move on. If you want to assign my book, that would be great. If not, that’s ok. Just teach social theory, not history.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($4.44 – cheap!!!!)/Theory for the Working Sociologist (discount code: ROJAS – 30% off!!)/From Black Power/Party in the Street 

Written by fabiorojas

June 7, 2017 at 12:01 am

class and college

When awareness about the impact of socio-economic class was not as prevalent among the public, one exercise I did with my undergraduates at elite institutions was to ask them to identify their class background.  Typically, students self-identified as being in the middle class, even when their families’ household incomes/net worth placed them in the upper class.  The NYT recently published this article showing the composition of undergraduate students, unveiling the concentration and dispersion of wealth at various higher education institutions.

As a professor who now teaches at the university listed as #2 in economic mobility (second to Vaughn College of Aeronautics and Technology ), I can testify to the issues that make an uneven playing field among undergraduates.  Unlike college students whose parents can “pink helicopter” on their behalf and cushion any challenges, undergraduates at CCNY are supporting their parents (if alive) and other family members, bearing the brunt of crushing challenges. (In a minority of cases, students’ parents might help out, say, with occasional childcare – but more likely, students are caring for sick family members or helping with younger siblings.)

To make the rent and cover other expenses in a high COL city, CCNY students work part-time and full-time, sometime with up to two jobs, in the low-wage retail sector.  They do so while juggling a full load of classes because their financial aid will not cover taking fewer classes.  For some students, these demands can create a vicious cycle of having to drop out of classes or earning low grades.

I always tell students to let me know of issues that might impact their academic performance. Over the years (and just this semester alone), students have described these challenges:

  • long commutes of up to 2 hours
  • landlord or housing problems
  • homelessness
  • repeated absences from class due to hospitalizations, illness/accidents, or doctor visits for prolonged health problems
  • self-medicating because of fear about high health care costs for a treatable illness
  • anxiety and depression
  • childcare issues (CCNY recently closed its on-campus childcare facility for students), such as a sick child who cannot attend school or daycare that day
  • difficulties navigating bureaucratic systems, particularly understaffed ones
  • inflexible work schedules

These are the tip of the iceberg, as students don’t always share what is happening in their lives and instead, just disappear from class.

For me, such inequalities were graphically summed up by a thank you card sent by a graduating undergraduate.  The writer penned the heartfelt wish that among other things (i.e., good health), that I always have a “full belly.”  Reflecting this concern about access to food, with the help of NYPIRG, CCNY now has a food pantry available to students.

Written by katherinechen

March 22, 2017 at 5:18 pm

global resistance in the neoliberal university

Those of you who are interested in fending off growing neoliberalism in the university might be interested in the following international  line-up at CUNY’s union, PSC.
You can watch a livestream of the conference via fb starting tonight, Fri., March 3, 6-9pm and Sat., March 4, 9:30am-6pm EST:
…an international conference on Global Resistance in the Neoliberal University organized by the union will be held today and tomorrow, 3/3rd-4th at the PSC, 61 Broadway.  
Scholars, activists and students from Mexico, South Africa, Turkey, Greece, India and the US will lead discussions on perspectives, strategies and tactics of resisting the neoliberal offensive in general, and in the context of the university in particular.
You can visit this site for a link to the conference program:
Due to space constraints, conference registration is now closed. But we’re thrilled by the tremendous interest in the event! You can watch a livestream of the conference here:  If you follow us on our Facebook page, you will receive a notification reminding you to watch.  
We look forward to seeing some of you tonight and to discussing the conference with many of you in the near future. 



Written by katherinechen

March 3, 2017 at 11:29 pm