Archive for the ‘education’ Category
Econjeff mentions my long standing critique of letters of recommendation (LoRs). Here, I describe my personal experience with them and then I restate the massive empirical research showing that LoRs are worthless.
Personal experience: In graduate school, I had enormous difficulty extracting three letters from faculty. For example, during my first year, when I was unfunded, I asked an instructor, who was very well known in sociology, for a letter. He flat out refused and told me that he didn’t think I’d succeed in this profession. In the middle of graduate school, I applied for an external fellowship and was informed by the institution that my third letter was missing. Repeatedly, I was told, “I will do it.” Never happened. Even on the job market, I had to go with only two letters. A third professor (different than the first two cases) simply refused to do it. Luckily, a sympathetic professor in another program wrote my third letter so I could be employed. Then, oddly, that recalcitrant member submitted a letter after I had gotten my job.
At that point, I had assumed that I was some sort of defective graduate student. Maybe I was just making people upset so they refused to write letters. When I was on the job, I realized that lots and lots of faculty never submit letters. During job searches at Indiana, I saw lots of files with missing letters, perhaps a third were missing at least one letter. Some were missing all letters. It was clear to me that l was not alone. Lots of faculty simply failed to complete their task of evaluating students due to incompetence, malice, or cowardice.
Research: As I grew older, I slowly realized that there are researchers in psychology, education and management dedicated to studying employment practices. Surely, if we demanded all these letters and we tolerated all these poor LoR practices, then surely there must be research showing the system works.
Wrong. With a few exceptions, LoRs are poor instruments for measuring future performance. Details are here, but here’s the summary: As early as 1962, researchers realized LoRs don’t predict performance. Then, in 1993, Aamondt, Bryan and Whitcomb show that LoRs work – but only if they are written in specific ways. The more recent literature refines this – medical school letters don’t predict performance unless the writer mentions very specific things; letter writers aren’t even reliable – their evaluations are all over the place; and even in educational settings, letters seem to have a very small correlation with a *few* outcomes. Also, recent research suggests that LoRs seem to biased against women in that writers are less likely to use “standout language” for women.
The summary from one researcher in the field: “Put another way, if letters were a new psychological test they would not come close to meeting minimum professional criteria (i.e., Standards) for use in decision making (AERA, APA, & NCME, 1999).”
The bottom line is this: Letters are unreliable (they vary too much in their measurements). They draw attention to the wrong things (people judge the status of the letter writer). They rarely focus on the few items that do predict performance (like explicit comparison). They have low correlations with performance and they used codes that bias against women.
A long, long time ago, I used to teach math. One of the central questions in mathematical education at the college level is how to teach mathematical proofs. Sometimes, you had pessimistic conversations. People simply had “mathematical maturity” and there wasn’t much you could do about it. There is truth to this – some people simply can’t grasp what a proof would entail.
Beyond this simple observation, there was remarkably little thinking about how to teach proofs. Of course, there are occasional books that try to break down the process of creating and writing proofs, such as How to Prove It. Still, I felt there was something missing in the conversation about proof teaching. This blog post is my modest contribution to the topic.
My hypothesis: An important barrier to teaching math proofs is that they combine two very, very hard skills and that most math teachers only focus on one of the those skills. Specifically, proofs entail (a) symbolic manipulation and (b) recipes that get you from A to B. Math teachers and books are actually pretty good at (a). For example, almost every text will teach you about the symbols – set theory; formal logic; deltas and epsilons; etc. What is almost completely overlooked is that students find it hard to glimmer the “recipes” that make up proofs and there is no theory, or set of instructional strategies, for helping students intuitively understand recipes. In practice, you simply take courses on various topics (numerical analysis or matrix theory) and you mimic the proofs that people give you. Not great, but better than nothing.
The old Dolciani high school text books had an interesting response to this issue. In the geometry text, the proofs would always have two parts: “analysis” (outline of the idea) and “proof” (traditional proof with all details). You also see this in advanced texts and journal articles. When a long, hard proof is coming up, the author will present an outline.
Here is my modest suggestion: When teaching proofs, always outline the proof as a flow chart. In other words, take the old notion of the proof outline (or “analysis” in Dolcian’s terms), make it visual, and then put it in front of all proofs that require more than a few sentences. By repeatedly visualizing proofs as chains, teachers will be forced to extract the recipe from the text in a way that more students can understand. They will also more easily identify common themes that appear in multiple visualizations of proofs. Also, pictures are easier to remember than dense, equation filled masses of text.
Psychology Today has an article on a new analysis of play and the mental health of young people. The gist is that (a) recently, we let kids have less unstructured play and (b) unstructured play increases the belief that one has direct control over their life, which in turn has positive effect on mental health and various measures of well being. From the article:
The standard measure of sense of control is a questionnaire developed by Julien Rotter in the late 1950s called the Internal-External Locus of Control Scale. The questionnaire consists of 23 pairs of statements. One statement in each pair represents belief in an Internal locus of control (control by the person) and the other represents belief in an External locus of control (control by circumstances outside of the person). The person taking the test must decide which statement in each pair is more true. One pair, for example, is the following:
- (a) I have found that what is going to happen will happen.
- (b) Trusting to fate has never turned out as well for me as making a decision to take a definite course of action.
In this case, choice (a) represents an External locus of control and (b) represents an Internal locus of control.
Many studies over the years have shown that people who score toward the Internal end of Rotter’s scale fare better in life than do those who score toward the External end. They are more likely to get good jobs that they enjoy, take care of their health, and play active roles in their communities—and they are less likely to become anxious or depressed.
In a research study published a few years ago, Twenge and her colleagues analyzed the results of many previous studies that used Rotter’s Scale with young people from 1960 through 2002. They found that over this period average scores shifted dramatically—for children aged 9 to 14 as well as for college students—away from the Internal toward the External end of the scale. In fact, the shift was so great that the average young person in 2002 was more External than were 80% of young people in the 1960s. The rise in Externality on Rotter’s scale over the 42-year period showed the same linear trend as did the rise in depression and anxiety.
[Correction: The locus of control data used by Twenge and her colleagues for children age 9 to 14 came from the Nowicki-Strickland Scale, developed by Bonnie Strickland and Steve Nowicki, not from the Rotter Scale. Their scale is similar to Rotter’s, but modified for use with children.]
It is reasonable to suggest that the rise of Externality (and decline of Internality) is causally related to the rise in anxiety and depression. When people believe that they have little or no control over their fate they become anxious: “Something terrible can happen to me at any time and I will be unable to do anything about it.” When the anxiety and sense of helplessness become too great people become depressed: “There is no use trying; I’m doomed.”
Wow. Later in the article, they talk about how this shift correlates with mental health outcomes. So don’t schedule them or boss the, let them play.
Two weeks ago, we had quite a discussion about critical thinking. There was one strand of debate the baffled me. Thomas wrote the following:
“If I don’t teach critical thinking, then what do I teach?” you ask. “Turns out that there is a simple answer: sociology.” It sounds good at first. But what you’re really saying is that you won’t acknowledge a kind of critique that isn’t sociological. That’s certainly the line Goffman’s defenders are taking. If you think she’s wrong, they say, it’s because you don’t understand how ethnographers think.
That is not what I was saying. What I said is this:
Aside from a very simple general rules of thumb, such as “don’t be emotional in arguing” or “show my your evidence,” the best way to be improve your thinking is to learn from those who have spent a lifetime actually trying to figure out specific problems.
I did not say: “only trust experts.” I said: “learn from.” In other words, there are people who have encountered certain problems and they have tried and tested the ideas that might have occurred to you. In the processes of trying out those ideas, they have probably learned important things about the phenomena that you are looking at. You should probably learn those lessons. Once you have absorbed those ideas, you are free to criticize as you will. No one has a monopoly on truth, even the experts. But that doesn’t mean experts are clueless fools. Certainly, “critical thinking” must have a place for learning from other people.
State defunding of public higher education has received a lot of attention in recent years. And budget cuts like the $250 million one Scott Walker made this year to the University of Wisconsin mean this trend continues to get media play.
Less visible in the media, but still well known, is that as public funding has eroded, colleges have become more dependent on tuition dollars for revenue. For public institutions, this has meant both tuition increases for in-state students and, where possible, a greater percentage of out-of-state and international students. While the net price of college hasn’t increased nearly as much as the sticker price, it’s still beat the cost of inflation year after year.
Both of these narratives are completely true. Yet this story of a shift from public to private funding overlooks one critical factor: the expansion of federal student aid.
During the past two decades, as state appropriations per postsecondary student flattened then declined, federally supported financial aid made massive gains. In 2002 its volume passed that of state appropriations, and by 2010 it was twice as large.
Stunning, right? This suggests a very different story than the one about the privatization of public universities we hear so much about. Instead, it looks like there’s been a shift from state funding of higher ed to federal funding. So what’s going on here?
Well, a couple of things. First, the federal aid figures include both grants and loans. Data sources like the College Board and the Delta Cost Project include loans as part of net tuition, not as federal funding. That makes sense, if you’re interested in the financial burden of college on students and their families. And the loans don’t cost the government anything like their face value.
But counting this way downplays the fact that those loans ultimately exist because the federal government makes them possible. Colleges are doubly dependent in this scenario: on students’ choices about where to attend, but also on the feds to make them available in the first place. And if you’re coming at this from an organizational perspective, we should expect resource dependence — whether on students, on the feds, or both — to have effects.
Second, this chart collapses public, private non-profit, and for-profit institutions together. The state appropriations are only going to publics (which also enroll about three-quarters of the students). But as of 2010, more than a quarter of student aid was going to the 10% of students enrolled in for-profit institutions. Moreover, because private colleges are so much more expensive than public colleges, they also receive a disproportionate fraction of federal loans. I haven’t pulled these numbers apart by institution type. But if we just compared state appropriations and federal aid to students at public institutions, the chart would surely be less dramatic.
It would be misrepresenting reality to say that public institutions have experienced a substantial shift from state to federal dependence (at least without substantially more number crunching). And it would be similarly wrong to argue that schools haven’t become more tuition dependent (since loans do come to schools via individual students).
But you can absolutely make the case that at the field level, higher education has increased its dependence on the federal government relative to state governments. And this makes colleges susceptible to a whole wave of federal demands that simply weren’t there before. The college ratings system Obama proposed and then abandoned is one example of this. Education Secretary Arne Duncan’s drumbeating for accountability is another.
Colleges have a lot of political clout and are well-organized. They ground the ratings proposal into a shadow of its former self. And it will take a lot of doing before we see No College Student Left Behind.
Nevertheless, if organization theory tells us anything, it’s that resource dependence matters. When, five years down the road, we get a Race to the Top rewarding colleges that meet completion and job placement goals at a given tuition cost, I know where I’ll be looking: at that point in 2002 where higher ed waved goodbye to the states and hello to the feds.