Archive for the ‘productivity and performance’ Category
A recent Atlantic article by Victoria Clayton makes the case that the GRE should be ditched based on some new research. The case for the GRE rests on the following:
- The GRE does actually, if modestly, predict early graduate school grades and you need to do well in courses to get the degree.
- Many other methods of evaluating graduate school applications are garbage. For example, nearly all research on letters of recommendation shows that they do not predict performance.
To reiterate: nobody says GRE scores are perfect predictor. I also believe that their predictive ability is lower for some groups. But the point is not perfection. Th point is that the GRE sorta, kinda works and the alternatives do not work
So what is the new evidence? Actually, the evidence is lame in some cases. For example, Clayton cites a 1997 Cornell study claiming that GRE’s don’t correlate with success. True, but if you actually read the research on GRE’s there have been meta-analyses that compile data from multiple studies and find that the GRE does actually predict performance. This study compiles data from over 1,700 samples and shows that, yes, GRE does predict performance. Sorry, it just does, test haters.
Clayton also cites a Nature column by Miller and Stassun that correctly laments the fact that standardized tests sometimes miss good students, especially minorities. As I pointed out above, no one claims the GRE makes perfect predictions. Only that the correlation is there and that is better than the alternatives that simply don’t predict performance. But at least Miller and Stassun offer a new alternative – in depth interviews. Miller and Stassun cite a study of 67 graduate students at Fisk and Vanderbilt selected via this method and note that their projected (not actual) completion rate is 80% – much better than the typical 50% of most grad programs.
Two comments: 1. I am intrigued. If the results can be replicated in other places, I would be thrilled. But so far, we have one (promising) study of a single program. Let’s see more. 2. I am still not about to ditch GRE’s because I am not persuaded that academia is ready to implement a very intensive in-depth interview admissions system as its primary selection mechanism. The Miller and Stassun column refers to a study of physics graduate students – small numbers. What is realistic for grad programs with many applicants is that you need to screen people for interviews and that screen will include, you guessed it, standardized tests.
Bottom line: The GRE is far from perfect but it is usable. There is no evidence to systematically undermine that claim. Some alternatives don’t work and the new proposed method, in depth interviews, will probably need to be coupled with GREs.
Econjeff mentions my long standing critique of letters of recommendation (LoRs). Here, I describe my personal experience with them and then I restate the massive empirical research showing that LoRs are worthless.
Personal experience: In graduate school, I had enormous difficulty extracting three letters from faculty. For example, during my first year, when I was unfunded, I asked an instructor, who was very well known in sociology, for a letter. He flat out refused and told me that he didn’t think I’d succeed in this profession. In the middle of graduate school, I applied for an external fellowship and was informed by the institution that my third letter was missing. Repeatedly, I was told, “I will do it.” Never happened. Even on the job market, I had to go with only two letters. A third professor (different than the first two cases) simply refused to do it. Luckily, a sympathetic professor in another program wrote my third letter so I could be employed. Then, oddly, that recalcitrant member submitted a letter after I had gotten my job.
At that point, I had assumed that I was some sort of defective graduate student. Maybe I was just making people upset so they refused to write letters. When I was on the job, I realized that lots and lots of faculty never submit letters. During job searches at Indiana, I saw lots of files with missing letters, perhaps a third were missing at least one letter. Some were missing all letters. It was clear to me that l was not alone. Lots of faculty simply failed to complete their task of evaluating students due to incompetence, malice, or cowardice.
Research: As I grew older, I slowly realized that there are researchers in psychology, education and management dedicated to studying employment practices. Surely, if we demanded all these letters and we tolerated all these poor LoR practices, then surely there must be research showing the system works.
Wrong. With a few exceptions, LoRs are poor instruments for measuring future performance. Details are here, but here’s the summary: As early as 1962, researchers realized LoRs don’t predict performance. Then, in 1993, Aamondt, Bryan and Whitcomb show that LoRs work – but only if they are written in specific ways. The more recent literature refines this – medical school letters don’t predict performance unless the writer mentions very specific things; letter writers aren’t even reliable – their evaluations are all over the place; and even in educational settings, letters seem to have a very small correlation with a *few* outcomes. Also, recent research suggests that LoRs seem to biased against women in that writers are less likely to use “standout language” for women.
The summary from one researcher in the field: “Put another way, if letters were a new psychological test they would not come close to meeting minimum professional criteria (i.e., Standards) for use in decision making (AERA, APA, & NCME, 1999).”
The bottom line is this: Letters are unreliable (they vary too much in their measurements). They draw attention to the wrong things (people judge the status of the letter writer). They rarely focus on the few items that do predict performance (like explicit comparison). They have low correlations with performance and they used codes that bias against women.
I’ve always known that some city projects are simply bad deals, like sport stadiums. What I didn’t know is that there is new research showing that maga-projects of all types are a giant rip-off. Bent Flyvberg of Oxford discusses this finding a new Econ Talk podcast. What Flyvberg did was collect data on “mega-projects” – construction efforts that cost at least a billion dollars and affects a million people. What was found is that 90% of the time, mega projects are over budget, not completed on time, or do not attract the customers that were predicted (i.e., the demand was wildly over estimated). This applies to both private and public sector projects. Flyvberg also reports that smaller projects tend to do much better, for a variety of reasons.
A few comments here:
- This is pretty strong evidence that states should completely avoid the expensive mega-sports projects like stadiums above a certain size. The Olympics, for example, should only be hosted in nations that have preexisting facilities.
- Flyvberg points out that mega projects can even destroy the engineers and other professionals who build them. The architect of the Sydney opera only did one building in his life. The cost-overruns and delays ruined his reputation.
- It is mainly state actors, contractors, and land owners who receive benefits.
- I recently saw the Church of the Sagrada Familia in Barcelona. Big project, but built little by little over a 100 years. A better model?
- There are some cases of success, but they seem hard to predict ex-ante.
Bottom line: The next time they tell you that we need this multi-billion dollar road, just say no.
Plummeting grant funding rates are back in the news, this time in the U.K., where success rates in the Economic and Social Research Council—a rough equivalent to NSF’s SBE division—have dropped to 13%. In sociology, it’s even lower—only 8% of applications were funded in 2014-15.
I’ve written before about the waste of resources associated with low funding rates. But this latest round prompted me to do some back-of-the-envelope calculations. Disclaimer: these numbers are total guesses based on my experience in the U.S. system. I think they are pretty conservative. But I would love to see more formal estimates.
Mikaila Mariel Lemonik Arthur is an associate professor of sociology at Rhode Island College. She is the author of Student Activism and Curricular Change in Higher Education.
My state, Rhode Island, is in the process of beginning an experiment with performance funding for public higher education. Because of our small size, we have only three institutions of public higher education: The University (URI), The College (RIC—my institution), and The Community College (CCRI), and thus our performance funding initiative cannot involve comparative metrics or those based on what “the top” institution in the state is doing. Therefore, the legislature instead decided to craft a performance funding formula based on their own goals for higher education outcomes. The current version of the bill—considerably improved from prior versions, due in large part to the concerted efforts of my colleagues who testified before the relevant state House and Senate committees—includes among its metrics the 100% and 150% of normative time graduation rates; the production of degrees tied to “high demand, high wage” employment opportunities in our (very small) state; and an additional measure to be decided by each of the three institutions in consultation with internal constituencies; with the potential to adjust the weights of these measures to reflect institutional missions, student body demographics, and “the economic needs of the state.”
But this is not a post about performance funding, at least not really. Rather, it is a post about what “success” means for colleges and university.
Critics charge that Simchowitz often preys on vulnerable young artists without gallery representation — some say without talent — and buys up huge quantities of their work, then flips the pieces back and forth at escalating prices among a cultivated group of buyers: a network of movie stars, professional poker players, orthodontists, nightclub promoters, financiers, football players and corned-beef magnates, many of whom hold Simchowitz in such high esteem that they’re willing to purchase the pieces he acquires for them sight unseen, artist unnamed. In March, in an online screed for New York magazine, the art critic Jerry Saltz tore into Simchowitz with unusual ferocity, dubbing him a “Sith Lord” and the Pied Piper of the “New Cynicism.” Simchowitz’s artists may enjoy a temporary surge in prices, his critics argue, but they typically see little of the upside; in any case, or so the story goes, once their bubbles pop, they’re left for dead.
Many important galleries have blacklisted Simchowitz as a buyer, forcing him to take extreme measures to secure desired work, including using consultants as undercover mules. Simchowitz told me about a recent scheme in which he had a consultant buy three pieces from Essex Street, a Lower East Side gallery. The purchase was nominally on behalf of another client, but the ultimate recipient was Simchowitz; by the time the gallery suspected the ruse, money had already changed hands, but the pieces had not been delivered. The gallery requested that Simchowitz not only cancel the purchase but also return another piece by the same artist that was already in his possession, which he did. Moreover, the gallerist, furious over what happened, called the other client to inform him that he was colluding in fraud, an accusation that heartily amused Simchowitz. (Asked for comment, the gallery responded, “Essex Street has never done business with Stefan Simchowitz.”)
I am a lot less alarmed by Simchowitz and even delight in his irreverence and trash talk. Aside from the trash talk, one reason that he draws controversy is his embrace of the market side of art. He thinks “flipping,” which is just another word for “quickly selling at a profit,” is great and has argued that it indicates strength of an artist. Another source of the controversy is patronage of young artists. One could argue that being so dependent on one person creates too much risk.
I tend to think these arguments, for the most part, are misplaced, or overblown. For example, almost all relationships in the art world have ups and downs. People can get traditional gallery representation and then have stalled careers. There are routinely lawsuits filed against galleries because of shady business deals. Artists can get burned as well. Private “dealers” like Simchowitz have no monopoly on good, or bad, business decisions.
If an artist strives to be in the “right” collections, Simchowitz, and flipping in general, may not be optimal, but he’s still preferable to not having a career at all. He’s probably equivalent to having a good, but not elite, dealer behind you.
On a deeper level, I have to give Simchowitz a thumbs up simply because he puts his money where his mouth is. If he likes it, he pays in cash. The trade off is that you are locked in. But, so what? That’s a standard way to reduce uncertainty. Also, he’s like a good business manager in that he provides personal support to help younger people who may not know how to deal with the business side of things. And worse comes to worse, if you hate him, it seems like you can “paint your way out of it,” in the same way a musician can finish a contract by just chunking out the last few records.
He may be crass and direct, and he may embrace practices the art world deems inappropriate. At the end of the day, there’s some artists who stayed in the game and succeeded because he gave them a room and ten thousand bucks. I wonder if he’ll take my calls?
Puzzle for you all your org behavior/theory of the firm types: Why does Sears still exist? Normally, when we think about why successful firms exist, we think that they have a unique product, fill a niche, or have some sort of incumbent advantage. Or just overly aggressive management. But in a world of Amazon, Target, and Home Depot, it’s difficult to understand what Sears does well. Why aren’t they gobbled up by Lowes or Home Depot? Or dismembered by selling off its prime real estate holdings?
You see the occasional article on the topic. Forbes in 2011 claimed that it was generating good cash flow, which owners used to fund purchases of other firms. But the question is – where is the cash flow coming from? Yahoo readers claim that Sears is less a retailer and more of a holding company for a few brands (like Diehard batteries or Land’s End clothes) that just happen to be sold in Sears big box stores. But you’d think that competition would make this set up hard to sustain. The Forbes article does note a massive drop in sales… yet wiki reports that 793 (!) stores were open in 2014. Please use the comments.