orgtheory.net

Archive for the ‘academia’ Category

i am so sorry, the GRE is still a valid tool in graduate admissions

A recent Atlantic article by Victoria Clayton makes the case that the GRE should be ditched based on some new research. The case for the GRE rests on the following:

  1. The GRE does actually, if modestly, predict early graduate school grades and you need to do well in courses to get the degree.
  2. Many other methods of evaluating graduate school applications  are garbage. For example, nearly all research on letters of recommendation shows that they do not predict performance.

To reiterate: nobody says GRE scores are perfect predictor. I also believe that their predictive ability is lower for some groups. But the point is not perfection. Th point is that the GRE sorta, kinda works and the alternatives do not work

So what is the new evidence? Actually, the evidence is lame in some cases. For example, Clayton cites a 1997 Cornell study claiming that GRE’s don’t correlate with success. True, but if you actually read the research on GRE’s there have been meta-analyses that compile data from multiple studies and find that the GRE does actually predict performance. This study compiles data from over 1,700 samples and shows that, yes, GRE does predict performance. Sorry, it just does, test haters.

Clayton also cites a Nature column by Miller and Stassun that correctly laments the fact that standardized tests sometimes miss good students, especially minorities. As I pointed out above, no one claims the GRE makes perfect predictions. Only that the correlation is there and that is better than the alternatives that simply don’t predict performance. But at least Miller and Stassun offer a new alternative – in depth interviews. Miller and Stassun cite a study of 67 graduate students at Fisk and Vanderbilt selected via this method and note that their projected (not actual) completion rate is 80% – much better than the typical 50% of most grad programs.

Two comments: 1. I am intrigued. If the results can be replicated in other places, I would be thrilled. But so far, we have one (promising) study of a single program. Let’s see more. 2. I am still not about to ditch GRE’s because I am not persuaded that academia is ready to implement a very intensive in-depth interview admissions system as its primary selection mechanism. The Miller and Stassun column refers to a study of physics graduate students – small numbers. What is realistic for grad programs with many applicants is that you need to screen people for interviews and that screen will include, you guessed it, standardized tests.

Bottom line: The GRE is far from perfect but it is usable. There is no evidence to systematically undermine that claim. Some alternatives don’t work and the new proposed method, in depth interviews, will probably need to be coupled with GREs.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

March 16, 2016 at 12:01 am

how to handle great uncertainty – pursuing a masters or PhD

Here on orgtheory, we’ve discussed the many steps of the grad school process and beyond.  One of the most difficult parts is deciding on the first step, whether to attend a program for a masters or PhD.  The anthology Should I Go To Grad School?: 41 Answers to An Impossible Question offers 41 other perspectives on this, from those who entered programs to those who decided on alternative ventures to get what they wanted.

Duncan Watts is one of the authors of the personal essays, a version of which is excerpted here.  In his essay, Watts describes his original trajectory, starting with his service in the Australian military and realizing his drive to learn more about chaos theory.  After being rejected by Oxford but accepted by Cornell, he decided to leave for the US and to enter the great unknown:

I was only 22 at the time, still young enough to make major life decisions without thinking too much about them. But as I hugged my sister and my best friend good-bye that day at the airport, the enormity of what I was doing finally hit me. I’m a big guy — 6-foot-2 and more than 200 pounds — and I thought I’d been through some tough moments during my years in the Navy, but this was more than I had bargained for. I broke down and cried like a baby, right there in front of hundreds of people in the passport line. I remember feeling horribly embarrassed, but I couldn’t help it. I was leaving my whole life behind and I had no idea where I was going, how long I was going for, or what I would do when I got there.

He candidly describes the alienation of graduate school and questioning his decision to pursue a PhD, especially upon realizing that the faculty he wanted to work with had departed for elsewhere:

So here I was in a strange place with no friends, taking courses I didn’t like and teaching courses I didn’t understand, feeling totally overwhelmed and wondering how I had managed to misjudge things so badly. The only reason I didn’t head straight back home was sheer stubbornness and pride.

Luckily for Watts, finding a kindred spirit in a new, young faculty member, who soon became his adviser, was a turning point in his graduate school trajectory.

At the end, he offers three suggestions.  The first of which concerns maintaining a “glass-half-full” outlook and getting the most out of the university’s offerings, including crossing disciplinary boundaries:

First, try to be positive. Grad school may be a time of great uncertainty about the future, but it is also a time when in the present you don’t have a whole lot of responsibility, you do have a lot of time to think and learn, and you are surrounded by an incredible depth and diversity of knowledge. There were academic programs at Cornell that I’d never even heard of until I got there, and once I got my feet on the ground, I started auditing classes in completely unrelated programs, such as philosophy and political science, just because I could. …

The second suggestion is about finding a supportive mentor who can help develop “a life of the mind”:

Second, find the right adviser. Unless you meet your future spouse in grad school (which actually is not all that unlikely), the single most important relationship you are likely to have is with your adviser. So it’s important, both to your success and even more so to your happiness, to find an adviser who is a good fit for your needs. In part that means that he or she knows enough about your intended field to guide your research. But learning how to do good research is about more than just mastering certain skills or domains of knowledge. It’s also about learning how to ask an interesting question, which after a lifetime of answering questions that have been handed to you suddenly requires you to think in a whole new way. You’re going to need some help with that.

Watts’ third suggestion is finding one’s drive through research questions:

And finally, find your question. The best piece of advice that Steve [Watts’ adviser] ever gave me was that I would know I had found my question when I found myself unable to articulate why it was so interesting to me. That I could go over all the usual reasons (interesting math, practical applications, etc.), but in the end it would be something else, something that — as Steve’s mentor, the great mathematical biologist Art Winfree, told Steve — “irrationally grips you by the imagination.” …

Many of us have similar stories of the ups and downs of navigating the academy, with long quests that are not readily apparent in our official bios.   However, most of us don’t have the same trajectory as Watts – in fact, Watts describes the role of chance in getting his co-authored paper into Nature, followed by a tenured position at Columbia, and he explicitly warns that aspirants only see the “winners” in a winner-take-all system:

…the odds of having a wildly successful academic career after going to grad school are about the same as the odds of becoming a movie star after studying acting. This reality is one of those obvious numerical facts that somehow many ambitious students miss until it’s too late. Because you spend most of your academic career studying the ideas of famous people and reading highly cited papers, it’s easy to fall into the trap of thinking that those are the norm, when in fact the overwhelming majority of papers receive very few citations and don’t end up in anyone’s curriculum.

This cautionary note seems to suggest that academics will have to value various outcomes (for instance, an academic career with more effort spent on teaching and mentoring students?  working with the community?) – or decide whether the uncertainty is too great to bear and pursue another path.

Written by katherinechen

March 10, 2016 at 5:42 pm

Posted in academia, books

poll: right of first refusal?

Today’s debate topic: In a world of multiple submission, should flagship journals have first dibs on articles submitted to them and to other journals? Please use the comments.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

March 9, 2016 at 12:01 am

draft of a petition for multiple submission

A few weeks ago, I suggested that we could speed up reviews by having “multiple submission” in sociology. Submit to as many journals as you want, when you want. Here is a draft of a petition for the ASA:

“Academic journals are central to the discipline of sociology. Our journals disseminate research and direct debate. The modern journal system in sociology has a draw back: it can take years for articles to be accepted for publication. Authors submit manuscripts to a journal and wait months, sometimes years, for a reply. Some journals prefer a system of extensive revision so that years can pass between initial submission and final decision.

“We petition the ASA to institute ‘multiple submission’ as a policy for its journals. Modeled on the journal system in some other disciplines, the idea is simple. An author may submit to as many ASA journals as desired. Editors then manage peer review in the way they see fit. Upon receiving reviews, they offer an invitation for revision or acceptance for publication. If an author receives more than one offer of acceptance, they will contact other journals to determine if other editors are interested in the manuscript.

“The ASA has a flagship journal, the American Sociological Review. We recognize that the ASA has an interest in highlighting the most compelling research in this venue. We suggest that the American Sociological Review retain a ‘right of first refusal.’ If a paper is submitted to the ASR and any other ASA journal, then the author has an obligation to inform the editors of the ASR and give them the option to publish the paper.

“Multiple submission exists in the world of law journals and a modified form exists in many physical sciences. It works – publishable articles can be accepted within a year, or less, rather than multiple years. This proposal is simple and easy to implement. Thank you for your consideration.”

Purpose: Keep proposal simple and direct. Add your revisions in the comments.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

 

Written by fabiorojas

March 8, 2016 at 2:56 am

visualizing the quant-qual divide in sociology

quant qual pic

The blog CWTS has a post that uses bibliometric data to map how quantitative and qualitative research segregates in sociology:

Perhaps we can conclude from this small foray into the quantitative-qualitative divide that particular research topics are often confined to one method. A qualitative approach yields a richer, thicker description, and embeds an analysis in a wider context. At the same time, it may trigger questions that require a more quantitative answer, which in turn may require again a more qualitative analysis. We may thus continuously switch between qualitative and quantitative methods. Rather than trying to integrate the two, which is for example promoted under the heading of mixed methods, we should perhaps mostly keep challenging both views from the other perspective. We should not be blind to the challenges posed by the other perspective, but accept that the other perspective can supplement and nuance our conclusions, rather than invalidate them. Fortunately, when looking at the distribution of publications in journals, some of the more general journals, such as American Sociological Review and American Journal of Sociology do include publications from both perspectives (although the quantitative perspective seems more present). More specialised journals, such as Cultural Sociology and Social Forces, mainly focus on respectively qualitative and quantitative research. At least, there are some common fora for discussion, but there is room for improvement.

Recommended.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street 

Written by fabiorojas

February 25, 2016 at 12:01 am

a simple way to speed up the journal review process

One of the main problems of scholarly publishing is that it takes too long. One reason is that there is simply no competition for articles. Since journals have a “single submission” policy, you are giving each journal a monopoly right to your publishing until they have the time to reject your paper. Not surprisingly, journals treat you like garbage. Personally, I once had an editor lose a paper twice (!), papers rejected after editors appoint entirely new reviewers, endless demands for revision from editors who won’t structure the revision process, and so forth. I can  take rejections, but don’t string me around. You are wasting my time if there isn’t a decent chance you will take my paper.

I think there is one simple way to improve the journal process: abolish the “single submission” rule. In other words, submit the paper  to as many journals you want at the same time until you get an accept. If multiple journals offer to publish, then you choose where it goes.

A while back I suggested this and people responded in a negative manner. But consider this:

  • You can submit applications to as many colleges as you want.
  • You can submit applications to as many grad programs you want.
  • You can submit applications for as many jobs as you want.
  • You can submit funding applications to as many agencies as you want.
  • You can submit book manuscripts to as many publishers as you want.

And they all work. The system does not break down. In all cases, you simply negotiate multiple offers if they happen and choose one.

The only reasonable objection is that reviewers may be overloaded. That is a response with a simple answer. If a reviewer is asked to review a paper more than once, they either decline the review (clicking a link in an email) or clip and paste the old review. If you are worried that journal editors will be overloaded, you can add some simple restrictions to lighten the load. For example, authors can only submit one paper every six months. Or if a paper is accepted, you can’t submit for another six months. Perhaps the simplest solution is a submission fee, which is used in many fields. In other words, reviewer overload is easy to solve. Journal overload is also quite solvable. There is no reason to maintain single submission and its abolition will quickly improve academic journal publishing.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

February 24, 2016 at 12:01 am

how scientists can help us avoid the next flint

In a story full of neglect and willful ignorance, there are a few heroes. One is Mona Hanna-Attisha, the Flint pediatrician and Michigan State professor who raised the alarm with data on kids’ blood-lead levels from the local hospital. Another is Marc Edwards, the Virginia Tech environmental engineer who took on the Michigan Department of Environmental Quality after a Flint resident sent him a lead-rich water sample for testing.

Hanna-Attisha and Edwards provide shining examples of how academics can use science to hold the powers-that-be accountable and make meaningful change.

Taking on the status quo is hard. But as Edwards discusses in the Chronicle, it’s becoming ever-harder to do that from within universities:

I am very concerned about the culture of academia in this country and the perverse incentives that are given to young faculty. The pressures to get funding are just extraordinary. We’re all on this hedonistic treadmill — pursuing funding, pursuing fame, pursuing h-index — and the idea of science as a public good is being lost….What faculty person out there is going to take on their state, the Michigan Department of Environmental Quality, and the U.S. Environmental Protection Agency?…When was the last time you heard anyone in academia publicly criticize a funding agency, no matter how outrageous their behavior? We just don’t do these things….Everyone’s invested in just cranking out more crap papers.

When faculty defend academic freedom, tenure is often the focus. And certainly tenure provides one kind of protection for scientists like Hanna-Attisha (though she doesn’t yet have it) or Edwards who want to piss off the powerful.

But as this interview — and you should really read the whole thing — makes clear, tenure isn’t the only element of the academic ecosystem that allows people to speak out. Scientists can’t do their work without research funding, or access to data. When funders have interests — whether directly economic, as when oil and gas companies fund research on the environmental impacts of fracking, or more organizational, as when environmental agencies just don’t want to rock the boat — that affects what scientists can do.

So in addition to tenure, a funding ecosystem that includes multiple potential sources and that excludes the most egregiously self-interested will encourage independent science.

But beyond that, we need to defend strong professional cultures. Hanna-Attisha emphasizes how the values of medicine both motivated her (“[T]his is what matters. This is what we do … This is why we’re here”) and prompted her boss’s support (“Kids’ health comes first”), despite the “politically messy situation” that might have encouraged the hospital’s silence. Edwards lectures his colleagues about “their obligation as civil engineers to protect the public” and says, “I didn’t get in this field to stand by and let science be used to poison little kids.”

Intense economic pressures, though, make it hard to protect such this kind of idealism. As market and financial logics come to dominate institutions like hospitals and universities, professional values gradually erode. It takes a concerted effort to defend them when everything else encourages you to keep your head down and leave well enough alone.

Promoting academic independence isn’t without its downsides. Scientists can become solipsistic, valuing internal status over real-world impact and complacently expecting government support as their due. The balance between preserving a robust and independent academic sector and ensuring scientists remain accountable to the public is a delicate one.

But if I have to choose between two risks—that science might be a bit insular and too focused on internal incentives, or that the only supporters of science have a one-sided interest in how the results turn out—I’ll take the first one every time.

Written by epopp

February 12, 2016 at 4:54 pm

Follow

Get every new post delivered to your Inbox.

Join 3,648 other followers