an inconvenient truth about GRE scores
There’s a lot of skepticism about GRE’s and their role in graduate admissions. The folk wisdom seems to be that GRE’s aren’t terribly useful. It turns out that there is actually extensive research investigating whether GRE scores actually do predict performance in graduate school. Check out this 2001 article from Psychological Bulletin. It’s called “A Comprehensive Meta-Analysis of the Predictive Validity of the Graduate Record Examinations: Implications for Graduate Student Selection and Performance.” Here’s the abstract:
This meta-analysis examined the validity of the Graduate Record Examinations (GRE) and undergraduate grade point average (UGPA) as predictors of graduate school performance. The study included samples from multiple disciplines, considered different criterion measures, and corrected for statistical artifacts. Data from 1,753 independent samples were included in the meta-analysis, yielding 6,589 correlations for 8 different criteria and 82,659 graduate students. The results indicated that the GRE and UGPA are generalizably valid predictors of graduate grade point average, Ist-year graduate grade point average, comprehensive examination scores, publication citation counts, and faculty ratings. GRE correlations with degree attainment and research productivity were consistently positive; however, some lower 90% credibility intervals included 0. Subject Tests tended to be better predictors than the Verbal, Quantitative, and Analytical tests.
Wow. GRE’s aren’t perfect, but they sure seem to do the job. With respect to graduate admissions, this basically says “go with GRE scores and grades unless there is strong credible signalling.” Any other interpretations? Any GRE skeptics willing to defend their position?
orgtheory.net 
Over the range of GRE scores of students applying to an elite program, I bet it does substantially worse.
LikeLiked by 1 person
yes
May 9, 2011 at 2:39 am
@Yes: This study in economics shows that, once again, GRE’s aren’t perfect, but they do predict job placement among elite grad students.
Click to access admission.pdf
The published version is in the Journal of Economic Education.
LikeLike
fabiorojas
May 9, 2011 at 3:15 am
http://ubiquity.acm.org/article.cfm?id=1071921
LikeLike
Randy
May 9, 2011 at 3:25 am
Randy: “Taken as a whole, the evidence suggests that there is some correlation between GRE scores and graduate achievement. But there is widespread disagreement about the degree of correlation.”
So even that article admits the correlation is there. Seriously, read the article I cited, which is more comprehensive than just about anything else and tell me what you think.
LikeLike
fabiorojas
May 9, 2011 at 3:28 am
Asserting that GRE scores have zero predictive value for graduate school success feels to me like an academic counterpart to birtherism, so I’m not sure what good citing studies will accomplish. Because it’s an issue laden with so much ideological and personal baggage, people who complain about how their intro sociology students can’t understand the difference between anecdotes and statistical population patterns will revert back to “Auguste Comte had low GRE scores, so there” or whatever else to support their position.
That said, I suspect the value of GRE scores declines steadily the more faculty-hours a department devotes to evaluating files, especially in terms of increasing the total number of faculty involved.
LikeLike
jeremy
May 9, 2011 at 3:53 am
@Jeremy: As social scientists, we should strive to base our decisions on the best available knowledge, even if we don’t feel comfortable. Also, as teachers, we should show people the evidence behind our reasoning. Citing these studies is what we should be doing.
And yes, the more we read files, the better. But also, there is only so much in each file – a GRE report, transcript, essays, letters and a writing sample (sometimes). And as far as I can tell, the evidence is that GRE’s and GPA is the most reliable part of the package. That doesn’t absolve us from reading because the correlation is good, not perfect.
LikeLike
fabiorojas
May 9, 2011 at 4:11 am
“people who complain about how their intro sociology students can’t understand the difference between anecdotes and statistical population patterns will revert back to “Auguste Comte had low GRE scores, so there” or whatever else to support their position.”
You need a little more venom on that dart?
I do not remember mentioning anything about the GRE specifically, and I did not do badly at it. However, the truth being pointed out here seems inconvenient to Fabio and Jeremy Freese: the parameters they defend may indeed predict success in school (if success is measured solely in grades and time to completion, that is). But is that all there is to it?
The point being made by several people in the other post is that school (even graduate school) is just training for real life; it’s not real life itself. So yes, both anecdotically (qualitatively, some sociology professors would say) and statistically, the people who end up having the highest achievements in life are not those who have the highest grades in school. The covariance between the two is so low it could be zero. So Comte and Durkheim and the rest can continue to give the finger to the GRE. Thanks.
LikeLiked by 1 person
Guillermo
May 9, 2011 at 4:25 am
At my low-tier PhD-granting sociology department you can tell that GREs matter, albeit in a nonlinear way. Below 1100, it is very unlikely the student will be particularly good, and very likely they will not finish.
But my own limited experience suggests that the benefits of high GRE scores trail off above 1200 or so (meaning that a student whose GREs are 1400 is not much more likely to be good than is a student whose GREs are 1200). You can get students who have very high GREs but get wrapped up in very narrow agendas.
LikeLiked by 1 person
eg
May 9, 2011 at 4:27 am
“But my own limited experience suggests that the benefits of high GRE scores trail off above 1200 or so…”
Yep. It would be an interesting exercise if someone would introduce selectivity controls in the analysis. Does it really make that much difference beyond a certain threshold?
But here is something I know about standardized tests: someone who became famous for mailing bombs to people did very well at them (he got admitted to Harvard and was made a professor at Berkeley).
LikeLiked by 1 person
Guillermo
May 9, 2011 at 4:39 am
And if there is still any doubt of the assertions made above, here is a list of people rejected by Harvard:
“Investor Warren Buffet, Sen. John Kerry, D-Mass, Rolling Stone magazine founder Jann Wenner, NBC “Today” show host Meredith Vieira, former “NBC Nightly News” anchor Tom Brokaw, New Yorker magazine editor David Remnick, CNN founder Ted Turner, folk rock legend Art Garfunkel, Matt Groening, creator of the animated television series “The Simpsons,” Sun Microsystems chairman Scott McNealy, Memorial Sloan-Kettering Cancer Center president Harold Varmus, and Columbia University President Lee Bollinger round out the list.”
http://abcnews.go.com/US/story?id=3201481&page=1
LikeLike
Guillermo
May 9, 2011 at 4:55 am
Guillermo, no one ever said that GRE’s predict all life success. It’s not a test of how rich or successful you will become outside of academia.
However, they do seem to predict well many of the things we want from graduate students: grades, degree completion, and subsequent publication. It’s not perfect and nobody ever said it was.
Furthermore, no one ever said that graduate admissions should be based ONLY on GRE’s or GPA’s. In my original post, I listed a number of other factors I look for in applicants. Listing the occasional false positive/negative does not deny the overall pattern.
LikeLike
fabiorojas
May 9, 2011 at 4:59 am
@Fabio: Don’t get me wrong, I appreciate your effort to introduce actual evidence into the matter.
Then again, I’m not sure what all this fuzzy correlational math amounts to in the face of a list of famous people–Meredith Vieira! Art Garfunkel!–who didn’t get into Harvard.
LikeLike
Jeremy
May 9, 2011 at 5:26 am
“Guillermo, no one ever said that GRE’s predict all life success.”
Then why place so much emphasis on it, especially when your goal is, I assume, to train people to be successful later in life?
“However, they do seem to predict well many of the things we want from graduate students: grades, degree completion, and subsequent publication.”
Absolutely no one present is disputing this*. The reflection one is trying to get across is that your measures for selecting students most likely have little to do with selecting people who will become future successful researchers. This is my original point in the previous post: your system attracts a lot of future research assistants (this is why even “elite” university departments are filled with them) and rejects a lot of people who could become actual successful researchers.
* It must be said, however, that data in your cited study refer to publications *as a student*; they do not include data on “subsequent” (postdoctoral) publications. Furthermore, the authors acknowledge that the correlation observed for this variable is small (0.2, I would say laughable), and can be attributed to randomness *except in the case of subject tests* (“0” intervals).
LikeLike
Guillermo
May 9, 2011 at 5:33 am
Two questions:
1. Any evidence concerning country of origin? Say, are scores for Chinese students more or less indicative to future success?
2. Some time ago, some economists (Mankiw?) claimed that econ grad students have the higher GRE score than sociology grad students, so it’s likely that economists are better than other social scientists. What do you think?
LikeLiked by 1 person
passerby
May 9, 2011 at 5:57 am
This conversation just keeps getting better and better (btw, this is one of the reasons I read orgtheory)
I’d like to hear people’s views on the following things and how you think they impact the admissions process:
1. Community colleges. Does it make any difference if an applicant went to a community college before going to a 4 year school? When you look at the transcripts, does an A at an intro class at an elite institution count count in the same way as an intro class at a community college?
2. International students. I’m interested in both international students who graduated from US institutions and from schools outside the US. Does it make a difference what country they’re coming from?
3. the names of the faculty who wrote the letters of recommendation.
3.1 Does it make a difference if the letter comes from say an assistant professor vs. a full professor?
3.2 Does it matter if that professor is a well known sociologist? And if that is the case, how do you balance this when you evaluate an application from someone with a different background?
4. Finally, once students are admitted, is it possible that their application materials will have an impact on how they get treated by faculty in the department?
LikeLike
Diego
May 9, 2011 at 6:00 am
I find this thread rather puzzling.
Sure, if I have a high vs. a low GRE score, I would know who to pick. But the issue at hand is how a medium GRE score with tough classes, excellent research papers, independent reflections vs. high GRE score and no additional bonuses compare.
So, what can the above cited study tell us about this decision?
To put it another way: As Fabio already indicates with his “unless there is credible signalling” – there is of course going to be credible signalling, why else look at a low score?.
LikeLike
Carsten Bergenholtz
May 9, 2011 at 6:09 am
“Meredith Vieira”. You don’t get to be the anchor of two of the most viewed TV programs in the US by accident.
“Art Garfunkel”. Yes, an inductee in the Rock and Roll Hall of Fame, next to Queen, Pink Floyd, R.E.M., The Doors…
LikeLike
Guillermo
May 9, 2011 at 6:34 am
The lack of maths in Sociology (IMO) hamstrings students when it comes to the quant. on a GRE.
LikeLike
Jon
May 9, 2011 at 7:07 am
@Fabio: Thanks for posting this. It’s good to see some evidence on a topic rife with anecdotal impressions, including my own.
Why is the GRE correlated with success in graduate school? What do you think the latent traits that the GRE measures are that are also correlated with success? Logic? The willingness to study hard to do well? Thinking clearly and correctly under pressure?
LikeLike
Brian
May 9, 2011 at 7:32 am
@Brian
Honestly, (and I think this was said before), I think it demonstrates a willingness to jump through yet another hoop in the application process. If you can spend the weeks or months preparing (which many of us need to do), then you’re able to compete for the prize. It’s a reflection of dedication – to playing the game though, not the discipline. If success is measured by completion, then sadly, why wouldn’t this be a good predictor?
Regarding myself: I can painfully accept that preparing for yet another standardized test is something I have to do, even if I can’t feel out the direct relationship it holds on my discipline. What’s tragic in a sense is my complete lack of motivation to do so. Honestly, I have the drive to read about Bayesian statistics, research methods, and my RSS feed of recent publications in topical literature, all independently (and I frequently do) because it’s interesting and beneficial in the long run regarding my focus. Studying geometry, algebra, and memorizing word lists doesn’t feel like it’s going to help me beyond acceptance. GRE prep in the short term feels like a poor way to budget time, in the long run it isn’t.
The question I would ask, similar to the dissent featured here by other posters opposed to the use of it, is if there’s a better predictor that is related to the discipline. I was thinking a portfolio of poster presentations and papers would be worthwhile – it’s just the time investment required by faculty (as mentioned in the prior thread) would increase.
LikeLike
Mike
May 9, 2011 at 1:04 pm
Art Garfunkel has an M.A. in Mathematics from Columbia.
LikeLike
Kieran
May 9, 2011 at 1:09 pm
I fondly recall the GRE, because I had to get the train to Dublin to take it—it was offered in Ireland exactly once per year—and it was the first standardized, bubble-fill test I had ever had to take.
My own sense is that, with respect to outcomes, as you go up the scale the GRE starts to look more and more like those Ragin-esque triangular distributions. There should be both a higher mean and more (probably much more) variance in outcomes when looking at the high-end of GRE scores than at the low end. But then this is why Departments look at more than one bit of information, and still either objectively get it wrong or subjectively intervene in the attainment process in a negative way, depending on the circumstances.
LikeLike
Kieran
May 9, 2011 at 1:17 pm
Fabio, I am enjoying this conversation. However, I find the insistence that GRE predicts anything to be troubling. The meta-analysis you posted shows correlations between GRE scores and some success measures (measured with error), but I am reluctant to define correlations at .35 or so to be prediction.
For those who have asked about country of origin, I find some evidence that GRE scores, especially on the verbal, are poor correlates with success in the social sciences for applicants from China.
LikeLike
Randy
May 9, 2011 at 1:57 pm
@Guillermo re: diminishing returns
I imagine that there are very likely diminishing marginal returns to each extra 10 points on a GRE score, I would guess that the difference between a 1500 and 1600 is a lot smaller than the difference between an 1100 and 1200. I doubt that the threshold is at 1200 though. No evidence to back that up, just a general hunch.
@Mike
I definitely agree that the central goal of the GRE is not a test of intellectual ability as much as a test of commitment and dedication to do what it takes to get into the best grad school you can, and that’s a characteristic that carries over to when you’re at grad school. Ultimately, it’s just a test of high school level knowledge. The people who are motivated to work hard to get a good GRE score are the same people who are motivated to work hard and become successful researchers. Doesn’t mean that they all will though. Motivation and work ethic are necessary, just not sufficient, determinants of success.
LikeLike
Sam
May 9, 2011 at 2:41 pm
Steven Stills was rejected from the Monkees, despite standardized test scores playing no role whatsoever in the selection process.
My hypothesis: Had Art Garfunkel been admitted to Harvard and gone there, Paul Simon would have switched over to someone else who stayed behind in NYC, and they would have become enormously famous, and the other guy’s having not gotten into Harvard would now be cited as an example of how flawed admissions processes are. (Unfortunately, I am teaching today and will not have the opportunity to build the machinery necessary to test this claim.)
LikeLike
jeremy
May 9, 2011 at 3:07 pm
This is the first time I’ve been made aware that PhD training is supposed to be life training. We’d better introduce some classes on relationship building and parenting to our curriculum.
LikeLike
brayden king
May 9, 2011 at 3:21 pm
In the universe of social science correlations, 0.35 has to be considered reasonably strong. No, it’s not determinative (and I don’t remember anyone saying it is), but it’s not ignorable noise, either.
My *hunch* is that the correlation would be stronger if one stratified by country of origin (not just international vs. non-international), given the oft-noted problems with GRE scores from east Asia. As I recall, these were just coming to light in the late 1990s/early 2000s, and presumably affected the original studies on which the 2001 metaanalysis was based.
LikeLike
krippendorf
May 9, 2011 at 3:52 pm
brayden,
exactly. i think guillermo is importing wholesale an anti-SAT polemic as applied to undergrad liberal education without appreciating that L&S grad school is in fact pretty much exclusively about developing scholars rather than about developing well-rounded people who are good at life. this is why grad school admissions committees don’t really give a shit if applicants played first chair violin, made it to the state finals in track and field, or spent their summer vacation doing service learning with Habitat for Humanity.
LikeLike
gabrielrossman
May 9, 2011 at 4:14 pm
Many of my students are not successful on standardized tests. They don’t have a history of successful standardized test preparation, in part because they were not concerned with earning high SAT scores given their choice to attend the local public college at which I teach. In addition, few have the spare cash for the test-prep courses which would raise their score. While I agree that the test correlates with the outcomes of interest, I don’t think that’s because the score predicts those outcomes–and it’s certainly not a perfect predictor, but rather one that further advantages those who begin with advantages.
Furthermore (perhaps more relevant to the prior post), I’m not sure how you can easily determine the difficulty of a course or a major without intimate knowledge of the institution in question. A couple of examples: at my own undergraduate institution, the average grades awarded in physics classes were quite a bit higher than the average grades awarded in sociology classes, and not because the sociology classes attracted less able students. At my current institution, the difficulty of courses is predicted most clearly by who is teaching them, not by what the course title is.
As for measuring success itself, many of my students who consider Ph.D. programs are not interested in academic careers at all, preferring instead the idea of working in NGOs or something. I don’t think looking at faculty ratings and citation counts would be a useful way to determine their success.
Finally, what about honors work/thesis completion? It seems to me that successfully completing a non-required independent sociological research project would be the greatest indicator of graduate school success. I know it–along with my research assistant experiences in survey construction–is what gave me an advantage in graduate school. I don’t teach at an institution offering graduate degrees, but I would rather work with a graduate student who had already demonstrated the capacity to do research successfully than one who appeared smart and academically capable but who might not be cut out for the research enterprise.
LikeLike
Mikaila
May 9, 2011 at 4:23 pm
Lol. Pile on, you people.
“This is the first time I’ve been made aware that PhD training is supposed to be life training. We’d better introduce some classes on relationship building and parenting to our curriculum.”
Now that you mention it, it wouldn’t be a bad idea. MBA executives are supposed to take courses on ethics, deontology, leadership, *organizational behaviour* and similar matters. Why shouldn’t a scientist be trained in similar matters? You are saying their work does not have implications for society?
LikeLike
Guillermo
May 9, 2011 at 4:31 pm
“Furthermore (perhaps more relevant to the prior post), I’m not sure how you can easily determine the difficulty of a course or a major without intimate knowledge of the institution in question.”
Mikaila is right. In many cases (specially in the social sciences) grades in college reflect not only hard work, but also the ability to massage the ego of professors by parroting their own ideas. They tell you little about whether the student is able to think sociologically (which is what undergraduate training should be for). A lot of people get rewarded with graduate admissions for being “good students”. This doesn’t mean they will be quality scientists.
LikeLike
Guillermo
May 9, 2011 at 4:46 pm
1) GREs are positively correlated with ability and ultimate success in grad school, but the correlation is modest. Ditto GPA — correlated, correlation modest. It is predictive, but there is still lots of error in the predictions. This is because what it takes to be a good researcher and prolific writer are not the same as what it takes to be good at taking standardized tests and doing what teachers tell you to do.
2) It’s been a couple of decades, but when Wisconsin folks last assessed their admissions data, the committee’s ranking of a candidate had a significant positive effect on outcome measures AFTER CONTROLLING FOR GRE AND GPA. Interpretation: the committee was good at identifying other factors to take into account, even if they could not be rigorously quantified.
3) Some of what predicts success in grad school are the same things that predict high GRES: coming from a highly educated family & having lots of cultural capital.
4) The verbal and math GREs have really different distributions and correlates, and the methodologically dumbest thing anybody can possibly do is to add the two scores, especially at the high end. The math GRE is too easy for people who know math and its mode is a perfect score. The difference between the 99th and 80th percentiles is just a few questions. There is a much larger gap in score/ability between the 99th and 80th percentiles on the verbal test. The verbal test, of course, favors people who are native English speakers with highly educated parents.
5) I repeat, I’m not denying that GREs correlate with ability. If other things are roughly equal, the person with high test scores is a better bet. But I’ve seen too many people use the tests in ways that I consider to be mindless.
LikeLike
olderwoman
May 9, 2011 at 4:47 pm
I’m always weary about the GRE studies and graduate admissions decisions. I’ll go ahead and admit that I did not have the highest scores in the world and I somewhat fall under Mikaila’s experience with students’ lack of standardized test prep given my low resource school system…but I did get into graduate school and finished my master’s and doctorate within 6 years.
With that said, the meta-analysis looks at first year graduate school GPA. I would almost throw this out of the analysis given my experience and understanding of how inflated grad school grades are (there was always this unspoken understanding that it was rare for someone to receive less than a B- in a class across most programs).
Secondly, I somewhat support Jon’s mention of the lack of math courses or components of current sociology programs (save a few). I’ve found that students that have the highest quant GRE scores applying to our program are ones that have a double major in psychology (the most likely combo).
Just my .02. As others have said, this is a good thread!
LikeLike
Hillbilly
May 9, 2011 at 4:52 pm
” I somewhat support Jon’s mention of the lack of math courses or components of current sociology programs (save a few).”
To answer what another poster asked several posts ago, I think the difference between GRE scores of economists and sociologists lies in the quantitative part (not so much in verbal). And yes, many sociology undergrads have serious defficiencies in arithmetic. How can we expect them to take courses that are heavy on statistics and calculus?
LikeLike
Guillermo
May 9, 2011 at 4:59 pm
“exactly. i think guillermo is importing wholesale an anti-SAT polemic as applied to undergrad liberal education without appreciating that L&S grad school is in fact pretty much exclusively about developing scholars rather than about developing well-rounded people who are good at life.”
I don’t know why at least two people got this completely false impression. What I understand as “training for life” is preparing individuals who are later going to be useful to society. It has nothing to do with playing the violin or any of that folderol. If that’s your idea you’re completely misunderstanding the argument. What I was telling Fabio is his criteria give too much importance to the incubator and too little to the viable product.
LikeLike
Guillermo
May 9, 2011 at 5:09 pm
A few correlations below .35 from the GSS:
* father’s education with respondent’s current income (.10)
* political views (liberal to conservative) with view about acceptability of abortion for any reason (.19)
* political views with church attendance (.18)
Yeah, so I feel pretty comfortable saying that GRE is an OK predictor of grad school success…
LikeLike
Steve Vaisey
May 9, 2011 at 5:46 pm
“A few correlations below .35 from the GSS:”
Satoshi Kanazawa has statistical studies proving that blacks are inferior to whites. Are we supposed to accept statistics without question or to critically analyze assumptions and non-controlled variables in those studies?
LikeLike
Guillermo
May 9, 2011 at 5:50 pm
It must also be said that the authors of that study use the word “small” to refer to the .2 coefficient. It’s not my words.
LikeLike
Guillermo
May 9, 2011 at 5:52 pm
I’ve been keeping track of the Berkeley admissions data for the past 3 years (N = 1100) and here is one potentially pertinent finding:
Given that an applicant’s letter writers presumably have much more information than we do about him or her (and in some cases do not know his or her GRE scores), the correlation between the quantitative score given by the letter writer* and GRE might be a useful indicator:
r(verbal GRE, average letter score) = .25
r(quant GRE, average letter score) = .20
(Lowess plots of these relationships suggest that the quant relationship is pretty much linear and the verbal relationship is linear until topping out around 700.)
So, again, these aren’t perfect by any means, but they’re clearly picking up something that “on-the-scene” advisers think is valuable.
* Letter writers rate the applicants on a 7-point scale from “top 1% – truly exceptional” down to “below average.”
LikeLike
Steve Vaisey
May 9, 2011 at 5:59 pm
Guillermo, at the risk of feeding the flames, you must understand that there is a difference between describing a statistical relationship and the goal of causal inference. No one is saying that “GRE ability” is a CAUSE of success in grad school. What some are saying is that it is a useful predictor of success in grad school, whatever the reason for the observed relationship. Your Kanazawa example is just emotional baiting.
LikeLike
Steve Vaisey
May 9, 2011 at 6:04 pm
” father’s education with respondent’s current income (.10)”
Out of curiosity: do you have the correlation between mother’s education and current income? (controlling for father’s income, of course).
LikeLike
Guillermo
May 9, 2011 at 6:06 pm
Knock yourself out:
http://sda.berkeley.edu/cgi-bin/hsda?harcsda+gss10
LikeLike
Steve Vaisey
May 9, 2011 at 6:07 pm
“father’s income” For father’s education, of course. Sorry.
LikeLike
Guillermo
May 9, 2011 at 6:08 pm
I think olderwoman’s points about how GRE scores likely correlate with other (stronger?) predictors of success like socio-economic background and cultural capital should be stressed.
I wonder if some of the issue here is the definition of success vs. expectations when admitting students to graduate programs. If success minimally is completing the degree and publishing enough to get a job in the field (likely an academic job for most fields as people like to self-replicate), then GRE’s might be a predictor as good as any other.
But is this the definition of success employed during the graduate admissions process? Do committees think ‘sure, I can see this candidate getting a job at low status institution x, so let’s admit them!’ Or are they trying to maximize their success given the applicant pool (in terms of whatever status markers the field might have)?
I can imagine it’s the latter. And it’s hard to see that GRE scores are primary predictors of that. Anecdotally, we have all the geniuses that tested poorly and performed well. More helpfully, we can probably infer that while many superstars have high GRE scores, many that have high GRE scores are not superstars. Perhaps a (somewhat) necessary, but insufficient condition of genius?
LikeLike
cwalken
May 9, 2011 at 6:14 pm
I’d love to see a study correlating respondent’s GRE scores vs. their obsession with them on Internet comment boards. If everyone starts posting their score in this thread we might be able to make some headway on this.
LikeLike
anonymous
May 9, 2011 at 8:00 pm
Guillermo & Hillbilly: Yep seems clear to me that if Sociology was a lot more ‘mathematical’ then your average Soc. applicant to grad. school would have much higher scores in the quant. Let’s be honest, the quant. section of a GRE is extremely easy if you’re doing a discipline that relies heavily on maths.
Personally, and this is a UK perspective here, I think Soc. degree’s ought to have a first year course called ‘Maths for Sociology’ giving students a framework in things like Calculus, Linear Algbra etc and then offer more intensive courses in the second and third years. I’m not calling for maths to colonise the discipline, rather just the introduction of it into the discipline! After all, mathematical sociology is a non-presence pretty much everywhere (in the UK at least).
Someone also mentioned Kanazawa using stats to ‘prove’ things. I hope this is a joke cos’ I think that guy has been shown (at least once) to be a tad incompetent with his stats.
LikeLike
Jon
May 9, 2011 at 8:53 pm
As far as socioeconomic background and cultural capital go, as per OW’s earlier comment, plainly this is an issue for GRE scores, and as far as I can tell awareness of this in sociology approaches 100%. On the other hand, I think writing samples are considerably laden with the same things and in similar ways, and I’m not sure that’s as widely recognized.
It would be an interesting study to see whether one can predict SES background of American applicants better from GRE scores or from the writing sample (by which I mean the sample of original writing/research, not the biographical statement of purpose). My wager would be on the writing sample, I think.
LikeLike
Jeremy
May 9, 2011 at 11:17 pm
Does the Kuncel et al. study account for the fact that those with higher GRE’s tend to go to better graduate programs? Some would call this “value-added”, while others would emphasize Matthew Effect arguments. Regardless, quality of program seems relevant (probably more relevant than GRE scores) in regards to predicting things like time to degree, publications and citations received (which, incidentally, were their weakest results).
LikeLike
QuadC
May 10, 2011 at 2:18 am
@Jeremy, I am glad you made the point about writing samples picking up on cultural and social capital. I am also surprised that this isn’t wider recognized. My intuition is that, if a grad school applicant is the son or daughter of a professor, they would probably have an excellent personal statement. I am less certain they would have a perfect GRE score. Even ignoring the issue of academic habitus, it’s far easier to get substantial help in formulating a plausible research plan and sounding like a serious aspiring academic than it is in getting an 800 on the math GRE.
LikeLike
Andrei
May 10, 2011 at 2:43 am
Agree that writing samples also exhibit class effects, and mentoring effects. I also thought I could see (sometimes) evidence of independent thinking, quality of intellect, creativity, energy and enthusiasm. It was my belief that having done and written up a “big” project that exhibited the qualities I’d like to see in seminar papers from first or second year grads is the best predictor of success in grad school. But I never had the data to check my theories. I recall a few students I was sure were winners despite low GREs based on writing samples; we did not admit them, so I don’t know how they ended up doing, but I’d really like to find out to see if I was right.
LikeLike
olderwoman
May 10, 2011 at 12:06 pm
One strength of the Kuncel piece is that it accounts for the attenuation in correlation coefficients you see in analyses of the GRE’s predictive validity (because the test is working with a restricted range of observations). Thorndike identified this problem for studies of GRE validity in 1949, but Kuncel et al are the first to address this with a statistical work-around.
However, they don’t disaggregate their analyses/findings by test-taking populations at all. See Sternberg and Williams (1997) for good data on why the GRE isn’t as valid a selection criterion for students who are underrepresented in their field (gender and race/ethnicty).
I’m writing a qualitative dissertation study on faculty judgment in PhD admissions, and finding a disturbing number of faculty have believe the GRE not only serves as a “credible signal”, but is a marker of intelligence (“native” and “raw” intelligence are the phrases I hear over and over). For all the reasons @Jeremy, @olderwoman, and others discussed, we know it’s not.
LikeLike
Julie
May 10, 2011 at 12:44 pm
Julie: If you would like to share your dissertation results on the blog, please send me an email.
LikeLike
Fabio Rojas
May 10, 2011 at 1:28 pm
Oh how I wished we used GRE scored for graduate admissions up here in Canada. We don’t even have a consistent GPA scale.
LikeLike
tina
May 10, 2011 at 3:12 pm
scores
LikeLike
tina
May 10, 2011 at 3:13 pm
“Oh how I wished we used GRE scored for graduate admissions up here in Canada. ”
Rather, make social science programs more research oriented and increase the level of exigence in courses.
LikeLike
Guillermo
May 10, 2011 at 3:25 pm
Julie,
>For all the reasons @Jeremy, @olderwoman, and others
>discussed, we know [the GRE is] not [a marker of intelligence].
You are setting up a false dichotomy that assumes a measure is either a perfect errorless representation of a latent variable or just pure noise/bias without allowing a middle ground for noisy but still non-trivially informative measures. There is no inconsistency between saying that GRE scores (and related measures) can exhibit some bias and saying that they measure intelligence. Certainly I’d like to meet a psychometrician who would tell us that GRE is not a g-loaded measure.
How relevant particular marginal shifts in g are to graduate studies in sociology (especially as compared to alternative observables), or whether g measures are systematically biased for particular populations, are questions about which we can reasonably debate. However saying that the faculty you are studying are wrong to think that GRE measures intelligence (albeit in a biased fashion) at all is just silly.
LikeLike
gabrielrossman
May 10, 2011 at 4:09 pm
Julie – I’m looking at the Sternberg and Williams (1997) in American Psychologist, and see little discussion or evidence in regards to gender and race/ethnicity. They do not touch on race at all, and the only meaningful gender difference they found was the old GRE-Analytical had predictive value for men, but not for women.
Regardless, Sternberg and Williams found little to no predictive ability of GRE scores for graduate students in the Yale psych department. If these null findings are legitimate, they seem worth discussing as well. Could there be scope conditions on the usefulness of the GRE?
LikeLike
QuadC
May 10, 2011 at 4:21 pm
Use of the GRE and GPA to make close distinctions among top-tier students (I don’t know what the threshold is) in a methodologically-varied discipline like sociology or political science seems problematic to me.
I think, in this age of rampant test prep, the GRE and GPA can be a proxy for how much value a student places on prestige. Do they have the inclination and ability to spend weeks (possibly months) doing test prep to make a modest or considerable jump in scores? If they do, then they probably will. If they don’t especially care whether they get a Ph.D. from a top-5 or a top-25 school, then they might not. Do students with less “credential ambition” have less to contribute to the discipline?
It’s not hard to see how this sort of “credential ambition” correlates with academic job placement and even, in many cases, scholarly productivity (as measured by publications). To the extent that this is the case, it might be worth admissions committee’s attention. However, this is much different than using the GRE as a metric of an applicant’s intellectual horsepower.
It also seems that once a certain threshold is reached, some considerations take on added significance. So, for instance, once ABC University has determined that all applicants with GPAs/ GRE scores over some fuzzy range could potentially do the work, the school should look closely at their other features and at mitigating factors those applicants might have faced, whether it is background-related (class, race, etc.) or something else (including such things as test anxiety, other mental health issues and learning disorders etc.).
The extent to which a student’s background determines his or her credential ambition is certainly worth keeping in mind. Much of this is cultural. For instance, the distinction between, say Vanderbilt and Cornell, was much more apparent to my classmates at a Northeastern liberal arts college than it was to my classmates in an honors program at a land-grant university (despite the fact that both groups had, on average, roughly the same high school board scores).
I don’t want to overstate my case, I do think academic ability can be and often is reflected in test performance, broadly-speaking. It’s the finer distinctions I find foolish. I think the effort hypothesis is bolstered somewhat the recent research by Duckworth at UPenn.
LikeLike
Kevin O'C.
May 10, 2011 at 4:22 pm
Steve (Vaisey): I only asked about maternal education and income because, in other countries, my experience is you’ll find a much stronger correlation between those two when controlling for paternal education. This is what jumped at me when you presented your .1 coefficient. But here we are jumping off-topic in what has been a most interesting topic (regardless of the occasional silly comment, such as the anonymous, May 9 at 8:00 PM. But this is the Internet, you know what you’re getting into when you buy your ticket, etc.).
LikeLike
Guillermo
May 10, 2011 at 5:04 pm
“Certainly I’d like to meet a psychometrician who would tell us that GRE is not a g-loaded measure.”
There is a strong correlation between g and SAT scores. There are also, however, strong criticisms of g as a measure of intelligence.
LikeLike
Guillermo
May 10, 2011 at 5:27 pm
[…] Lots of debate over the predictive validity of the GRE over at OrgTheory. […]
LikeLike
Assorted Links « Permutations
May 10, 2011 at 5:27 pm
So,… Fabio, what is the inconvenient truth at this point of the conversation? Perhaps that admissions is a very noisy process, whose outputs (matriculants) enter a very noisy process wherein measurement of success is multidimensional and subjective, subject to a number of mediating, moderating, and attenuating factors. Cool.
LikeLike
Randy
May 10, 2011 at 5:56 pm
Rnady: The inconvenient truth is that
(a) GRE’s correlate about .35 – that’s actually higher than many other social science findings that are viewed as “strong results.”
(b) No one every said that GRE’s were *perfect* predictors and nobody treats them that way.
(c) There are some exceptions for elite programs where variation in GRE’s is low because everyone scores high.
(d) Anecdotaly, there seem to be threshold effects. People don’t report a lot of success with low GRE students.
(e) There is some evidence that faculty can spot students beyond their GRE’s.
(f) GREs are waaaaaaay better than letters of recommendation, which almost all research has shown to be junk.
(g) GRE’s do measure hoop jumping and that’s a good thing. Academia is non-stop hoop jumping.
Overall, this supports my original point that GRE’s are a pretty good tool for sorting graduate students. And that we should also look for other signs for students with decent scores.
LikeLike
fabiorojas
May 10, 2011 at 7:02 pm
(d) would actually be a convenient truth if it were true, but I don’t think it is. It’s easy to project thresholds onto fully monotonic gradients. See, e.g, the health disparities literature.
(This isn’t a claim about how much should be made of score differences of a given magnitude; making too much of modest differences is misguided no matter where in the continuum we are talking about.)
LikeLike
Jeremy
May 10, 2011 at 9:11 pm
@Jeremy: Yes, I think I may have over stated (d). but here’s a question: Has NWU soc, in your time, admitted *anyone* with all GRE’s below 1 standard deviation below the mean? If so, were they successful in completing the program? David Meyer indicated working with successful low GRE students, but he didn’t specify what would count as low.
LikeLike
fabiorojas
May 10, 2011 at 9:18 pm
i wonder if anyone has, or even can, control for test prep. i’ve heard people poo-hoo kaplan and princeton review’s abilities to raise sat/gre/lsat/mcat. but i can tell you as someone who worked at kaplan and knows many others who’ve worked at kaplan and princeton that they do raise those test scores phenomenally.
and how does gender play into it — women tend to score lower right (i remember reading that, but maybe that’s dated info)? but they tend to have higher gpa’s, so…
LikeLike
syed ali
May 11, 2011 at 1:13 am
“and how does gender play into it — women tend to score lower right”
Foreign students consistently tend to score lower, as even ETS acknowledges.
It’s also worth noting that according to Kaplan, each “reform” to the test results in a drop in mean test scores across the entire pool. Not that colleges take this into account much, though.
LikeLike
Guillermo
May 11, 2011 at 2:10 am
Yes, check out this article:
1985 Journal of educational measurement
EFFECTS OF COACHING ON GRE APTITUDE TEST SCORES
DONALD E. POWERS
Abstract
Test preparation activities were determined for a large representative sample of Graduate Record Examination (GRE) Aptitude Test takers. About 3% of these examinees had attended formal coaching programs for one or more sections of the test.
After adjusting for differences in the background characteristics of coached and uncoached students, effects on test scores were related to the length and the type of programs offered. The effects on GRE verbal ability scores were not significantly related to the amount of coaching examinees received, and quantitative coaching effects increased slightly but not significantly with additional coaching. Effects on analytical ability scores, on the other hand, were related significantly to the length of coaching programs, through improved performance on two analytical item types, which have since been deleted from the test.
Overall, the data suggest that, when compared with the two highly susceptible item types that have been removed from the GRE Aptitude Test, the test item types in the current version of the test (now called the GRE General Test) appear to show relatively little susceptibility to formal coaching experiences of the kinds considered here.
http://onlinelibrary.wiley.com/doi/10.1111/j.1745-3984.1985.tb01052.x/abstract
LikeLike
fabiorojas
May 11, 2011 at 2:23 am
I’m always amazed at the overwhelming power attributed to test prep as a mechanism for the reproduction of inequality. If Kaplan didn’t exist, the critics of standardized testing would have to invent it as a sort of Satan for the theodicy of the academic labor market.
LikeLike
gabrielrossman
May 11, 2011 at 2:42 am
Gabriel (and everyone else): My PhD adviser was Rafe Stolzenberg. At one point in his career, he was the VP for Research (i.e., test development and evaluation) at GMAC. So I asked him if coaching improved test performance. As usual, he gave a very careful and meticulous answer. What he learned from his time at GMAC was that coaching gave a moderate boost to a student who had the ability to do better but lacked specific knowledge.
In other words, coaching is not a magic bullet that transforms students. High performing students don’t need it and it won’t help low performing students. However, there’s a specific type of student who has the ability to do well but simply deficient in some very specific area, such as not knowing vocabulary or simply not having much exposure to analytical puzzle solving. Then you get a good, but not great, test boost.
This tends to explain a puzzle about coaching – why everyone doesn’t do it. If all that separates students is coaching, then all you need is to give $600 to each person so they can take the coaching course and you’d have a massive shift in scores. The reason this doesn’t work is that coaching has a modest effect for students who are good, but not great.
Let me finish by pointing out why coaching persists. Aside from high aspirations, numerically speaking, there’s a lot of people who may be in that margin. Shifting from 60% to 80% on the LSAT may mean the difference between a top 100 law school and top 20. So the modest effect of coaching has huge marginal effects on life course outcomes.
LikeLike
fabiorojas
May 11, 2011 at 3:47 am
on gender differences: as of a decade ago women scored, on average, 9 points lower on GRE-V, 25 points lower on GRE-A, and 97 points lower on GRE-M (source: ETS 2001/02). These gaps are narrower than they were in the early 1990s, and the gender gaps are presumably even narrower now.
The more surprising gender story, in my view, is that a much higher percentage of test-takers are women: in the late 1990s, roughly 60% of all test takers and 65% of white test-takers were women, approximately the inverse of the percentage of women among new grad school entrants. I didn’t dig deep enough into the ETS report to discover how people who took the test multiple times were counted. Either way, though, the gender distribution of the test-takers makes it pretty safe to assume that the gender means are affected by some wacky selection effects.
LikeLike
krippendorf
May 11, 2011 at 5:14 am
But as I know, for Chinese, or other East Asia students, they are very used to exam-oriented education, and they can handle standardized test quite well, but generally weaker in other more practical areas. Is the research that “coaching is not useful” consistent to the performance of Asian students?
LikeLike
passerby
May 11, 2011 at 6:01 am
[…] Fabio Rojas provoked an intense discussion about GRE scores – here and here. Initially, I did not understand why a number of responders seemed to be disappointed that the […]
LikeLike
Does Your GRE Score Say Something About You? | The Sociological Imagination
May 11, 2011 at 1:43 pm
“The more surprising gender story, in my view, is that a much higher percentage of test-takers are women”
For years, women have been increasing their participation in higher education, to the point of surpassing them. This is not surprising at all.
LikeLike
Guillermo
May 11, 2011 at 10:24 pm
On test prep:
Before I went to grad school, I did one on one test prep (N=150). There are varying kinds of plateaus one finds with different types of students, i.e. someone who is at 1050 on the SAT (the old version) might reasonably be prepped up to 1200, but is *highly* unlikely to reach 1300, let alone anything higher. The kid who’s at 1400 to start might make 1450 with prep, or might make 1580, but the latter has to do more with some form of native intelligence that processes what test prep is teaching (logic, reverse engineering) in a much more efficient way that gets them past the testing bottleneck (the point is that they more quickly realize what’s being tested and adapt). What that kind of intelligence this points to is not certain, but it is a distinguishing feature. I will grant that it may not overlap in a meaningful way with the kind of intelligence required for successful navigation of grad school (but it may!).
I think, generally, that the differences in the highest end of the testing distribution are the most telling. You *can’t* prep your way into a 1600. You might be able to pre your way into a 1470, but the very high end requires an attention to detail, a processing of very fine distinctions that (though while arbitrary) point to a suppleness of mind perhaps not present in 1200 level students or even 1400 level students. This all comes with the proviso that unprepped students can’t be compared to prepped students. Prep improves, but most students who are competing at the highest levels of graduate admissions have at least had the opportunity to do some test prep even if self-instructed.
The ultimate question of interest is whether what the SAT/GRE is testing relates to the practical skills necessary to be a successful academic. I imagine this varies somewhat by field, in any case. But how often do we encounter “successful” members of our fields that we mostly discount as having ridden a wave of faddishness to success? Does GRE testing offer a counterargument? I don’t know, and I’m not quite sure how to make that kind of question appropriate.
Two final notes: at the high end of the distribution, my intuition is that very high scores provide a clearer signal about attention to detail, nuanced thinking, perseverance, than do scores in the 1200-1400 range. Whether those skills matter in successful completion of graduate education (still kind of undefined) is beyond my ability to know at this point. Finally, not all standardized tests can be similarly analyzed. For the LSAT, for instance, it’s many magnitudes harder than the GRE to place into the top percentiles. What it’s measuring is certainly different, but discussion of individual tests should press upon the closeness of fit between the test and the desired admissions’ goal. By way of comparison, no amount of prep will get you in the 170s on the LSAT even if your starting point is in the 160s. What the test is actually measuring is certainly up for debate, but different tests have different susceptibilities to test prep.
LikeLike
George
May 12, 2011 at 4:57 am
Guillermo: “Surprising” is in the eye of the beholder, and if you know everything, you will rarely be surprised. Personally, I find it curious (and surprising, too!) that in 1999/2000-ish, women’s representation among GRE test-takers exceeded their representation among BA recipients by about 5 percentage points, it exceeded their representation among new graduate degree recipients by about 20 percentage points, and it exceeded their representation among grad degree recipients in non-health fields by closer to 25 percentage points. (The latter is relevant because many nursing and related health programs don’t require applicants to take the GRE.)
LikeLike
krippendorf
May 12, 2011 at 1:32 pm
krippendorf: Thanks, I love you too.
“Personally, I find it curious (and surprising, too!) that in 1999/2000-ish, women’s representation among GRE test-takers exceeded their representation among BA recipients by about 5 percentage points…”
Meaning that a lot of women who graduated *before* 1999/2000 decided to pursue graduate studies that year? (progressive feminization of higher education hypothesis).
“…and it exceeded their representation among grad degree recipients in non-health fields by closer to 25 percentage points.”
Because health-related areas of education tend to take in a lot of the share of women pursuing graduate studies? (sex segmentation hypothesis).
LikeLike
Guillermo
May 12, 2011 at 3:24 pm
In regards to test prep: the key thing that it provides are the strategies for approaching the test, which can make a big difference in the score for at least a small number of students. For example, I have smart, gifted students–including one with a 4.0 GPA–who have never learned to look through the exam and start with the easier questions. Similarly, the analytical reasoning questions that used to be on the GRE and are still on the LSAT are of a type that is particularly coachable, as students are unlikely to ever have encountered them before. I myself raised my analytical score on the GRE at least 100 points using a CD prep course.
Now, the changes in the GRE may have made coaching less useful in recent years. I don’t have the data to answer that question. But I know that on the LSAT those $1400 prep courses (and not, for most students, practice alone from the book) easily and often raise scores 10 points or more. In the law school context, that can be the difference between getting in nowhere and getting in to a strong school or earning a scholarship.
LikeLike
Mikaila
May 12, 2011 at 5:57 pm
I just wanted to toss in my N=1 story re: earlier comments about the quant GRE being easy for students from math-heavy fields. I was on math team in high school, and about 1/2 of the courses I took in undergrad were applied math of one form or another. The math on the GRE is elementary, and I could have gotten a 700 without studying. But getting that 800 required about 4 months of intensive daily self-training. This makes me think that two things could have been the case: (1) I could have secretly been worse at math or test-taking than my previous experiences have led me to believe, or (2) an 800 on the math GRE actually takes both an excellent mathematical ability and a lot of hard work. Not to be too self-congratulatory here, but I suspect it was (2). So, put me down as a firm believer in high GRE scores being a reliable indicator of both intelligence and effort. I think that low GRE scores are a less reliable indicator, though: some people just get too stressed to do well. On the extreme end, if someone has a nervous breakdown during the GRE, the only thing their score will signal is that they get nervous breakdowns.
LikeLike
anon
May 12, 2011 at 8:20 pm
I have been directing an MA program for about 4 years. Currently, we do not require GRE, but I could imagine why some programs would like to use:
1. High GPAs are very difficult to compare between many schools. Four years into this DGS gig, and I frankly still do not have a strong sense of how all of my region’s colleges compare. For several reasons, I would like to be able to consider very high-performing students from less reputable universities, both on principle and because we’ve found really good talent in relatively unknown colleges. The problem is that some colleges in our region can produce students with 3.6 or 3.7 GPAs who lack basic skills. The GRE at least provides a common measure of _some_ academically-relevant abilities.
2. GREs probably thin out your applications pile. Many undergrads are intimidated by the GRE. Sometimes they lack the discipline or ability to prepare on their own. Sometimes they’re disorganized or lack motivation. In any case, these are qualities that programs might want to weed out of their applicant pool.
3. Signalling. I have gotten looks from colleagues when I tell them that we don’t require GREs. To them, this is the mark of a bad program. We’re quite competitive in our market, which insulates us from the need to adopt the GRE based on peer pressure. However, if we weren’t in a secure position in our market, I would be anxious to signal our program’s quality by requiring the GRE in the first place and concentrating on upping our incoming classes’ scores (if they play into some magazine rating or whatever).
I imagine the GRE is good for identifying students with tremendous abilities and sifting out basket cases. However, you can miss out on good students if you fetishize the test. In particular, we’ve trained Asian immigrants whose TOEFL scores were not all that terrific (so I’m guessing that they would have bombed the GRE verbal), but they have strong scores in quantitative classes from reputable Asian universities. In some cases – though not always – they have become highly effective data analysts who ended up doing good work on big projects. They are also smart enough that they would probably be decent writers after two or three years of a doctoral program (their thinking is well-organized and logical, but their vocab, speaking ability and listening ability can be rough).
LikeLike
Joe
May 13, 2011 at 6:03 am
It’s late in this thread, but I recalled the “GRE scandal” of the late 1990s and early 2000s. http://www.nytimes.com/2002/10/15/news/15iht-rchina.html
Graduate programs started seeing large numbers of applications from overseas with very high GRE verbal scores. Turns out there were web sites where you could study the specific questions. Many experienced grad directors started discounting overseas GREs long before ETS acted, but when I was doing grad admissions I worked with colleagues who would say “how can you turn down someone with high GRE scores?” I do think too many people make a fetish of the test as a gold standard even in the face of contrary evidence. I think it is because it is a nice simple number, and everything else takes more work to evaluate. (BTW I was also seeing fake letters of reference, fake transcripts.)
I talked to a friend who was a linguistics professor from China. He said he would not trust a test score as a measure of someone’s verbal ability — “it is just a test. You can study for a test.”
Again, I am not disputing that the GRE correlates with ability for the appropriate populations. But blindly giving to much weight to test scores without examining corroborating evidence is a big mistake.
LikeLike
olderwoman
May 13, 2011 at 12:28 pm
I agree with the folks who said that the GRE shows dedication.
Contrary to what some people have advocated here, I do think the test can be studied for, and yes, I do think people in the 1200-1400 range can reach the 1400-1600 range by studying. The first time I took the test, I scored 900 combined Q and V. I studied (hating every minute) for months and improved my score to a 1340 combined. I believe it would be very difficult for me to reach a 1600 but I do believe that if I studied very hard that I could get in the 1500-1600 range. I believe this because when I take the test in my room with no interruptions, timed, I score around 1400-1500, consistently. I think the test day conditions pose significant stress to my ability to think clearly. The test proctor walked by me, what seemed to be 20 times. Listening to people talk and others typing did make it so that some questions I had to read several times. By the end of the test I was extremely frustrated, agitated and I didn’t even finish the last three or four questions on the Q, and I rushed through a few V questions, essentially making uneducated guesses at times. Also, I read a blog of someone who scored a 1600 on the GRE on the internet. This guy said that he was a math tutor for years (I believe) and that he seemed to have quite a bit of prior training, and that he studied for months for the test. I also have a close friend who scored a near perfect score who has several degrees who made it clear to me that training helps significantly.
That being said, I learned probably close to 3000 words for this test and my reading comprehension has absolutely improved as a result. Do I think that is essential for someone entering a verbal-intensive program? No. But, I do think that it will probably save a lot of time and improve performance because I have already looked up and studied thousands of words so that I don’t have to waste my time doing that later in my studies. Someone without this knowledge will more easily be bogged down. My mind can only pick up so many words a day. I’ve tried reading things, for example, in Spanish where I had to look up five words per sentence and I got about 2 pages through and had to stop. I imagine someone with a weak vocabulary would have a similar experience in a verbal-intensive program, which would pose a significant obstacle to his or her success.
I do not specialize in intelligence testing but I am suspicious of these types of measurements. That being said, there does seem to be obvious differences between people’s abilities. Exceptionally smart people may be able to study for the SAT or GRE for a few months and score a 1600, or barely study at all, whereas it might take me years, and someone of less intelligence, decades. My .02.
LikeLike
David
July 10, 2011 at 9:28 am
One point to add is, as I know, why most Chinese (mainlanders) can get high score is because they spend a lot of time studying for it. The cheating by memorizing test banks from previous test takers has some, but I suppose little, effect. Competition is fierce in China and there is no good signalling mechanism, so everyone has got to take all kinds of exams and get a high score to grasp any chance.
LikeLike
passerby
July 12, 2011 at 9:29 am
http://abcnews.go.com/Politics/chinas-education-prepares-students-tests/story?id=12348599
“It would be the equivalent of American students spending four or five years preparing for the GRE or SAT, he said. So when it comes to exams, Chinese students are stellar performers. But when it comes to other barometers of success, they fall short.
“In terms of imaginative talking, coming up with good ideas, taking risks, those are actually very weak,” said Huang.
Such weaknesses are closely linked to an education system that effectively shortchanges Chinese students.
“When you spend all your time as a student at school going after high scores, you lose opportunity to develop anything else,” said Yong Zhao, an education professor at Michigan State University. “
LikeLike
Guillermo
July 12, 2011 at 6:43 pm
I took the Kaplan course recently, enjoyed it, studied hard and had terrible GRE scores. I panicked on the first question and lost five minutes and knew that I would not have enought time to finish. I have never done well on these types of intelligence tests. Even though I am a technical writer by profession, I did poorly (550) on the verbal part of the exam.
I am a professional with over 30 years experience in my field attempting to enter the Information Architecture program at Bentley University. I have no idea if my scores will make a difference in the decision to admit me to the program. But I do know that all the studying in the world would not have helped me pass the test. I understood the material and upgraded my math skills considerably. I felt that after I took the course that I had learned many useful methods of approaching problems in math.
I also felt that it helped my ability to solve both verbal and math problems. However, the actual test is designed to be study proof. Our instructor explained that the designers of the GRE test require that you not only had fully understand how to solve problems using what may be described as a normal approach, but the questions required the use of some trick, some quick way of figuring out answers. Time is a big factor.
I learned many of these “tricks” and it still did not help me. I feel as though I did the required study to update my skill in doing straightforward problems, but I lacked the “creativity” (for lack of a better word) to find the answers quickly and did not have time to work the problems the “long” way. What is interesting is that I am a composer, musician, painter, fiction writer, and technical writer (my profession). Many of the skills that I have creatively are far beyond the ability of many of my friends who are bright and intelligent and did well on SATs and GREs and have successful careers in their fields. I have produced hundreds of paintings, well over a hundred songs, many of which are jazz pieces.
Some of the projects I have worked in my field involve working with computer chip designers, business analysists, network and system designers, and so on. What bothers me is that bad scores on my GREs may prevent me from using my “creativity” to solve problems in a new field (or logical extension of my current field). How many people like me, who do no have measurable (quantifiable) talent are passed over and pushed aside by a system that does not accurately measure creativity and innovation? What does the ability to answer a question on a test really measure? How useful is that skill in solving complex problems in the real world?
LikeLike
Richard Nollman
August 24, 2011 at 10:03 pm
[…] Rules For Top-Percentile GRE Test […]
LikeLike
Gre Study | REFERENCE BOOK
November 14, 2011 at 9:24 pm
[…] we discussed before, GREs and GPA correlate well with admissions and future graduate school performance. This is a very strong finding from the higher education […]
LikeLike
is academia meritocratic? « orgtheory.net
February 13, 2012 at 12:05 am
I think that the test cannot be studied for in the traditional sense. I went through 2 Kaplan prep courses (did all the homework assignments), both of which only minimally raised my score (from a predicted 298 to 303). That is a dastardly score if I may say so. I had a 3.7+ GPA as an undergraduate, and even went to graduate school and did better (3.8+ GPA in a totally different subject than my undergraduate work). The third prep program I tried was Magoosh, which has obscenely hard math questions, this helped me out some (though not a whole lot) so I was able to raise my score to a 309/ or a 1230 on the old scale. However, I took the test three times in a row. I had the same stupid score 3 times. It never improved, even after memorizing countless, if not hundreds of words a day and buying and going through every available Barrons, Kaplan, ETS, and Princeton Review book known to man. The ETS practice tests gave me an estimate of 315-330 for my score. Obviously I was 20 points lower! This really frustrated the hell out of me (and still does). If I was clearly able to complete an MA program with a high GPA and publish research (which I did), I don’t understand why major institutions require Phd candidates who already possess an MA and have a GPA above 3.6+ from an American institution to even submit GRE scores. The fact that an admissions or funding basis at a top school can come down to 80 questions you are asked on a timed-scale not in your field of proposed study is absolutely absurd. Students such as myself have spent hundreds, if not thousands of dollars on this exam (personally I’m up to the 3k tabulation, and I still do not have the score I want).
What if someone doesn’t have the 3k to spend on the test? (I was lucky and someone paid it for me). How are most students supposed to afford tutoring, test prep classes, and taking the exam multiple times? They simple cannot. If it weren’t for someone elses’ generosity I couldn’t begin to try. This test is seriously wrong as it prevents many people from even attempting graduate school. Moreover, how many individuals is this test stopping? The great historical and cultural leaders in history never had to take the GRE to go to prestigious schools, or even bad schools for that matter.
LikeLike
Jamie
September 30, 2012 at 2:30 am
@Jamie: On one hand, you’re saying that you can’t study for the test, but on the other, you’re saying that the reason the test is unfair is because of the costs associated with preparing for it. If the preparation doesn’t help, why do the costs matter? Preparation was a huge help for me, and while not prohibitively expensive (I bought two Kaplan books), I can imagine how someone needing a more structured environment could be at a real disadvantage.
As for the usefulness of the GRE, what would you do if you were on an admissions committee at a top program, received 300 applications for 10-15 spots, and 100+ people had GPA’s of 3.5+ from a wide variety of universities and masters programs (the quality of which you weren’t always familiar with), and 75 applicants had letters of recommendation saying, “Amazing student!!” Among those 75 students with great GPAs and letters, let’s say 15 scored in the 90th% on the GRE, 25 between the 80th and 90th% and 35 below the 80th%, with 25 of those 35 scoring below the 60th%. Would this information about a standard, though imperfect, measure of reasoning ability not be helpful in distinguishing among the quality of applicants?
While there is certainly disagreement among faculty about the importance of GRE scores, my guess is that faculty pay attention to things like the relationship between scores and actual success on the job market. If there weren’t a relationship between the two, they’d stop paying attention to such things.
As for great historical and cultural leaders not having to take GREs despite going to prestigious institutions — I don’t know how far back in history you’re going, but the graduate schools they attended probably didn’t face the kind of application burden that today’s admissions committees face. The GRE has only been administered since 1949.
LikeLike
whyaskwhybuddry
October 2, 2012 at 12:27 pm
The multivariate specification we want here is one that uses a content analysis to break qualitative components like letters and SOP down into numerarie and regresses them along with GREs and GPAs on future performance indicators.
Until that happens, considering they are potentially hugely confounding covariates, any specification suffers considerable omitted variable bias.
LikeLike
Graham Peterson
December 4, 2012 at 11:22 pm
[…] […]
LikeLike
More GRE Info -- Because There Hasn't Been Enough Discussion Yet
December 4, 2012 at 11:25 pm
I am living proof that the GRE is a) a worthless test and waste of money b) a money-making scheme for ETS and c) completely biased and d) does very little predicting graduate success. I did about a 1000 on my GRE’s….not good at all. However, I was able to attain a 3.71 GPA from Johns Hopkins University because the school did not require the test for admission. There have been many studies conducted most notably by Cornell University that show how the GRE does very little predicting graduate school success. I look at the GRE along with other standardized tests with this analogy: I once knew a man who could count cards and remember 13 hands played in a card game. That’s great if you want to be a professional gambler but does very little on learning how to run a business.
LikeLike
Geoff McCord
December 21, 2012 at 3:02 pm
Your point is contingent on your GPA being in some way unsullied by the same metaphor you present at the end of your rant.
LikeLike
Ger
December 21, 2012 at 9:29 pm
When wondering whether a person’s scores on one single test (the GRE) can determine their success in graduate school or in life, please consider the circumstances.
Let’s say, hypothetically a person graduated 6 years ago with a degree in biochemistry, and has been working at a top 10 biotech/pharmaceutical company, has published 12 papers, 5 as a main author, has 3 drug patents and 1 transgenic animal patent, and is highly paid and happy at their successful career, yet just wants to get their PhD just to fulfill their personal goal.
This person has been out of school for years, and had been studying for over a year and still cannot get a good GRE score, and has been turned down by grad schools despite their publications and work experience, probably in favor of some 18 year old who never synthesized a compound in their life but who had a high GRE score.
How can this test predict success when one is already successful, but just wants the pay raise by getting their PhD?
It seems crazy that the test is supposed to “prepare you for life” by being able to calculate the area of a polyhedron, which has nothing to do with most biological scientific careers, whereas the ability to actually carry out scientific experiments, publish, and patent drugs to help sick people live better lives means absolutely NOTHING to grad schools.
Yes, I am peeved about that. Hypothetically, that is.
LikeLike
SamIAm
April 18, 2013 at 3:04 am