orgtheory.net

curating PLoS One

On the Facebook group, Jerry finally admitted that PLoS One was not the journal of the cheeto eating antichrist. It has highly cited articles. It has good papers. It has a high impact factor. In other words, it’s gonna be fine. But Jerry did raise one legitimate issue – how to curate the massive stream of PLoS One papers? There will obviously be many papers of low quality in the PLoS One model.

At first, I thought it was a problem. Then, I realized it wasn’t a problem at all. There are fairly easy ways to curate:

  • Self-curation: People can publicize their own work.
  • Crowd sourcing: Papers acquire reputation from informal networks. It’s happening on twitter right now.
  • Citation count: Papers that the community cites get highlighted.
  • Media attention: Papers attracting the media get highlighted.
  • Prizes: PLoS – or any other group – can award prizes for excellence.
  • Editorial/professional curation: People select good papers within their area of expertise. E.g., “Best PLoS Papers in Nuclear Fission 2014.”

Here’s the ironic thing – ASQ – Jerry’s journal – already curates papers for people who won’t read the whole journal. There is the ASQ award. The ASQ staff reports media mentions for specific papers. The ASQ blog summarizes papers for a larger audience. I couldn’t find it on the current website, but I think ASQ editors used to list papers from recent years fitting with a certain topic. ASQ isn’t alone. Other publishers use similar methods. For example, SSRN lists articles by “most downloaded.” Curation already exists and it works. In other words, Jerry should encourage the PLoS One community to emulate ASQ’s curation practices. It would be generous and help PLoS One reach the next stage in its development.

50+ chapters of grad skool advice goodness: From Black Power/Grad Skool Rulz 

Written by fabiorojas

March 12, 2014 at 12:11 am

15 Responses

Subscribe to comments with RSS.

  1. A lot of issues in the open source model, but one that worries me: popular media and popular confusion. The media already swoop up whatever exotic research reports they come across, the weirder the better. But at least now there is a sense that studies published in heavily screened academic journals are probably more trustworthy, in general, than those that aren’t. In the open source model, if I understand correctly, the hurdle to publish is significantly lower. Academics can reason together to separate the wheat from the chaff, but the public is going to get whipsawed by reporters and bloggers jumping on the latest dramatic “findings.” (And Andy Gelman will be overworked knocking the studies down.)

    Like

    Claude Fischer

    March 12, 2014 at 1:45 am

  2. Claude, I am more than willing to let the public have its red meat if we can get more science out there. The goal of the PLoS One model is to broadly expand what is available. The occasional bomb in PLoS One, for me, is outweighed by the larger number of papers that would otherwise get suppressed by the extended publication process.

    Like

    fabiorojas

    March 12, 2014 at 3:17 am

  3. As someone who studies the Oscars and pop music, I for one welcome our new-pop-media-as-gatekeeper overlords.

    Like

    gabrielrossman

    March 12, 2014 at 2:12 pm

  4. “Getting more science out there” does not seem to be a problem. The number of journals is increasing at a crazy rate, and every morning brings emailed invitations to publish in new sketchy journals. In PLoS One this morning, they list 93,762 articles (it was 92,000 when the last discussion started), and 4298 in Sociology alone. I have to imagine that this is more than all ASA journals combined during the same period (since 2006 or so), and probably as much as the top 10 sociology journals combined. As Kim W pointed out, there are journals willing to publish computer-generated gibberish, apparently in bulk. We are not suffering a shortage of articles.

    In principle, we could abandon all journals and agree to publish exclusively in PLoS One. Then curation and dialogue would let the important things emerge from the less-important. Some hazards: (1) # of downloads favors the sensational, and can generate self-reinforcing dynamics that don’t distinguish quality [see Salaganik and pals’ study]; (2) media attention as a guide – don’t get me started; (3) networks or self-promotion by authors may favor things other than quality [catchy topic, famous author, fancy institution], and can induce unfortunate practices — hell,people might include advertisements in their signature line; (4) citation measures are increasingly worthless because there are well-known and widely-used practices to inflate Google Scholar, and editors have learned how to use practices like citation cartels to artificially inflate ISI numbers.

    At ASQ we hired a computational linguist to take the corpus of our published work since 1999 and figure out what were the big themes and questions. She tossed all our papers into the linguistic centrifuge and came out with about a dozen clusters based on similarity of terms used, citations, etc.; I then went through to pick out exemplars of each topic over the years to convey a sense of what gets published in ASQ, the results of which are here: http://asq.sagepub.com/cgi/collection (Lots of interesting findings, like we publish much more on gender and inequality or social movements than one might expect.)

    Computerized content analysis followed by human curation took a fair bit of effort for just ~300 papers in one journal. I can’t imagine what it would take for 4300 sociology papers in PLoS One, but I’m less sanguine than Gabriel that our pop media overlords are the right ones to do it.

    Sounds like an opportunity for a new curation app. I volunteer Fabio for this.

    Like

    jerrydavisumich

    March 12, 2014 at 2:41 pm

  5. Jerry,

    I was making a joke about pop media mentions serving my own self interest as someone with relatively appealing research topics, see http://knowyourmeme.com/memes/i-for-one-welcome-our-new-insect-overlords

    As for the idea that download counts follow a cumulative advantage process, the same is true of citation counts as you can tell by the fact that the distribution approximates a power law.

    Like

    gabrielrossman

    March 12, 2014 at 3:58 pm

  6. Gabriel — yep, caught that allusion, although I did not realize it predated Kent Brockman. (Perhaps only a true scholar of Hollywood would recognize Joan Collins as the original speaker of that line, and thus your comment burnishes your academic cred even further. You shall not dwell in the long tail…unless I got that power law thingy wrong.)
    Now if we can just get Fabio started on that app.

    Like

    jerrydavisumich

    March 12, 2014 at 4:28 pm

  7. Fabio, I’d like to hear your take on the value-added of journals like PLoS One. I understand what it means for the author. The benefits for the author have been spelled out many times on this blog. But what if I don’t really care whether a given a given author has a difficult time of it publishing? What if I am in it for the science? “More Tweets, More Votes” excepted of course, how have the 4,298 papers published in sociology there advanced our knowledge?

    I’ll admit that I’m old, old enough to remember the episode of the Bionic Woman that was on the teevee a few days ago. Back in my day we had these things called “vanity presses.” It was where kooks took things that couldn’t otherwise pass professional muster. These days, the kids not only have vanity presses, but a whole intertubes full of things that – to an old – look a lot like vanity presses. Standards are changing. In my day, a kook was no less of a kook for having paid to publish with a vanity press. Today, I’m not so sure. Publishing papers in PLoS One that no journal with professional standards would publish is A-OK with the kids. Good for the kids, I guess, but, once again, what if I really don’t care about the kids (other than wanting them to stay off my lawn)? What does work of that sort contribute to our knowledge? I’m teachable, so please inform.

    Like

    Oldy McOld

    March 12, 2014 at 11:07 pm

  8. Oldy, PLoS One’s model is *not* that there are no professional standards. It is a peer-reviewed journal. What’s most distinctive about its model is that papers are never rejected on the basis of their “significance” to a discipline — only their quality.

    Why does this matter? For one thing, because it has the potentially to substantially cut down on the positive results bias in published material. As someone who is leery of overtreatment in medical contexts, I care about this quite a bit. I think PLosOne is a significant service to science.

    Like

    gradstudentbyday

    March 12, 2014 at 11:47 pm

  9. This interview with a PLoS One editorial board member might be educational about how the journal works.

    Like

    gradstudentbyday

    March 12, 2014 at 11:49 pm

  10. What gradstudentbyday said. PLOS ONE has standards – the standard is basically methodological competence. It is agnostic on importance or how interesting the paper is with the idea that this will be determined by the field post publication. I have reviewed for PLOS ONE (and published in it). One paper I reviewed for them ended up getting rejected because the method was inappropriate for the data. The other went through multiple rounds of R&R (though each round only took about 6 weeks). One useful niche it does fill is that it allows people to publish interesting empirical findings that might have difficulty finding a home in traditional journals which are more focused on theoretical development.

    Like

    JD

    March 13, 2014 at 12:52 am

  11. I submitted something to PLoS One. It was rejected on the basis of a single unqualified reviewer’s review. I subsequently got it published in a soc journal. Those 4298 sociology articles must be pretty good!

    Like

    Philip N. Cohen

    March 13, 2014 at 11:24 pm

  12. i think the plos one evangelists’ claim is that because journal reviews are (almost) random draws, by publishing lots and lots of articles (4298 whoa) the good ones on average will make it through even with all the extra noise.

    Like

    gradstudentbytheBay

    March 14, 2014 at 5:37 am

  13. Basically I still can’t get how 4300 additional sociology papers with higher variation in quality can improve our understanding about what’s happening in the world. As an author who needs publication and especially those who see lower possibility of publishing in the traditional journal would like this model, but as social scientist who also needs to care for the ‘entire’ human knowledge, this looks more than less of agency problem.

    Like

    anotherGradStudent

    March 15, 2014 at 6:54 am

  14. There aren’t really 4,300 sociology articles. When you publish you can tag your article in as many subject areas as you want. Basically people whose articles have anything to do with social behavior, social science, health, medicine, etc tag their articles as sociology. I just clicked on the sociology subject section and the first two articles listed are called “Exome Sequencing Analysis Identifies Compound Heterozygous Mutation in ABCA4 in a Chinese Family” and “Nontuberculous Mycobacterial Disease Mortality in the United States, 1999-2010.”

    Like

    JD

    March 15, 2014 at 7:31 am

  15. If that’s how they categories articles, I wonder who picked to review my paper.

    Like

    Philip N. Cohen

    March 15, 2014 at 9:00 pm


Comments are closed.

%d bloggers like this: